2024-05-03T21:03:00.784 INFO:root:teuthology version: 0.0.1.dev271+g1ae7ad8 2024-05-03T21:03:00.785 DEBUG:teuthology.run:Teuthology command: teuthology --description fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} ms_mode/crc wsync/yes} objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/random export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/no 5-quiesce/with-quiesce 6-workunit/suites/pjd}} --verbose --owner scheduled_rishabh@teuthology --name rishabh-2024-05-03_13:56:19-fs-wip-rishabh-testing-20240501.193033-testing-default-smithi --archive /home/teuthworker/archive/rishabh-2024-05-03_13:56:19-fs-wip-rishabh-testing-20240501.193033-testing-default-smithi/7687512 -- /home/teuthworker/archive/rishabh-2024-05-03_13:56:19-fs-wip-rishabh-testing-20240501.193033-testing-default-smithi/7687512/orig.config.yaml 2024-05-03T21:03:00.843 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-05-03T21:03:00.944 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/rishabh-2024-05-03_13:56:19-fs-wip-rishabh-testing-20240501.193033-testing-default-smithi/7687512 branch: wip-rishabh-testing-20240501.193033 description: fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} ms_mode/crc wsync/yes} objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/random export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/no 5-quiesce/with-quiesce 6-workunit/suites/pjd}} email: null first_in_suite: false job_id: '7687512' kernel: client: sha1: distro flavor: default kdb: true sha1: 806ec36b21129f113ce356c007dd59694d340b37 ktype: distro last_in_suite: false log-rotate: ceph-mds: 10G ceph-osd: 10G machine_type: smithi mgrmodules: sequential: - print: Enabling mgr modules name: rishabh-2024-05-03_13:56:19-fs-wip-rishabh-testing-20240501.193033-testing-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 20 - machine: disk: 200 os_type: centos os_version: 9.stream overrides: admin_socket: branch: wip-rishabh-testing-20240501.193033 ceph: cephfs: max_mds: 3 session_timeout: 300 standby_replay: true conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 fuse default permissions: false fuse set user groups: true rados mon op timeout: 900 rados osd op timeout: 900 global: ms die on skipped message: false mds: debug mds: 20 debug mds balancer: 20 debug ms: 1 mds bal fragment size max: 10000 mds bal merge size: 5 mds bal split bits: 3 mds bal split size: 100 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true mds_bal_replicate_threshold: 1 mds_export_ephemeral_random_max: 0.1 osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mgr: cephsqlite lock renewal timeout: 900000 debug client: 20 debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon down mkfs grace: 300 mon op complaint time: 120 osd: bdev async discard: true bdev enable discard: true bluestore allocator: bitmap bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug ms: 1 debug osd: 20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd op complaint time: 180 osd shutdown pgref assert: true osd_max_omap_entries_per_request: 10 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - FS_DEGRADED - FS_INLINE_DATA_DEPRECATED - FS_WITH_FAILED_MDS - MDS_ALL_DOWN - MDS_DAMAGE - MDS_DEGRADED - MDS_FAILED - MDS_INSUFFICIENT_STANDBY - MDS_UP_LESS_THAN_MAX - filesystem is online with fewer MDS than max_mds - POOL_APP_NOT_ENABLED - do not have an application enabled - overall HEALTH_ - Replacing daemon - deprecated feature inline_data - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding - PG_AVAILABILITY - PG_DEGRADED - Reduced data availability sha1: 9a047fe7599a549042d06a3fb5b6813a203eabea subvols: create: 2 ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs ceph-fuse: client.0: mount_subvol_num: 0 check-counter: counters: mds: - mds.exported - mds.imported - min: 3 name: mds_cache.dir_update - min: 3 name: mds_cache.dir_update_receipt install: ceph: flavor: default sha1: 9a047fe7599a549042d06a3fb5b6813a203eabea kclient: client.0: mount_subvol_num: 1 mntopts: - ms_mode=crc - wsync syntax: v1 selinux: allowlist: - scontext=system_u:system_r:logrotate_t:s0 - scontext=system_u:system_r:getty_t:s0 - scontext=system_u:system_r:getty_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-rishabh-testing-20240501.193033 sha1: 9a047fe7599a549042d06a3fb5b6813a203eabea owner: scheduled_rishabh@teuthology priority: 75 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mgr.x - mds.a - mds.d - mds.g - mds.j - osd.0 - osd.3 - osd.6 - osd.9 - client.0 - - mon.b - mgr.y - mds.b - mds.e - mds.h - mds.k - osd.1 - osd.4 - osd.7 - osd.10 - - mon.c - mgr.z - mds.c - mds.f - mds.i - mds.l - osd.2 - osd.5 - osd.8 - osd.11 seed: 1698 sha1: 9a047fe7599a549042d06a3fb5b6813a203eabea sleep_before_teardown: 0 subset: 434/512 suite: fs suite_branch: wip-rishabh-testing-20240501.193033 suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_9a047fe7599a549042d06a3fb5b6813a203eabea/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: 9a047fe7599a549042d06a3fb5b6813a203eabea syslog: ignorelist: - WARNING*.*check_session_state - WARNING*.*__ceph_remove_cap targets: smithi067.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJ81i6lrC83Ijvfp+6tde6t2slK88S6qzizyaiMEqOih/v1PVXq/+9QalUEb6RtSdFWo7xnz5zxR5rSIGbsIoss= smithi082.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBBf+8ZvkK/kJ4MVElFtVYn6yZjyxDjUiAZ+5izCOsagcpMwafTi+CIW9h9yFR8CBaFd/ZOMPEwtGEOrcEcAraXs= smithi138.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBHEyjDehjF3HCOIxy1P7bmtpFVZVF14Po16BQ9rniDH6Tziv/6VXwtuGd4VmH5YOppSr3AzIImUREb0gZYfCQzc= tasks: - install: extra_packages: deb: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror rpm: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror extra_system_packages: deb: - bison - flex - libelf-dev - libssl-dev - network-manager - iproute2 - util-linux - dump - indent - libaio-dev - libtool-bin - uuid-dev - xfslibs-dev - postgresql - postgresql-client - postgresql-common - postgresql-contrib rpm: - bison - flex - elfutils-libelf-devel - openssl-devel - NetworkManager - iproute - util-linux - libacl-devel - libaio-devel - libattr-devel - libtool - libuuid-devel - xfsdump - xfsprogs - xfsprogs-devel - libaio-devel - libtool - libuuid-devel - xfsprogs-devel - postgresql - postgresql-server - postgresql-contrib - cephadm: roleless: false - cephadm.shell: mon.a: - ceph orch status - ceph orch ps - ceph orch ls - ceph orch host ls - ceph orch device ls - cephadm.shell: mon.a: - ceph fs dump - ceph osd dump - fs.ready: timeout: 300 - sequential: - mgrmodules - kclient: null - exec: mon.a: - ceph fs set cephfs balance_automate false - ceph fs subvolumegroup pin cephfs qa random 0.10 - check-counter: {} - quiescer: null - workunit: clients: all: - suites/pjd.sh timeout: 6h teuthology: fragments_dropped: [] meta: {} postmerge: - local function is_kupstream() return yaml.ktype == 'upstream' end - local function is_kdistro() return yaml.ktype == 'distro' end - local function is_fuse() return false end - local function syntax_version() return yaml.overrides.kclient.syntax end teuthology_branch: main teuthology_sha1: 1ae7ad82388e92a475afff437d49054826c019a1 timestamp: 2024-05-03_13:56:19 tube: smithi user: rishabh verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-05-03T21:03:00.944 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_9a047fe7599a549042d06a3fb5b6813a203eabea/qa; will attempt to use it 2024-05-03T21:03:00.945 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_9a047fe7599a549042d06a3fb5b6813a203eabea/qa/tasks 2024-05-03T21:03:00.945 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-05-03T21:03:00.947 INFO:teuthology.task.internal:Checking packages... 2024-05-03T21:03:00.970 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '9a047fe7599a549042d06a3fb5b6813a203eabea' 2024-05-03T21:03:00.970 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-05-03T21:03:00.970 INFO:teuthology.packaging:ref: None 2024-05-03T21:03:00.970 INFO:teuthology.packaging:tag: None 2024-05-03T21:03:00.971 INFO:teuthology.packaging:branch: wip-rishabh-testing-20240501.193033 2024-05-03T21:03:00.971 INFO:teuthology.packaging:sha1: 9a047fe7599a549042d06a3fb5b6813a203eabea 2024-05-03T21:03:00.971 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=wip-rishabh-testing-20240501.193033 2024-05-03T21:03:01.187 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-3450.g9a047fe7 2024-05-03T21:03:01.189 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-05-03T21:03:01.196 INFO:teuthology.task.internal:no buildpackages task found 2024-05-03T21:03:01.196 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-05-03T21:03:01.206 INFO:teuthology.task.internal:Saving configuration 2024-05-03T21:03:01.226 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-05-03T21:03:01.235 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-05-03T21:03:01.260 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi067.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/rishabh-2024-05-03_13:56:19-fs-wip-rishabh-testing-20240501.193033-testing-default-smithi/7687512', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-05-03 20:57:55.108515', 'locked_by': 'scheduled_rishabh@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJ81i6lrC83Ijvfp+6tde6t2slK88S6qzizyaiMEqOih/v1PVXq/+9QalUEb6RtSdFWo7xnz5zxR5rSIGbsIoss='} 2024-05-03T21:03:01.280 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi082.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/rishabh-2024-05-03_13:56:19-fs-wip-rishabh-testing-20240501.193033-testing-default-smithi/7687512', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-05-03 20:57:55.110312', 'locked_by': 'scheduled_rishabh@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBBf+8ZvkK/kJ4MVElFtVYn6yZjyxDjUiAZ+5izCOsagcpMwafTi+CIW9h9yFR8CBaFd/ZOMPEwtGEOrcEcAraXs='} 2024-05-03T21:03:01.304 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi138.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/rishabh-2024-05-03_13:56:19-fs-wip-rishabh-testing-20240501.193033-testing-default-smithi/7687512', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-05-03 20:57:55.111498', 'locked_by': 'scheduled_rishabh@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBHEyjDehjF3HCOIxy1P7bmtpFVZVF14Po16BQ9rniDH6Tziv/6VXwtuGd4VmH5YOppSr3AzIImUREb0gZYfCQzc='} 2024-05-03T21:03:01.304 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-05-03T21:03:01.313 INFO:teuthology.task.internal:roles: ubuntu@smithi067.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.d', 'mds.g', 'mds.j', 'osd.0', 'osd.3', 'osd.6', 'osd.9', 'client.0'] 2024-05-03T21:03:01.313 INFO:teuthology.task.internal:roles: ubuntu@smithi082.front.sepia.ceph.com - ['mon.b', 'mgr.y', 'mds.b', 'mds.e', 'mds.h', 'mds.k', 'osd.1', 'osd.4', 'osd.7', 'osd.10'] 2024-05-03T21:03:01.313 INFO:teuthology.task.internal:roles: ubuntu@smithi138.front.sepia.ceph.com - ['mon.c', 'mgr.z', 'mds.c', 'mds.f', 'mds.i', 'mds.l', 'osd.2', 'osd.5', 'osd.8', 'osd.11'] 2024-05-03T21:03:01.313 INFO:teuthology.run_tasks:Running task console_log... 2024-05-03T21:03:01.418 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7fba028cc160>, signals=[15]) 2024-05-03T21:03:01.418 INFO:teuthology.run_tasks:Running task internal.connect... 2024-05-03T21:03:01.424 INFO:teuthology.task.internal:Opening connections... 2024-05-03T21:03:01.424 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi067.front.sepia.ceph.com 2024-05-03T21:03:01.426 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi067.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-03T21:03:01.502 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi082.front.sepia.ceph.com 2024-05-03T21:03:01.503 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi082.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-03T21:03:01.576 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi138.front.sepia.ceph.com 2024-05-03T21:03:01.577 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi138.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-03T21:03:01.650 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-05-03T21:03:01.658 DEBUG:teuthology.orchestra.run.smithi067:> uname -m 2024-05-03T21:03:01.676 INFO:teuthology.orchestra.run.smithi067.stdout:x86_64 2024-05-03T21:03:01.677 DEBUG:teuthology.orchestra.run.smithi067:> cat /etc/os-release 2024-05-03T21:03:01.734 INFO:teuthology.orchestra.run.smithi067.stdout:NAME="CentOS Stream" 2024-05-03T21:03:01.734 INFO:teuthology.orchestra.run.smithi067.stdout:VERSION="9" 2024-05-03T21:03:01.734 INFO:teuthology.orchestra.run.smithi067.stdout:ID="centos" 2024-05-03T21:03:01.735 INFO:teuthology.orchestra.run.smithi067.stdout:ID_LIKE="rhel fedora" 2024-05-03T21:03:01.735 INFO:teuthology.orchestra.run.smithi067.stdout:VERSION_ID="9" 2024-05-03T21:03:01.735 INFO:teuthology.orchestra.run.smithi067.stdout:PLATFORM_ID="platform:el9" 2024-05-03T21:03:01.735 INFO:teuthology.orchestra.run.smithi067.stdout:PRETTY_NAME="CentOS Stream 9" 2024-05-03T21:03:01.735 INFO:teuthology.orchestra.run.smithi067.stdout:ANSI_COLOR="0;31" 2024-05-03T21:03:01.735 INFO:teuthology.orchestra.run.smithi067.stdout:LOGO="fedora-logo-icon" 2024-05-03T21:03:01.735 INFO:teuthology.orchestra.run.smithi067.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-05-03T21:03:01.735 INFO:teuthology.orchestra.run.smithi067.stdout:HOME_URL="https://centos.org/" 2024-05-03T21:03:01.735 INFO:teuthology.orchestra.run.smithi067.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-05-03T21:03:01.735 INFO:teuthology.orchestra.run.smithi067.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-05-03T21:03:01.735 INFO:teuthology.orchestra.run.smithi067.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-05-03T21:03:01.736 INFO:teuthology.lock.ops:Updating smithi067.front.sepia.ceph.com on lock server 2024-05-03T21:03:01.762 DEBUG:teuthology.orchestra.run.smithi082:> uname -m 2024-05-03T21:03:01.784 INFO:teuthology.orchestra.run.smithi082.stdout:x86_64 2024-05-03T21:03:01.784 DEBUG:teuthology.orchestra.run.smithi082:> cat /etc/os-release 2024-05-03T21:03:01.841 INFO:teuthology.orchestra.run.smithi082.stdout:NAME="CentOS Stream" 2024-05-03T21:03:01.841 INFO:teuthology.orchestra.run.smithi082.stdout:VERSION="9" 2024-05-03T21:03:01.841 INFO:teuthology.orchestra.run.smithi082.stdout:ID="centos" 2024-05-03T21:03:01.841 INFO:teuthology.orchestra.run.smithi082.stdout:ID_LIKE="rhel fedora" 2024-05-03T21:03:01.841 INFO:teuthology.orchestra.run.smithi082.stdout:VERSION_ID="9" 2024-05-03T21:03:01.841 INFO:teuthology.orchestra.run.smithi082.stdout:PLATFORM_ID="platform:el9" 2024-05-03T21:03:01.841 INFO:teuthology.orchestra.run.smithi082.stdout:PRETTY_NAME="CentOS Stream 9" 2024-05-03T21:03:01.841 INFO:teuthology.orchestra.run.smithi082.stdout:ANSI_COLOR="0;31" 2024-05-03T21:03:01.841 INFO:teuthology.orchestra.run.smithi082.stdout:LOGO="fedora-logo-icon" 2024-05-03T21:03:01.842 INFO:teuthology.orchestra.run.smithi082.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-05-03T21:03:01.842 INFO:teuthology.orchestra.run.smithi082.stdout:HOME_URL="https://centos.org/" 2024-05-03T21:03:01.842 INFO:teuthology.orchestra.run.smithi082.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-05-03T21:03:01.842 INFO:teuthology.orchestra.run.smithi082.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-05-03T21:03:01.842 INFO:teuthology.orchestra.run.smithi082.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-05-03T21:03:01.842 INFO:teuthology.lock.ops:Updating smithi082.front.sepia.ceph.com on lock server 2024-05-03T21:03:01.862 DEBUG:teuthology.orchestra.run.smithi138:> uname -m 2024-05-03T21:03:01.880 INFO:teuthology.orchestra.run.smithi138.stdout:x86_64 2024-05-03T21:03:01.880 DEBUG:teuthology.orchestra.run.smithi138:> cat /etc/os-release 2024-05-03T21:03:01.937 INFO:teuthology.orchestra.run.smithi138.stdout:NAME="CentOS Stream" 2024-05-03T21:03:01.937 INFO:teuthology.orchestra.run.smithi138.stdout:VERSION="9" 2024-05-03T21:03:01.937 INFO:teuthology.orchestra.run.smithi138.stdout:ID="centos" 2024-05-03T21:03:01.938 INFO:teuthology.orchestra.run.smithi138.stdout:ID_LIKE="rhel fedora" 2024-05-03T21:03:01.938 INFO:teuthology.orchestra.run.smithi138.stdout:VERSION_ID="9" 2024-05-03T21:03:01.938 INFO:teuthology.orchestra.run.smithi138.stdout:PLATFORM_ID="platform:el9" 2024-05-03T21:03:01.938 INFO:teuthology.orchestra.run.smithi138.stdout:PRETTY_NAME="CentOS Stream 9" 2024-05-03T21:03:01.938 INFO:teuthology.orchestra.run.smithi138.stdout:ANSI_COLOR="0;31" 2024-05-03T21:03:01.938 INFO:teuthology.orchestra.run.smithi138.stdout:LOGO="fedora-logo-icon" 2024-05-03T21:03:01.938 INFO:teuthology.orchestra.run.smithi138.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-05-03T21:03:01.938 INFO:teuthology.orchestra.run.smithi138.stdout:HOME_URL="https://centos.org/" 2024-05-03T21:03:01.938 INFO:teuthology.orchestra.run.smithi138.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-05-03T21:03:01.938 INFO:teuthology.orchestra.run.smithi138.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-05-03T21:03:01.938 INFO:teuthology.orchestra.run.smithi138.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-05-03T21:03:01.939 INFO:teuthology.lock.ops:Updating smithi138.front.sepia.ceph.com on lock server 2024-05-03T21:03:01.961 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-05-03T21:03:01.977 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-05-03T21:03:02.003 INFO:teuthology.task.internal:Checking for old test directory... 2024-05-03T21:03:02.003 DEBUG:teuthology.orchestra.run.smithi067:> test '!' -e /home/ubuntu/cephtest 2024-05-03T21:03:02.005 DEBUG:teuthology.orchestra.run.smithi082:> test '!' -e /home/ubuntu/cephtest 2024-05-03T21:03:02.008 DEBUG:teuthology.orchestra.run.smithi138:> test '!' -e /home/ubuntu/cephtest 2024-05-03T21:03:02.024 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-05-03T21:03:02.031 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-05-03T21:03:02.031 DEBUG:teuthology.orchestra.run.smithi067:> test -z $(ls -A /var/lib/ceph) 2024-05-03T21:03:02.063 DEBUG:teuthology.orchestra.run.smithi082:> test -z $(ls -A /var/lib/ceph) 2024-05-03T21:03:02.065 DEBUG:teuthology.orchestra.run.smithi138:> test -z $(ls -A /var/lib/ceph) 2024-05-03T21:03:02.095 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-05-03T21:03:02.179 INFO:teuthology.run_tasks:Running task kernel... 2024-05-03T21:03:02.197 INFO:teuthology.task.kernel:normalize config orig: {'client': {'sha1': 'distro'}, 'flavor': 'default', 'kdb': True, 'sha1': '806ec36b21129f113ce356c007dd59694d340b37'} 2024-05-03T21:03:02.198 INFO:teuthology.task.kernel:normalize config final: {'client.0': {'sha1': 'distro'}} 2024-05-03T21:03:02.198 INFO:teuthology.task.kernel:config {'client.0': {'sha1': 'distro'}}, timeout 300 2024-05-03T21:03:02.198 DEBUG:teuthology.orchestra.run.smithi067:> test -f /run/.containerenv -o -f /.dockerenv 2024-05-03T21:03:02.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-03T21:03:02.216 DEBUG:teuthology.orchestra.run.smithi067:> uname -r 2024-05-03T21:03:02.274 INFO:teuthology.orchestra.run.smithi067.stdout:5.14.0-437.el9.x86_64 2024-05-03T21:03:02.274 INFO:teuthology.task.kernel:Running kernel on smithi067: 5.14.0-437.el9.x86_64 2024-05-03T21:03:02.274 DEBUG:teuthology.orchestra.run.smithi067:> sudo yum install -y kernel 2024-05-03T21:03:04.806 INFO:teuthology.orchestra.run.smithi067.stdout:CentOS Stream 9 - BaseOS 8.5 MB/s | 8.0 MB 00:00 2024-05-03T21:03:06.771 INFO:teuthology.orchestra.run.smithi067.stdout:CentOS Stream 9 - AppStream 22 MB/s | 19 MB 00:00 2024-05-03T21:03:11.947 INFO:teuthology.orchestra.run.smithi067.stdout:CentOS Stream 9 - CRB 4.2 MB/s | 6.2 MB 00:01 2024-05-03T21:03:13.323 INFO:teuthology.orchestra.run.smithi067.stdout:CentOS Stream 9 - Extras packages 98 kB/s | 16 kB 00:00 2024-05-03T21:03:14.816 INFO:teuthology.orchestra.run.smithi067.stdout:Extra Packages for Enterprise Linux 15 MB/s | 21 MB 00:01 2024-05-03T21:03:19.999 INFO:teuthology.orchestra.run.smithi067.stdout:lab-extras 42 kB/s | 1.7 kB 00:00 2024-05-03T21:03:22.080 INFO:teuthology.orchestra.run.smithi067.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-05-03T21:03:22.081 INFO:teuthology.orchestra.run.smithi067.stdout:Package kernel-5.14.0-437.el9.x86_64 is already installed. 2024-05-03T21:03:22.173 INFO:teuthology.orchestra.run.smithi067.stdout:Dependencies resolved. 2024-05-03T21:03:22.174 INFO:teuthology.orchestra.run.smithi067.stdout:Nothing to do. 2024-05-03T21:03:22.174 INFO:teuthology.orchestra.run.smithi067.stdout:Complete! 2024-05-03T21:03:22.291 DEBUG:teuthology.orchestra.run.smithi067:> echo no | sudo yum reinstall kernel || true 2024-05-03T21:03:22.814 INFO:teuthology.orchestra.run.smithi067.stdout:Last metadata expiration check: 0:00:03 ago on Fri 03 May 2024 09:03:19 PM UTC. 2024-05-03T21:03:22.945 INFO:teuthology.orchestra.run.smithi067.stdout:Dependencies resolved. 2024-05-03T21:03:22.960 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2024-05-03T21:03:22.960 INFO:teuthology.orchestra.run.smithi067.stdout: Package Architecture Version Repository Size 2024-05-03T21:03:22.961 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2024-05-03T21:03:22.961 INFO:teuthology.orchestra.run.smithi067.stdout:Reinstalling: 2024-05-03T21:03:22.961 INFO:teuthology.orchestra.run.smithi067.stdout: kernel x86_64 5.14.0-437.el9 baseos 255 k 2024-05-03T21:03:22.961 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-05-03T21:03:22.961 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction Summary 2024-05-03T21:03:22.961 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2024-05-03T21:03:22.961 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-05-03T21:03:22.961 INFO:teuthology.orchestra.run.smithi067.stdout:Total download size: 255 k 2024-05-03T21:03:22.961 INFO:teuthology.orchestra.run.smithi067.stdout:Installed size: 0 2024-05-03T21:03:22.961 INFO:teuthology.orchestra.run.smithi067.stderr:Operation aborted. 2024-05-03T21:03:23.019 INFO:teuthology.orchestra.run.smithi067.stdout:Is this ok [y/N]: 2024-05-03T21:03:23.020 DEBUG:teuthology.orchestra.run.smithi067:> sudo yum reinstall -y kernel || true 2024-05-03T21:03:23.536 INFO:teuthology.orchestra.run.smithi067.stdout:Last metadata expiration check: 0:00:04 ago on Fri 03 May 2024 09:03:19 PM UTC. 2024-05-03T21:03:23.657 INFO:teuthology.orchestra.run.smithi067.stdout:Dependencies resolved. 2024-05-03T21:03:23.658 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2024-05-03T21:03:23.658 INFO:teuthology.orchestra.run.smithi067.stdout: Package Architecture Version Repository Size 2024-05-03T21:03:23.658 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2024-05-03T21:03:23.658 INFO:teuthology.orchestra.run.smithi067.stdout:Reinstalling: 2024-05-03T21:03:23.659 INFO:teuthology.orchestra.run.smithi067.stdout: kernel x86_64 5.14.0-437.el9 baseos 255 k 2024-05-03T21:03:23.659 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-05-03T21:03:23.659 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction Summary 2024-05-03T21:03:23.659 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2024-05-03T21:03:23.659 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-05-03T21:03:23.659 INFO:teuthology.orchestra.run.smithi067.stdout:Total download size: 255 k 2024-05-03T21:03:23.659 INFO:teuthology.orchestra.run.smithi067.stdout:Installed size: 0 2024-05-03T21:03:23.659 INFO:teuthology.orchestra.run.smithi067.stdout:Downloading Packages: 2024-05-03T21:03:24.135 INFO:teuthology.orchestra.run.smithi067.stdout:kernel-5.14.0-437.el9.x86_64.rpm 1.2 MB/s | 255 kB 00:00 2024-05-03T21:03:24.135 INFO:teuthology.orchestra.run.smithi067.stdout:-------------------------------------------------------------------------------- 2024-05-03T21:03:24.135 INFO:teuthology.orchestra.run.smithi067.stdout:Total 535 kB/s | 255 kB 00:00 2024-05-03T21:03:24.151 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction check 2024-05-03T21:03:24.337 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction check succeeded. 2024-05-03T21:03:24.337 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction test 2024-05-03T21:03:24.342 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction test succeeded. 2024-05-03T21:03:24.343 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction 2024-05-03T21:03:24.468 INFO:teuthology.orchestra.run.smithi067.stdout: Preparing : 1/1 2024-05-03T21:03:24.561 INFO:teuthology.orchestra.run.smithi067.stdout: Reinstalling : kernel-5.14.0-437.el9.x86_64 1/2 2024-05-03T21:03:24.561 INFO:teuthology.orchestra.run.smithi067.stdout: Cleanup : kernel-5.14.0-437.el9.x86_64 2/2 2024-05-03T21:03:25.245 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: kernel-5.14.0-437.el9.x86_64 2/2 2024-05-03T21:03:25.246 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : kernel-5.14.0-437.el9.x86_64 1/2 2024-05-03T21:03:25.912 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : kernel-5.14.0-437.el9.x86_64 2/2 2024-05-03T21:03:25.912 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-05-03T21:03:25.912 INFO:teuthology.orchestra.run.smithi067.stdout:Reinstalled: 2024-05-03T21:03:25.912 INFO:teuthology.orchestra.run.smithi067.stdout: kernel-5.14.0-437.el9.x86_64 2024-05-03T21:03:25.913 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-05-03T21:03:25.913 INFO:teuthology.orchestra.run.smithi067.stdout:Complete! 2024-05-03T21:03:26.092 DEBUG:teuthology.orchestra.run.smithi067:> rpm -q kernel | sort -rV | head -n 1 2024-05-03T21:03:26.140 INFO:teuthology.orchestra.run.smithi067.stdout:kernel-5.14.0-437.el9.x86_64 2024-05-03T21:03:26.141 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-437.el9.x86_64 2024-05-03T21:03:26.141 INFO:teuthology.task.kernel:Newest distro kernel installed and running 2024-05-03T21:03:26.141 DEBUG:teuthology.parallel:result is None 2024-05-03T21:03:26.141 INFO:teuthology.run_tasks:Running task internal.base... 2024-05-03T21:03:26.148 INFO:teuthology.task.internal:Creating test directory... 2024-05-03T21:03:26.149 DEBUG:teuthology.orchestra.run.smithi067:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-05-03T21:03:26.151 DEBUG:teuthology.orchestra.run.smithi082:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-05-03T21:03:26.154 DEBUG:teuthology.orchestra.run.smithi138:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-05-03T21:03:26.172 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-05-03T21:03:26.179 INFO:teuthology.run_tasks:Running task internal.archive... 2024-05-03T21:03:26.185 INFO:teuthology.task.internal:Creating archive directory... 2024-05-03T21:03:26.186 DEBUG:teuthology.orchestra.run.smithi067:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-05-03T21:03:26.206 DEBUG:teuthology.orchestra.run.smithi082:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-05-03T21:03:26.211 DEBUG:teuthology.orchestra.run.smithi138:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-05-03T21:03:26.261 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-05-03T21:03:26.268 INFO:teuthology.task.internal:Enabling coredump saving... 2024-05-03T21:03:26.268 DEBUG:teuthology.orchestra.run.smithi082:> test -f /run/.containerenv -o -f /.dockerenv 2024-05-03T21:03:26.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-03T21:03:26.285 DEBUG:teuthology.orchestra.run.smithi138:> test -f /run/.containerenv -o -f /.dockerenv 2024-05-03T21:03:26.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-03T21:03:26.310 DEBUG:teuthology.orchestra.run.smithi067:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-05-03T21:03:26.312 DEBUG:teuthology.orchestra.run.smithi082:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-05-03T21:03:26.328 DEBUG:teuthology.orchestra.run.smithi138:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-05-03T21:03:26.339 INFO:teuthology.orchestra.run.smithi067.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-03T21:03:26.355 INFO:teuthology.orchestra.run.smithi067.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-03T21:03:26.371 INFO:teuthology.orchestra.run.smithi082.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-03T21:03:26.383 INFO:teuthology.orchestra.run.smithi138.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-03T21:03:26.390 INFO:teuthology.orchestra.run.smithi082.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-03T21:03:26.401 INFO:teuthology.orchestra.run.smithi138.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-03T21:03:26.403 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-05-03T21:03:26.410 INFO:teuthology.task.internal:Configuring sudo... 2024-05-03T21:03:26.411 DEBUG:teuthology.orchestra.run.smithi067:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-05-03T21:03:26.413 DEBUG:teuthology.orchestra.run.smithi082:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-05-03T21:03:26.416 DEBUG:teuthology.orchestra.run.smithi138:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-05-03T21:03:26.471 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-05-03T21:03:26.480 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-05-03T21:03:26.480 DEBUG:teuthology.orchestra.run.smithi067:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-05-03T21:03:26.482 DEBUG:teuthology.orchestra.run.smithi082:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-05-03T21:03:26.485 DEBUG:teuthology.orchestra.run.smithi138:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-05-03T21:03:26.527 DEBUG:teuthology.orchestra.run.smithi067:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-05-03T21:03:26.575 DEBUG:teuthology.orchestra.run.smithi067:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-05-03T21:03:26.632 DEBUG:teuthology.orchestra.run.smithi067:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-05-03T21:03:26.714 DEBUG:teuthology.orchestra.run.smithi067:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-05-03T21:03:26.754 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-05-03T21:03:26.754 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-05-03T21:03:26.819 DEBUG:teuthology.orchestra.run.smithi082:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-05-03T21:03:26.860 DEBUG:teuthology.orchestra.run.smithi082:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-05-03T21:03:26.908 DEBUG:teuthology.orchestra.run.smithi082:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-05-03T21:03:26.991 DEBUG:teuthology.orchestra.run.smithi082:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-05-03T21:03:27.033 DEBUG:teuthology.orchestra.run.smithi082:> set -ex 2024-05-03T21:03:27.033 DEBUG:teuthology.orchestra.run.smithi082:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-05-03T21:03:27.098 DEBUG:teuthology.orchestra.run.smithi138:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-05-03T21:03:27.143 DEBUG:teuthology.orchestra.run.smithi138:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-05-03T21:03:27.186 DEBUG:teuthology.orchestra.run.smithi138:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-05-03T21:03:27.271 DEBUG:teuthology.orchestra.run.smithi138:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-05-03T21:03:27.311 DEBUG:teuthology.orchestra.run.smithi138:> set -ex 2024-05-03T21:03:27.311 DEBUG:teuthology.orchestra.run.smithi138:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-05-03T21:03:27.376 DEBUG:teuthology.orchestra.run.smithi067:> sudo service rsyslog restart 2024-05-03T21:03:27.378 DEBUG:teuthology.orchestra.run.smithi082:> sudo service rsyslog restart 2024-05-03T21:03:27.380 DEBUG:teuthology.orchestra.run.smithi138:> sudo service rsyslog restart 2024-05-03T21:03:27.411 INFO:teuthology.orchestra.run.smithi082.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-05-03T21:03:27.420 INFO:teuthology.orchestra.run.smithi067.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-05-03T21:03:27.452 INFO:teuthology.orchestra.run.smithi138.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-05-03T21:03:27.857 INFO:teuthology.run_tasks:Running task internal.timer... 2024-05-03T21:03:27.866 INFO:teuthology.task.internal:Starting timer... 2024-05-03T21:03:27.866 INFO:teuthology.run_tasks:Running task pcp... 2024-05-03T21:03:27.876 INFO:teuthology.run_tasks:Running task selinux... 2024-05-03T21:03:27.884 DEBUG:teuthology.task:Applying overrides for task selinux: {'allowlist': ['scontext=system_u:system_r:logrotate_t:s0', 'scontext=system_u:system_r:getty_t:s0', 'scontext=system_u:system_r:getty_t:s0']} 2024-05-03T21:03:27.885 DEBUG:teuthology.orchestra.run.smithi067:> sudo service auditd rotate 2024-05-03T21:03:27.968 INFO:teuthology.orchestra.run.smithi067.stdout:Rotating logs: 2024-05-03T21:03:27.969 DEBUG:teuthology.orchestra.run.smithi082:> sudo service auditd rotate 2024-05-03T21:03:28.017 INFO:teuthology.orchestra.run.smithi082.stdout:Rotating logs: 2024-05-03T21:03:28.019 DEBUG:teuthology.orchestra.run.smithi138:> sudo service auditd rotate 2024-05-03T21:03:28.061 INFO:teuthology.orchestra.run.smithi138.stdout:Rotating logs: 2024-05-03T21:03:28.063 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-05-03T21:03:28.063 DEBUG:teuthology.orchestra.run.smithi067:> /usr/sbin/getenforce 2024-05-03T21:03:28.095 INFO:teuthology.orchestra.run.smithi067.stdout:Permissive 2024-05-03T21:03:28.095 DEBUG:teuthology.orchestra.run.smithi082:> /usr/sbin/getenforce 2024-05-03T21:03:28.121 INFO:teuthology.orchestra.run.smithi082.stdout:Permissive 2024-05-03T21:03:28.121 DEBUG:teuthology.orchestra.run.smithi138:> /usr/sbin/getenforce 2024-05-03T21:03:28.150 INFO:teuthology.orchestra.run.smithi138.stdout:Permissive 2024-05-03T21:03:28.150 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi067.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi082.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi138.front.sepia.ceph.com': 'permissive'} 2024-05-03T21:03:28.150 DEBUG:teuthology.orchestra.run.smithi067:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-05-03T21:03:28.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-03T21:03:28.176 DEBUG:teuthology.orchestra.run.smithi082:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-05-03T21:03:28.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-03T21:03:28.204 DEBUG:teuthology.orchestra.run.smithi138:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-05-03T21:03:28.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-03T21:03:28.229 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-05-03T21:03:28.229 DEBUG:teuthology.orchestra.run.smithi067:> sudo /usr/sbin/setenforce permissive 2024-05-03T21:03:28.254 DEBUG:teuthology.orchestra.run.smithi082:> sudo /usr/sbin/setenforce permissive 2024-05-03T21:03:28.290 DEBUG:teuthology.orchestra.run.smithi138:> sudo /usr/sbin/setenforce permissive 2024-05-03T21:03:28.315 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-05-03T21:03:28.328 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2024-05-03T21:03:28.340 INFO:teuthology.repo_utils:Fetching git.ceph.com_ceph-cm-ansible_main from origin 2024-05-03T21:03:28.492 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-05-03T21:03:28.508 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-05-03T21:03:28.510 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi067.front.sepia.ceph.com,smithi082.front.sepia.ceph.com,smithi138.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-05-03T21:07:38.335 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi067.front.sepia.ceph.com'), Remote(name='ubuntu@smithi082.front.sepia.ceph.com'), Remote(name='ubuntu@smithi138.front.sepia.ceph.com')] 2024-05-03T21:07:38.337 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi067.front.sepia.ceph.com' 2024-05-03T21:07:38.338 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi067.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-03T21:07:38.413 DEBUG:teuthology.orchestra.run.smithi067:> true 2024-05-03T21:07:38.499 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi067.front.sepia.ceph.com' 2024-05-03T21:07:38.500 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi082.front.sepia.ceph.com' 2024-05-03T21:07:38.501 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi082.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-03T21:07:38.574 DEBUG:teuthology.orchestra.run.smithi082:> true 2024-05-03T21:07:38.653 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi082.front.sepia.ceph.com' 2024-05-03T21:07:38.654 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi138.front.sepia.ceph.com' 2024-05-03T21:07:38.654 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi138.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-03T21:07:38.732 DEBUG:teuthology.orchestra.run.smithi138:> true 2024-05-03T21:07:38.809 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi138.front.sepia.ceph.com' 2024-05-03T21:07:38.810 INFO:teuthology.run_tasks:Running task clock... 2024-05-03T21:07:38.819 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-05-03T21:07:38.820 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-05-03T21:07:38.820 DEBUG:teuthology.orchestra.run.smithi067:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-05-03T21:07:38.823 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-05-03T21:07:38.823 DEBUG:teuthology.orchestra.run.smithi082:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-05-03T21:07:38.826 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-05-03T21:07:38.826 DEBUG:teuthology.orchestra.run.smithi138:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-05-03T21:07:38.851 INFO:teuthology.orchestra.run.smithi067.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-05-03T21:07:38.853 INFO:teuthology.orchestra.run.smithi082.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-05-03T21:07:38.866 INFO:teuthology.orchestra.run.smithi067.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-05-03T21:07:38.866 INFO:teuthology.orchestra.run.smithi082.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-05-03T21:07:38.877 INFO:teuthology.orchestra.run.smithi138.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-05-03T21:07:38.889 INFO:teuthology.orchestra.run.smithi082.stderr:sudo: ntpd: command not found 2024-05-03T21:07:38.890 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: ntpd: command not found 2024-05-03T21:07:38.890 INFO:teuthology.orchestra.run.smithi138.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-05-03T21:07:38.900 INFO:teuthology.orchestra.run.smithi082.stdout:506 Cannot talk to daemon 2024-05-03T21:07:38.901 INFO:teuthology.orchestra.run.smithi067.stdout:506 Cannot talk to daemon 2024-05-03T21:07:38.913 INFO:teuthology.orchestra.run.smithi082.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-05-03T21:07:38.913 INFO:teuthology.orchestra.run.smithi138.stderr:sudo: ntpd: command not found 2024-05-03T21:07:38.915 INFO:teuthology.orchestra.run.smithi067.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-05-03T21:07:38.923 INFO:teuthology.orchestra.run.smithi138.stdout:506 Cannot talk to daemon 2024-05-03T21:07:38.926 INFO:teuthology.orchestra.run.smithi082.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-05-03T21:07:38.929 INFO:teuthology.orchestra.run.smithi067.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-05-03T21:07:38.937 INFO:teuthology.orchestra.run.smithi138.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-05-03T21:07:38.950 INFO:teuthology.orchestra.run.smithi138.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-05-03T21:07:38.979 INFO:teuthology.orchestra.run.smithi082.stderr:bash: line 1: ntpq: command not found 2024-05-03T21:07:38.982 INFO:teuthology.orchestra.run.smithi082.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-05-03T21:07:38.982 INFO:teuthology.orchestra.run.smithi082.stdout:=============================================================================== 2024-05-03T21:07:38.982 INFO:teuthology.orchestra.run.smithi082.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-05-03T21:07:38.982 INFO:teuthology.orchestra.run.smithi082.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-05-03T21:07:38.983 INFO:teuthology.orchestra.run.smithi082.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-05-03T21:07:38.983 INFO:teuthology.orchestra.run.smithi082.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-05-03T21:07:38.985 INFO:teuthology.orchestra.run.smithi067.stderr:bash: line 1: ntpq: command not found 2024-05-03T21:07:38.988 INFO:teuthology.orchestra.run.smithi067.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-05-03T21:07:38.988 INFO:teuthology.orchestra.run.smithi067.stdout:=============================================================================== 2024-05-03T21:07:38.989 INFO:teuthology.orchestra.run.smithi067.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-05-03T21:07:38.989 INFO:teuthology.orchestra.run.smithi067.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-05-03T21:07:38.989 INFO:teuthology.orchestra.run.smithi067.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-05-03T21:07:38.989 INFO:teuthology.orchestra.run.smithi067.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-05-03T21:07:39.013 INFO:teuthology.orchestra.run.smithi138.stderr:bash: line 1: ntpq: command not found 2024-05-03T21:07:39.018 INFO:teuthology.orchestra.run.smithi138.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-05-03T21:07:39.018 INFO:teuthology.orchestra.run.smithi138.stdout:=============================================================================== 2024-05-03T21:07:39.018 INFO:teuthology.orchestra.run.smithi138.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-05-03T21:07:39.018 INFO:teuthology.orchestra.run.smithi138.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-05-03T21:07:39.018 INFO:teuthology.orchestra.run.smithi138.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-05-03T21:07:39.018 INFO:teuthology.orchestra.run.smithi138.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-05-03T21:07:39.019 INFO:teuthology.run_tasks:Running task install... 2024-05-03T21:07:39.027 DEBUG:teuthology.task.install:project ceph 2024-05-03T21:07:39.027 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '9a047fe7599a549042d06a3fb5b6813a203eabea'}} 2024-05-03T21:07:39.028 DEBUG:teuthology.task.install:config {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': '9a047fe7599a549042d06a3fb5b6813a203eabea'} 2024-05-03T21:07:39.028 INFO:teuthology.task.install:Using flavor: default 2024-05-03T21:07:39.035 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-05-03T21:07:39.035 INFO:teuthology.task.install:extra packages: {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']} 2024-05-03T21:07:39.036 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '9a047fe7599a549042d06a3fb5b6813a203eabea', 'tag': None, 'wait_for_package': False} 2024-05-03T21:07:39.036 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=9a047fe7599a549042d06a3fb5b6813a203eabea 2024-05-03T21:07:39.038 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '9a047fe7599a549042d06a3fb5b6813a203eabea', 'tag': None, 'wait_for_package': False} 2024-05-03T21:07:39.038 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=9a047fe7599a549042d06a3fb5b6813a203eabea 2024-05-03T21:07:39.039 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '9a047fe7599a549042d06a3fb5b6813a203eabea', 'tag': None, 'wait_for_package': False} 2024-05-03T21:07:39.040 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=9a047fe7599a549042d06a3fb5b6813a203eabea 2024-05-03T21:07:39.260 INFO:teuthology.task.install.rpm:Pulling from https://4.chacra.ceph.com/r/ceph/wip-rishabh-testing-20240501.193033/9a047fe7599a549042d06a3fb5b6813a203eabea/centos/9/flavors/default/ 2024-05-03T21:07:39.260 INFO:teuthology.task.install.rpm:Package version is 19.0.0-3450.g9a047fe7 2024-05-03T21:07:39.268 INFO:teuthology.task.install.rpm:Pulling from https://4.chacra.ceph.com/r/ceph/wip-rishabh-testing-20240501.193033/9a047fe7599a549042d06a3fb5b6813a203eabea/centos/9/flavors/default/ 2024-05-03T21:07:39.268 INFO:teuthology.task.install.rpm:Package version is 19.0.0-3450.g9a047fe7 2024-05-03T21:07:39.269 INFO:teuthology.task.install.rpm:Pulling from https://4.chacra.ceph.com/r/ceph/wip-rishabh-testing-20240501.193033/9a047fe7599a549042d06a3fb5b6813a203eabea/centos/9/flavors/default/ 2024-05-03T21:07:39.269 INFO:teuthology.task.install.rpm:Package version is 19.0.0-3450.g9a047fe7 2024-05-03T21:07:39.376 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://4.chacra.ceph.com/r/ceph/wip-rishabh-testing-20240501.193033/9a047fe7599a549042d06a3fb5b6813a203eabea/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-rishabh-testing-20240501.193033/9a047fe7599a549042d06a3fb5b6813a203eabea/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-rishabh-testing-20240501.193033/9a047fe7599a549042d06a3fb5b6813a203eabea/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-05-03T21:07:39.377 DEBUG:teuthology.orchestra.run.smithi138:> set -ex 2024-05-03T21:07:39.377 DEBUG:teuthology.orchestra.run.smithi138:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-05-03T21:07:39.380 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://4.chacra.ceph.com/r/ceph/wip-rishabh-testing-20240501.193033/9a047fe7599a549042d06a3fb5b6813a203eabea/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-rishabh-testing-20240501.193033/9a047fe7599a549042d06a3fb5b6813a203eabea/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-rishabh-testing-20240501.193033/9a047fe7599a549042d06a3fb5b6813a203eabea/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-05-03T21:07:39.380 DEBUG:teuthology.orchestra.run.smithi082:> set -ex 2024-05-03T21:07:39.381 DEBUG:teuthology.orchestra.run.smithi082:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-05-03T21:07:39.387 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://4.chacra.ceph.com/r/ceph/wip-rishabh-testing-20240501.193033/9a047fe7599a549042d06a3fb5b6813a203eabea/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-rishabh-testing-20240501.193033/9a047fe7599a549042d06a3fb5b6813a203eabea/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-rishabh-testing-20240501.193033/9a047fe7599a549042d06a3fb5b6813a203eabea/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-05-03T21:07:39.388 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-05-03T21:07:39.388 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-05-03T21:07:39.401 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-05-03T21:07:39.401 DEBUG:teuthology.orchestra.run.smithi138:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/9a047fe7599a549042d06a3fb5b6813a203eabea/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-05-03T21:07:39.405 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-05-03T21:07:39.406 DEBUG:teuthology.orchestra.run.smithi082:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/9a047fe7599a549042d06a3fb5b6813a203eabea/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-05-03T21:07:39.413 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-05-03T21:07:39.413 DEBUG:teuthology.orchestra.run.smithi067:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/9a047fe7599a549042d06a3fb5b6813a203eabea/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-05-03T21:07:39.466 DEBUG:teuthology.orchestra.run.smithi138:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-05-03T21:07:39.471 DEBUG:teuthology.orchestra.run.smithi082:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-05-03T21:07:39.479 DEBUG:teuthology.orchestra.run.smithi067:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-05-03T21:07:39.539 DEBUG:teuthology.orchestra.run.smithi138:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-05-03T21:07:39.543 DEBUG:teuthology.orchestra.run.smithi082:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-05-03T21:07:39.553 DEBUG:teuthology.orchestra.run.smithi067:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-05-03T21:07:39.602 INFO:teuthology.orchestra.run.smithi138.stdout:check_obsoletes = 1 2024-05-03T21:07:39.604 DEBUG:teuthology.orchestra.run.smithi138:> sudo yum clean all 2024-05-03T21:07:39.606 INFO:teuthology.orchestra.run.smithi082.stdout:check_obsoletes = 1 2024-05-03T21:07:39.608 DEBUG:teuthology.orchestra.run.smithi082:> sudo yum clean all 2024-05-03T21:07:39.616 INFO:teuthology.orchestra.run.smithi067.stdout:check_obsoletes = 1 2024-05-03T21:07:39.618 DEBUG:teuthology.orchestra.run.smithi067:> sudo yum clean all 2024-05-03T21:07:39.827 INFO:teuthology.orchestra.run.smithi138.stdout:50 files removed 2024-05-03T21:07:39.831 INFO:teuthology.orchestra.run.smithi082.stdout:50 files removed 2024-05-03T21:07:39.852 INFO:teuthology.orchestra.run.smithi067.stdout:50 files removed 2024-05-03T21:07:39.855 DEBUG:teuthology.orchestra.run.smithi138:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-05-03T21:07:39.858 DEBUG:teuthology.orchestra.run.smithi082:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-05-03T21:07:39.879 DEBUG:teuthology.orchestra.run.smithi067:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-05-03T21:07:40.475 INFO:teuthology.orchestra.run.smithi082.stdout:ceph packages for x86_64 233 kB/s | 84 kB 00:00 2024-05-03T21:07:40.697 INFO:teuthology.orchestra.run.smithi138.stdout:ceph packages for x86_64 144 kB/s | 84 kB 00:00 2024-05-03T21:07:40.706 INFO:teuthology.orchestra.run.smithi067.stdout:ceph packages for x86_64 152 kB/s | 84 kB 00:00 2024-05-03T21:07:40.901 INFO:teuthology.orchestra.run.smithi082.stdout:ceph noarch packages 31 kB/s | 12 kB 00:00 2024-05-03T21:07:41.114 INFO:teuthology.orchestra.run.smithi138.stdout:ceph noarch packages 31 kB/s | 12 kB 00:00 2024-05-03T21:07:41.335 INFO:teuthology.orchestra.run.smithi082.stdout:ceph source packages 4.7 kB/s | 1.9 kB 00:00 2024-05-03T21:07:41.391 INFO:teuthology.orchestra.run.smithi067.stdout:ceph noarch packages 18 kB/s | 12 kB 00:00 2024-05-03T21:07:41.465 INFO:teuthology.orchestra.run.smithi138.stdout:ceph source packages 6.0 kB/s | 1.9 kB 00:00 2024-05-03T21:07:41.725 INFO:teuthology.orchestra.run.smithi067.stdout:ceph source packages 6.4 kB/s | 1.9 kB 00:00 2024-05-03T21:07:41.793 INFO:teuthology.orchestra.run.smithi082.stdout:CentOS Stream 9 - BaseOS 19 MB/s | 8.0 MB 00:00 2024-05-03T21:07:42.065 INFO:teuthology.orchestra.run.smithi138.stdout:CentOS Stream 9 - BaseOS 14 MB/s | 8.0 MB 00:00 2024-05-03T21:07:42.433 INFO:teuthology.orchestra.run.smithi067.stdout:CentOS Stream 9 - BaseOS 12 MB/s | 8.0 MB 00:00 2024-05-03T21:07:44.115 INFO:teuthology.orchestra.run.smithi082.stdout:CentOS Stream 9 - AppStream 15 MB/s | 19 MB 00:01 2024-05-03T21:07:44.293 INFO:teuthology.orchestra.run.smithi067.stdout:CentOS Stream 9 - AppStream 26 MB/s | 19 MB 00:00 2024-05-03T21:07:48.070 INFO:teuthology.orchestra.run.smithi082.stdout:CentOS Stream 9 - CRB 14 MB/s | 6.2 MB 00:00 2024-05-03T21:07:48.635 INFO:teuthology.orchestra.run.smithi067.stdout:CentOS Stream 9 - CRB 9.1 MB/s | 6.2 MB 00:00 2024-05-03T21:07:49.249 INFO:teuthology.orchestra.run.smithi138.stdout:CentOS Stream 9 - AppStream 3.1 MB/s | 19 MB 00:06 2024-05-03T21:07:50.070 INFO:teuthology.orchestra.run.smithi067.stdout:CentOS Stream 9 - Extras packages 63 kB/s | 16 kB 00:00 2024-05-03T21:07:50.312 INFO:teuthology.orchestra.run.smithi082.stdout:CentOS Stream 9 - Extras packages 15 kB/s | 16 kB 00:01 2024-05-03T21:07:51.305 INFO:teuthology.orchestra.run.smithi082.stdout:Extra Packages for Enterprise Linux 24 MB/s | 21 MB 00:00 2024-05-03T21:07:51.354 INFO:teuthology.orchestra.run.smithi067.stdout:Extra Packages for Enterprise Linux 18 MB/s | 21 MB 00:01 2024-05-03T21:07:53.533 INFO:teuthology.orchestra.run.smithi138.stdout:CentOS Stream 9 - CRB 8.5 MB/s | 6.2 MB 00:00 2024-05-03T21:07:54.976 INFO:teuthology.orchestra.run.smithi138.stdout:CentOS Stream 9 - Extras packages 73 kB/s | 16 kB 00:00 2024-05-03T21:07:56.069 INFO:teuthology.orchestra.run.smithi138.stdout:Extra Packages for Enterprise Linux 21 MB/s | 21 MB 00:00 2024-05-03T21:07:56.305 INFO:teuthology.orchestra.run.smithi082.stdout:lab-extras 27 kB/s | 1.7 kB 00:00 2024-05-03T21:07:56.621 INFO:teuthology.orchestra.run.smithi067.stdout:lab-extras 40 kB/s | 1.7 kB 00:00 2024-05-03T21:07:58.439 INFO:teuthology.orchestra.run.smithi082.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-05-03T21:07:58.439 INFO:teuthology.orchestra.run.smithi082.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-05-03T21:07:58.450 INFO:teuthology.orchestra.run.smithi082.stdout:Package NetworkManager-1:1.47.4-1.el9.x86_64 is already installed. 2024-05-03T21:07:58.451 INFO:teuthology.orchestra.run.smithi082.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-05-03T21:07:58.451 INFO:teuthology.orchestra.run.smithi082.stdout:Package util-linux-2.37.4-18.el9.x86_64 is already installed. 2024-05-03T21:07:58.452 INFO:teuthology.orchestra.run.smithi082.stdout:Package libacl-devel-2.3.1-4.el9.x86_64 is already installed. 2024-05-03T21:07:58.454 INFO:teuthology.orchestra.run.smithi082.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-05-03T21:07:58.456 INFO:teuthology.orchestra.run.smithi082.stdout:Package xfsdump-3.1.12-4.el9.x86_64 is already installed. 2024-05-03T21:07:58.457 INFO:teuthology.orchestra.run.smithi082.stdout:Package xfsprogs-6.3.0-1.el9.x86_64 is already installed. 2024-05-03T21:07:58.517 INFO:teuthology.orchestra.run.smithi082.stdout:Dependencies resolved. 2024-05-03T21:07:58.534 INFO:teuthology.orchestra.run.smithi082.stdout:======================================================================================= 2024-05-03T21:07:58.535 INFO:teuthology.orchestra.run.smithi082.stdout: Package Arch Version Repository Size 2024-05-03T21:07:58.535 INFO:teuthology.orchestra.run.smithi082.stdout:======================================================================================= 2024-05-03T21:07:58.535 INFO:teuthology.orchestra.run.smithi082.stdout:Installing: 2024-05-03T21:07:58.535 INFO:teuthology.orchestra.run.smithi082.stdout: bison x86_64 3.7.4-5.el9 appstream 943 k 2024-05-03T21:07:58.535 INFO:teuthology.orchestra.run.smithi082.stdout: ceph x86_64 2:19.0.0-3450.g9a047fe7.el9 ceph 6.6 k 2024-05-03T21:07:58.535 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-base x86_64 2:19.0.0-3450.g9a047fe7.el9 ceph 5.4 M 2024-05-03T21:07:58.535 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-fuse x86_64 2:19.0.0-3450.g9a047fe7.el9 ceph 891 k 2024-05-03T21:07:58.535 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-3450.g9a047fe7.el9 ceph 145 k 2024-05-03T21:07:58.535 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-mgr x86_64 2:19.0.0-3450.g9a047fe7.el9 ceph 1.5 M 2024-05-03T21:07:58.535 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-mgr-cephadm noarch 2:19.0.0-3450.g9a047fe7.el9 ceph-noarch 142 k 2024-05-03T21:07:58.535 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-mgr-dashboard noarch 2:19.0.0-3450.g9a047fe7.el9 ceph-noarch 3.5 M 2024-05-03T21:07:58.535 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-3450.g9a047fe7.el9 ceph-noarch 7.4 M 2024-05-03T21:07:58.535 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-mgr-rook noarch 2:19.0.0-3450.g9a047fe7.el9 ceph-noarch 49 k 2024-05-03T21:07:58.536 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-radosgw x86_64 2:19.0.0-3450.g9a047fe7.el9 ceph 11 M 2024-05-03T21:07:58.536 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-test x86_64 2:19.0.0-3450.g9a047fe7.el9 ceph 49 M 2024-05-03T21:07:58.536 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-volume noarch 2:19.0.0-3450.g9a047fe7.el9 ceph-noarch 273 k 2024-05-03T21:07:58.536 INFO:teuthology.orchestra.run.smithi082.stdout: cephadm noarch 2:19.0.0-3450.g9a047fe7.el9 ceph-noarch 763 k 2024-05-03T21:07:58.536 INFO:teuthology.orchestra.run.smithi082.stdout: cephfs-mirror x86_64 2:19.0.0-3450.g9a047fe7.el9 ceph 225 k 2024-05-03T21:07:58.536 INFO:teuthology.orchestra.run.smithi082.stdout: cephfs-shell x86_64 2:19.0.0-3450.g9a047fe7.el9 ceph 27 k 2024-05-03T21:07:58.536 INFO:teuthology.orchestra.run.smithi082.stdout: cephfs-top noarch 2:19.0.0-3450.g9a047fe7.el9 ceph-noarch 18 k 2024-05-03T21:07:58.536 INFO:teuthology.orchestra.run.smithi082.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 appstream 21 k 2024-05-03T21:07:58.536 INFO:teuthology.orchestra.run.smithi082.stdout: flex x86_64 2.6.4-9.el9 appstream 313 k 2024-05-03T21:07:58.536 INFO:teuthology.orchestra.run.smithi082.stdout: libaio-devel x86_64 0.3.111-13.el9 appstream 11 k 2024-05-03T21:07:58.536 INFO:teuthology.orchestra.run.smithi082.stdout: libcephfs-devel x86_64 2:19.0.0-3450.g9a047fe7.el9 ceph 32 k 2024-05-03T21:07:58.536 INFO:teuthology.orchestra.run.smithi082.stdout: libcephfs2 x86_64 2:19.0.0-3450.g9a047fe7.el9 ceph 747 k 2024-05-03T21:07:58.537 INFO:teuthology.orchestra.run.smithi082.stdout: librados-devel x86_64 2:19.0.0-3450.g9a047fe7.el9 ceph 127 k 2024-05-03T21:07:58.537 INFO:teuthology.orchestra.run.smithi082.stdout: libtool x86_64 2.4.6-45.el9 appstream 582 k 2024-05-03T21:07:58.537 INFO:teuthology.orchestra.run.smithi082.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-05-03T21:07:58.537 INFO:teuthology.orchestra.run.smithi082.stdout: openssl-devel x86_64 1:3.2.1-1.el9 appstream 4.4 M 2024-05-03T21:07:58.537 INFO:teuthology.orchestra.run.smithi082.stdout: postgresql x86_64 13.14-1.el9 appstream 1.6 M 2024-05-03T21:07:58.537 INFO:teuthology.orchestra.run.smithi082.stdout: postgresql-contrib x86_64 13.14-1.el9 appstream 911 k 2024-05-03T21:07:58.537 INFO:teuthology.orchestra.run.smithi082.stdout: postgresql-server x86_64 13.14-1.el9 appstream 5.8 M 2024-05-03T21:07:58.537 INFO:teuthology.orchestra.run.smithi082.stdout: python3-cephfs x86_64 2:19.0.0-3450.g9a047fe7.el9 ceph 161 k 2024-05-03T21:07:58.537 INFO:teuthology.orchestra.run.smithi082.stdout: python3-rados x86_64 2:19.0.0-3450.g9a047fe7.el9 ceph 320 k 2024-05-03T21:07:58.537 INFO:teuthology.orchestra.run.smithi082.stdout: python3-rbd x86_64 2:19.0.0-3450.g9a047fe7.el9 ceph 298 k 2024-05-03T21:07:58.537 INFO:teuthology.orchestra.run.smithi082.stdout: python3-rgw x86_64 2:19.0.0-3450.g9a047fe7.el9 ceph 100 k 2024-05-03T21:07:58.537 INFO:teuthology.orchestra.run.smithi082.stdout: rbd-fuse x86_64 2:19.0.0-3450.g9a047fe7.el9 ceph 85 k 2024-05-03T21:07:58.537 INFO:teuthology.orchestra.run.smithi082.stdout: rbd-mirror x86_64 2:19.0.0-3450.g9a047fe7.el9 ceph 3.1 M 2024-05-03T21:07:58.538 INFO:teuthology.orchestra.run.smithi082.stdout: rbd-nbd x86_64 2:19.0.0-3450.g9a047fe7.el9 ceph 170 k 2024-05-03T21:07:58.538 INFO:teuthology.orchestra.run.smithi082.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 appstream 94 k 2024-05-03T21:07:58.538 INFO:teuthology.orchestra.run.smithi082.stdout:Upgrading: 2024-05-03T21:07:58.538 INFO:teuthology.orchestra.run.smithi082.stdout: librados2 x86_64 2:19.0.0-3450.g9a047fe7.el9 ceph 3.4 M 2024-05-03T21:07:58.538 INFO:teuthology.orchestra.run.smithi082.stdout: librbd1 x86_64 2:19.0.0-3450.g9a047fe7.el9 ceph 3.1 M 2024-05-03T21:07:58.538 INFO:teuthology.orchestra.run.smithi082.stdout: openssl x86_64 1:3.2.1-1.el9 baseos 1.3 M 2024-05-03T21:07:58.538 INFO:teuthology.orchestra.run.smithi082.stdout: openssl-libs x86_64 1:3.2.1-1.el9 baseos 2.4 M 2024-05-03T21:07:58.538 INFO:teuthology.orchestra.run.smithi082.stdout:Installing dependencies: 2024-05-03T21:07:58.538 INFO:teuthology.orchestra.run.smithi082.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-05-03T21:07:58.538 INFO:teuthology.orchestra.run.smithi082.stdout: automake noarch 1.16.2-8.el9 appstream 689 k 2024-05-03T21:07:58.539 INFO:teuthology.orchestra.run.smithi082.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-05-03T21:07:58.539 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-common x86_64 2:19.0.0-3450.g9a047fe7.el9 ceph 21 M 2024-05-03T21:07:58.539 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-grafana-dashboards noarch 2:19.0.0-3450.g9a047fe7.el9 ceph-noarch 34 k 2024-05-03T21:07:58.539 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-mds x86_64 2:19.0.0-3450.g9a047fe7.el9 ceph 2.4 M 2024-05-03T21:07:58.539 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-mgr-modules-core noarch 2:19.0.0-3450.g9a047fe7.el9 ceph-noarch 274 k 2024-05-03T21:07:58.539 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-mon x86_64 2:19.0.0-3450.g9a047fe7.el9 ceph 4.7 M 2024-05-03T21:07:58.539 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-osd x86_64 2:19.0.0-3450.g9a047fe7.el9 ceph 17 M 2024-05-03T21:07:58.539 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-prometheus-alerts noarch 2:19.0.0-3450.g9a047fe7.el9 ceph-noarch 16 k 2024-05-03T21:07:58.539 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-selinux x86_64 2:19.0.0-3450.g9a047fe7.el9 ceph 25 k 2024-05-03T21:07:58.539 INFO:teuthology.orchestra.run.smithi082.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-05-03T21:07:58.539 INFO:teuthology.orchestra.run.smithi082.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-05-03T21:07:58.539 INFO:teuthology.orchestra.run.smithi082.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-05-03T21:07:58.539 INFO:teuthology.orchestra.run.smithi082.stdout: fuse x86_64 2.9.9-15.el9 baseos 80 k 2024-05-03T21:07:58.540 INFO:teuthology.orchestra.run.smithi082.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-05-03T21:07:58.540 INFO:teuthology.orchestra.run.smithi082.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-05-03T21:07:58.540 INFO:teuthology.orchestra.run.smithi082.stdout: libarrow x86_64 9.0.0-11.el9 epel 4.4 M 2024-05-03T21:07:58.540 INFO:teuthology.orchestra.run.smithi082.stdout: libarrow-doc noarch 9.0.0-11.el9 epel 26 k 2024-05-03T21:07:58.540 INFO:teuthology.orchestra.run.smithi082.stdout: libcephsqlite x86_64 2:19.0.0-3450.g9a047fe7.el9 ceph 163 k 2024-05-03T21:07:58.540 INFO:teuthology.orchestra.run.smithi082.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-05-03T21:07:58.540 INFO:teuthology.orchestra.run.smithi082.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-05-03T21:07:58.540 INFO:teuthology.orchestra.run.smithi082.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-05-03T21:07:58.540 INFO:teuthology.orchestra.run.smithi082.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-05-03T21:07:58.540 INFO:teuthology.orchestra.run.smithi082.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-05-03T21:07:58.540 INFO:teuthology.orchestra.run.smithi082.stdout: libradosstriper1 x86_64 2:19.0.0-3450.g9a047fe7.el9 ceph 500 k 2024-05-03T21:07:58.540 INFO:teuthology.orchestra.run.smithi082.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-05-03T21:07:58.540 INFO:teuthology.orchestra.run.smithi082.stdout: librgw2 x86_64 2:19.0.0-3450.g9a047fe7.el9 ceph 5.4 M 2024-05-03T21:07:58.540 INFO:teuthology.orchestra.run.smithi082.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-05-03T21:07:58.540 INFO:teuthology.orchestra.run.smithi082.stdout: libzstd-devel x86_64 1.5.1-2.el9 appstream 47 k 2024-05-03T21:07:58.541 INFO:teuthology.orchestra.run.smithi082.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-05-03T21:07:58.541 INFO:teuthology.orchestra.run.smithi082.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-05-03T21:07:58.541 INFO:teuthology.orchestra.run.smithi082.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-05-03T21:07:58.541 INFO:teuthology.orchestra.run.smithi082.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-05-03T21:07:58.541 INFO:teuthology.orchestra.run.smithi082.stdout: openblas x86_64 0.3.21-2.el9 appstream 35 k 2024-05-03T21:07:58.541 INFO:teuthology.orchestra.run.smithi082.stdout: openblas-openmp x86_64 0.3.21-2.el9 appstream 4.7 M 2024-05-03T21:07:58.541 INFO:teuthology.orchestra.run.smithi082.stdout: parquet-libs x86_64 9.0.0-11.el9 epel 839 k 2024-05-03T21:07:58.541 INFO:teuthology.orchestra.run.smithi082.stdout: postgresql-private-libs x86_64 13.14-1.el9 appstream 133 k 2024-05-03T21:07:58.541 INFO:teuthology.orchestra.run.smithi082.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-05-03T21:07:58.541 INFO:teuthology.orchestra.run.smithi082.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-05-03T21:07:58.541 INFO:teuthology.orchestra.run.smithi082.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-05-03T21:07:58.541 INFO:teuthology.orchestra.run.smithi082.stdout: python3-attrs noarch 20.3.0-7.el9 appstream 87 k 2024-05-03T21:07:58.541 INFO:teuthology.orchestra.run.smithi082.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-05-03T21:07:58.542 INFO:teuthology.orchestra.run.smithi082.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-05-03T21:07:58.542 INFO:teuthology.orchestra.run.smithi082.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-05-03T21:07:58.542 INFO:teuthology.orchestra.run.smithi082.stdout: python3-ceph-argparse x86_64 2:19.0.0-3450.g9a047fe7.el9 ceph 45 k 2024-05-03T21:07:58.542 INFO:teuthology.orchestra.run.smithi082.stdout: python3-ceph-common x86_64 2:19.0.0-3450.g9a047fe7.el9 ceph 135 k 2024-05-03T21:07:58.542 INFO:teuthology.orchestra.run.smithi082.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-05-03T21:07:58.542 INFO:teuthology.orchestra.run.smithi082.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-05-03T21:07:58.542 INFO:teuthology.orchestra.run.smithi082.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-05-03T21:07:58.542 INFO:teuthology.orchestra.run.smithi082.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-05-03T21:07:58.542 INFO:teuthology.orchestra.run.smithi082.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-05-03T21:07:58.542 INFO:teuthology.orchestra.run.smithi082.stdout: python3-colorama noarch 0.4.4-12.el9 epel 35 k 2024-05-03T21:07:58.542 INFO:teuthology.orchestra.run.smithi082.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-05-03T21:07:58.542 INFO:teuthology.orchestra.run.smithi082.stdout: python3-devel x86_64 3.9.18-3.el9 appstream 244 k 2024-05-03T21:07:58.543 INFO:teuthology.orchestra.run.smithi082.stdout: python3-google-auth noarch 1:2.29.0-1.el9 epel 206 k 2024-05-03T21:07:58.543 INFO:teuthology.orchestra.run.smithi082.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-05-03T21:07:58.543 INFO:teuthology.orchestra.run.smithi082.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-05-03T21:07:58.543 INFO:teuthology.orchestra.run.smithi082.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-05-03T21:07:58.543 INFO:teuthology.orchestra.run.smithi082.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-05-03T21:07:58.543 INFO:teuthology.orchestra.run.smithi082.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-05-03T21:07:58.543 INFO:teuthology.orchestra.run.smithi082.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-05-03T21:07:58.543 INFO:teuthology.orchestra.run.smithi082.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-05-03T21:07:58.543 INFO:teuthology.orchestra.run.smithi082.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-05-03T21:07:58.543 INFO:teuthology.orchestra.run.smithi082.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-05-03T21:07:58.543 INFO:teuthology.orchestra.run.smithi082.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-05-03T21:07:58.543 INFO:teuthology.orchestra.run.smithi082.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-05-03T21:07:58.543 INFO:teuthology.orchestra.run.smithi082.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-05-03T21:07:58.543 INFO:teuthology.orchestra.run.smithi082.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-05-03T21:07:58.543 INFO:teuthology.orchestra.run.smithi082.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-05-03T21:07:58.544 INFO:teuthology.orchestra.run.smithi082.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-05-03T21:07:58.544 INFO:teuthology.orchestra.run.smithi082.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-05-03T21:07:58.544 INFO:teuthology.orchestra.run.smithi082.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-05-03T21:07:58.544 INFO:teuthology.orchestra.run.smithi082.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-05-03T21:07:58.544 INFO:teuthology.orchestra.run.smithi082.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-05-03T21:07:58.544 INFO:teuthology.orchestra.run.smithi082.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-05-03T21:07:58.544 INFO:teuthology.orchestra.run.smithi082.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-05-03T21:07:58.544 INFO:teuthology.orchestra.run.smithi082.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-05-03T21:07:58.544 INFO:teuthology.orchestra.run.smithi082.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-05-03T21:07:58.544 INFO:teuthology.orchestra.run.smithi082.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-05-03T21:07:58.544 INFO:teuthology.orchestra.run.smithi082.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-05-03T21:07:58.544 INFO:teuthology.orchestra.run.smithi082.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-05-03T21:07:58.544 INFO:teuthology.orchestra.run.smithi082.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-05-03T21:07:58.544 INFO:teuthology.orchestra.run.smithi082.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-05-03T21:07:58.544 INFO:teuthology.orchestra.run.smithi082.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-05-03T21:07:58.545 INFO:teuthology.orchestra.run.smithi082.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-05-03T21:07:58.545 INFO:teuthology.orchestra.run.smithi082.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-05-03T21:07:58.545 INFO:teuthology.orchestra.run.smithi082.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-05-03T21:07:58.545 INFO:teuthology.orchestra.run.smithi082.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-05-03T21:07:58.545 INFO:teuthology.orchestra.run.smithi082.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-05-03T21:07:58.545 INFO:teuthology.orchestra.run.smithi082.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-05-03T21:07:58.545 INFO:teuthology.orchestra.run.smithi082.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-05-03T21:07:58.545 INFO:teuthology.orchestra.run.smithi082.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-05-03T21:07:58.545 INFO:teuthology.orchestra.run.smithi082.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-05-03T21:07:58.545 INFO:teuthology.orchestra.run.smithi082.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-05-03T21:07:58.545 INFO:teuthology.orchestra.run.smithi082.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-05-03T21:07:58.545 INFO:teuthology.orchestra.run.smithi082.stdout: python3-urllib3 noarch 1.26.5-5.el9 baseos 215 k 2024-05-03T21:07:58.545 INFO:teuthology.orchestra.run.smithi082.stdout: python3-wcwidth noarch 0.2.5-8.el9 appstream 44 k 2024-05-03T21:07:58.545 INFO:teuthology.orchestra.run.smithi082.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-05-03T21:07:58.545 INFO:teuthology.orchestra.run.smithi082.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-05-03T21:07:58.546 INFO:teuthology.orchestra.run.smithi082.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-05-03T21:07:58.546 INFO:teuthology.orchestra.run.smithi082.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-05-03T21:07:58.546 INFO:teuthology.orchestra.run.smithi082.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-05-03T21:07:58.546 INFO:teuthology.orchestra.run.smithi082.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-05-03T21:07:58.546 INFO:teuthology.orchestra.run.smithi082.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-05-03T21:07:58.546 INFO:teuthology.orchestra.run.smithi082.stdout: socat x86_64 1.7.4.1-5.el9 appstream 305 k 2024-05-03T21:07:58.546 INFO:teuthology.orchestra.run.smithi082.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-05-03T21:07:58.546 INFO:teuthology.orchestra.run.smithi082.stdout: uuid x86_64 1.6.2-55.el9 appstream 58 k 2024-05-03T21:07:58.546 INFO:teuthology.orchestra.run.smithi082.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-05-03T21:07:58.546 INFO:teuthology.orchestra.run.smithi082.stdout: zlib-devel x86_64 1.2.11-41.el9 appstream 45 k 2024-05-03T21:07:58.546 INFO:teuthology.orchestra.run.smithi082.stdout:Installing weak dependencies: 2024-05-03T21:07:58.546 INFO:teuthology.orchestra.run.smithi082.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-05-03T21:07:58.546 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-05-03T21:07:58.547 INFO:teuthology.orchestra.run.smithi082.stdout:Transaction Summary 2024-05-03T21:07:58.547 INFO:teuthology.orchestra.run.smithi082.stdout:======================================================================================= 2024-05-03T21:07:58.547 INFO:teuthology.orchestra.run.smithi082.stdout:Install 148 Packages 2024-05-03T21:07:58.547 INFO:teuthology.orchestra.run.smithi082.stdout:Upgrade 4 Packages 2024-05-03T21:07:58.547 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-05-03T21:07:58.547 INFO:teuthology.orchestra.run.smithi082.stdout:Total download size: 222 M 2024-05-03T21:07:58.548 INFO:teuthology.orchestra.run.smithi082.stdout:Downloading Packages: 2024-05-03T21:07:58.848 INFO:teuthology.orchestra.run.smithi067.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-05-03T21:07:58.849 INFO:teuthology.orchestra.run.smithi067.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-05-03T21:07:58.860 INFO:teuthology.orchestra.run.smithi067.stdout:Package NetworkManager-1:1.47.4-1.el9.x86_64 is already installed. 2024-05-03T21:07:58.861 INFO:teuthology.orchestra.run.smithi067.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-05-03T21:07:58.861 INFO:teuthology.orchestra.run.smithi067.stdout:Package util-linux-2.37.4-18.el9.x86_64 is already installed. 2024-05-03T21:07:58.862 INFO:teuthology.orchestra.run.smithi067.stdout:Package libacl-devel-2.3.1-4.el9.x86_64 is already installed. 2024-05-03T21:07:58.864 INFO:teuthology.orchestra.run.smithi067.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-05-03T21:07:58.866 INFO:teuthology.orchestra.run.smithi067.stdout:Package xfsdump-3.1.12-4.el9.x86_64 is already installed. 2024-05-03T21:07:58.867 INFO:teuthology.orchestra.run.smithi067.stdout:Package xfsprogs-6.3.0-1.el9.x86_64 is already installed. 2024-05-03T21:07:58.931 INFO:teuthology.orchestra.run.smithi067.stdout:Dependencies resolved. 2024-05-03T21:07:58.941 INFO:teuthology.orchestra.run.smithi067.stdout:======================================================================================= 2024-05-03T21:07:58.941 INFO:teuthology.orchestra.run.smithi067.stdout: Package Arch Version Repository Size 2024-05-03T21:07:58.941 INFO:teuthology.orchestra.run.smithi067.stdout:======================================================================================= 2024-05-03T21:07:58.941 INFO:teuthology.orchestra.run.smithi067.stdout:Installing: 2024-05-03T21:07:58.941 INFO:teuthology.orchestra.run.smithi067.stdout: bison x86_64 3.7.4-5.el9 appstream 943 k 2024-05-03T21:07:58.941 INFO:teuthology.orchestra.run.smithi067.stdout: ceph x86_64 2:19.0.0-3450.g9a047fe7.el9 ceph 6.6 k 2024-05-03T21:07:58.942 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-base x86_64 2:19.0.0-3450.g9a047fe7.el9 ceph 5.4 M 2024-05-03T21:07:58.942 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-fuse x86_64 2:19.0.0-3450.g9a047fe7.el9 ceph 891 k 2024-05-03T21:07:58.942 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-3450.g9a047fe7.el9 ceph 145 k 2024-05-03T21:07:58.942 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-mgr x86_64 2:19.0.0-3450.g9a047fe7.el9 ceph 1.5 M 2024-05-03T21:07:58.942 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-mgr-cephadm noarch 2:19.0.0-3450.g9a047fe7.el9 ceph-noarch 142 k 2024-05-03T21:07:58.942 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-mgr-dashboard noarch 2:19.0.0-3450.g9a047fe7.el9 ceph-noarch 3.5 M 2024-05-03T21:07:58.942 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-3450.g9a047fe7.el9 ceph-noarch 7.4 M 2024-05-03T21:07:58.942 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-mgr-rook noarch 2:19.0.0-3450.g9a047fe7.el9 ceph-noarch 49 k 2024-05-03T21:07:58.942 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-radosgw x86_64 2:19.0.0-3450.g9a047fe7.el9 ceph 11 M 2024-05-03T21:07:58.942 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-test x86_64 2:19.0.0-3450.g9a047fe7.el9 ceph 49 M 2024-05-03T21:07:58.943 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-volume noarch 2:19.0.0-3450.g9a047fe7.el9 ceph-noarch 273 k 2024-05-03T21:07:58.943 INFO:teuthology.orchestra.run.smithi067.stdout: cephadm noarch 2:19.0.0-3450.g9a047fe7.el9 ceph-noarch 763 k 2024-05-03T21:07:58.943 INFO:teuthology.orchestra.run.smithi067.stdout: cephfs-mirror x86_64 2:19.0.0-3450.g9a047fe7.el9 ceph 225 k 2024-05-03T21:07:58.943 INFO:teuthology.orchestra.run.smithi067.stdout: cephfs-shell x86_64 2:19.0.0-3450.g9a047fe7.el9 ceph 27 k 2024-05-03T21:07:58.943 INFO:teuthology.orchestra.run.smithi067.stdout: cephfs-top noarch 2:19.0.0-3450.g9a047fe7.el9 ceph-noarch 18 k 2024-05-03T21:07:58.943 INFO:teuthology.orchestra.run.smithi067.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 appstream 21 k 2024-05-03T21:07:58.943 INFO:teuthology.orchestra.run.smithi067.stdout: flex x86_64 2.6.4-9.el9 appstream 313 k 2024-05-03T21:07:58.943 INFO:teuthology.orchestra.run.smithi067.stdout: libaio-devel x86_64 0.3.111-13.el9 appstream 11 k 2024-05-03T21:07:58.943 INFO:teuthology.orchestra.run.smithi067.stdout: libcephfs-devel x86_64 2:19.0.0-3450.g9a047fe7.el9 ceph 32 k 2024-05-03T21:07:58.943 INFO:teuthology.orchestra.run.smithi067.stdout: libcephfs2 x86_64 2:19.0.0-3450.g9a047fe7.el9 ceph 747 k 2024-05-03T21:07:58.943 INFO:teuthology.orchestra.run.smithi067.stdout: librados-devel x86_64 2:19.0.0-3450.g9a047fe7.el9 ceph 127 k 2024-05-03T21:07:58.943 INFO:teuthology.orchestra.run.smithi067.stdout: libtool x86_64 2.4.6-45.el9 appstream 582 k 2024-05-03T21:07:58.943 INFO:teuthology.orchestra.run.smithi067.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-05-03T21:07:58.943 INFO:teuthology.orchestra.run.smithi067.stdout: openssl-devel x86_64 1:3.2.1-1.el9 appstream 4.4 M 2024-05-03T21:07:58.943 INFO:teuthology.orchestra.run.smithi067.stdout: postgresql x86_64 13.14-1.el9 appstream 1.6 M 2024-05-03T21:07:58.944 INFO:teuthology.orchestra.run.smithi067.stdout: postgresql-contrib x86_64 13.14-1.el9 appstream 911 k 2024-05-03T21:07:58.944 INFO:teuthology.orchestra.run.smithi067.stdout: postgresql-server x86_64 13.14-1.el9 appstream 5.8 M 2024-05-03T21:07:58.944 INFO:teuthology.orchestra.run.smithi067.stdout: python3-cephfs x86_64 2:19.0.0-3450.g9a047fe7.el9 ceph 161 k 2024-05-03T21:07:58.944 INFO:teuthology.orchestra.run.smithi067.stdout: python3-rados x86_64 2:19.0.0-3450.g9a047fe7.el9 ceph 320 k 2024-05-03T21:07:58.944 INFO:teuthology.orchestra.run.smithi067.stdout: python3-rbd x86_64 2:19.0.0-3450.g9a047fe7.el9 ceph 298 k 2024-05-03T21:07:58.944 INFO:teuthology.orchestra.run.smithi067.stdout: python3-rgw x86_64 2:19.0.0-3450.g9a047fe7.el9 ceph 100 k 2024-05-03T21:07:58.944 INFO:teuthology.orchestra.run.smithi067.stdout: rbd-fuse x86_64 2:19.0.0-3450.g9a047fe7.el9 ceph 85 k 2024-05-03T21:07:58.944 INFO:teuthology.orchestra.run.smithi067.stdout: rbd-mirror x86_64 2:19.0.0-3450.g9a047fe7.el9 ceph 3.1 M 2024-05-03T21:07:58.944 INFO:teuthology.orchestra.run.smithi067.stdout: rbd-nbd x86_64 2:19.0.0-3450.g9a047fe7.el9 ceph 170 k 2024-05-03T21:07:58.944 INFO:teuthology.orchestra.run.smithi067.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 appstream 94 k 2024-05-03T21:07:58.945 INFO:teuthology.orchestra.run.smithi067.stdout:Upgrading: 2024-05-03T21:07:58.945 INFO:teuthology.orchestra.run.smithi067.stdout: librados2 x86_64 2:19.0.0-3450.g9a047fe7.el9 ceph 3.4 M 2024-05-03T21:07:58.945 INFO:teuthology.orchestra.run.smithi067.stdout: librbd1 x86_64 2:19.0.0-3450.g9a047fe7.el9 ceph 3.1 M 2024-05-03T21:07:58.945 INFO:teuthology.orchestra.run.smithi067.stdout: openssl x86_64 1:3.2.1-1.el9 baseos 1.3 M 2024-05-03T21:07:58.945 INFO:teuthology.orchestra.run.smithi067.stdout: openssl-libs x86_64 1:3.2.1-1.el9 baseos 2.4 M 2024-05-03T21:07:58.945 INFO:teuthology.orchestra.run.smithi067.stdout:Installing dependencies: 2024-05-03T21:07:58.945 INFO:teuthology.orchestra.run.smithi067.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-05-03T21:07:58.945 INFO:teuthology.orchestra.run.smithi067.stdout: automake noarch 1.16.2-8.el9 appstream 689 k 2024-05-03T21:07:58.945 INFO:teuthology.orchestra.run.smithi067.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-05-03T21:07:58.945 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-common x86_64 2:19.0.0-3450.g9a047fe7.el9 ceph 21 M 2024-05-03T21:07:58.945 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-grafana-dashboards noarch 2:19.0.0-3450.g9a047fe7.el9 ceph-noarch 34 k 2024-05-03T21:07:58.945 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-mds x86_64 2:19.0.0-3450.g9a047fe7.el9 ceph 2.4 M 2024-05-03T21:07:58.946 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-mgr-modules-core noarch 2:19.0.0-3450.g9a047fe7.el9 ceph-noarch 274 k 2024-05-03T21:07:58.946 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-mon x86_64 2:19.0.0-3450.g9a047fe7.el9 ceph 4.7 M 2024-05-03T21:07:58.946 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-osd x86_64 2:19.0.0-3450.g9a047fe7.el9 ceph 17 M 2024-05-03T21:07:58.946 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-prometheus-alerts noarch 2:19.0.0-3450.g9a047fe7.el9 ceph-noarch 16 k 2024-05-03T21:07:58.946 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-selinux x86_64 2:19.0.0-3450.g9a047fe7.el9 ceph 25 k 2024-05-03T21:07:58.946 INFO:teuthology.orchestra.run.smithi067.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-05-03T21:07:58.946 INFO:teuthology.orchestra.run.smithi067.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-05-03T21:07:58.946 INFO:teuthology.orchestra.run.smithi067.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-05-03T21:07:58.946 INFO:teuthology.orchestra.run.smithi067.stdout: fuse x86_64 2.9.9-15.el9 baseos 80 k 2024-05-03T21:07:58.946 INFO:teuthology.orchestra.run.smithi067.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-05-03T21:07:58.946 INFO:teuthology.orchestra.run.smithi067.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-05-03T21:07:58.946 INFO:teuthology.orchestra.run.smithi067.stdout: libarrow x86_64 9.0.0-11.el9 epel 4.4 M 2024-05-03T21:07:58.946 INFO:teuthology.orchestra.run.smithi067.stdout: libarrow-doc noarch 9.0.0-11.el9 epel 26 k 2024-05-03T21:07:58.947 INFO:teuthology.orchestra.run.smithi067.stdout: libcephsqlite x86_64 2:19.0.0-3450.g9a047fe7.el9 ceph 163 k 2024-05-03T21:07:58.947 INFO:teuthology.orchestra.run.smithi067.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-05-03T21:07:58.947 INFO:teuthology.orchestra.run.smithi067.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-05-03T21:07:58.947 INFO:teuthology.orchestra.run.smithi067.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-05-03T21:07:58.947 INFO:teuthology.orchestra.run.smithi067.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-05-03T21:07:58.947 INFO:teuthology.orchestra.run.smithi067.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-05-03T21:07:58.947 INFO:teuthology.orchestra.run.smithi067.stdout: libradosstriper1 x86_64 2:19.0.0-3450.g9a047fe7.el9 ceph 500 k 2024-05-03T21:07:58.947 INFO:teuthology.orchestra.run.smithi067.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-05-03T21:07:58.948 INFO:teuthology.orchestra.run.smithi067.stdout: librgw2 x86_64 2:19.0.0-3450.g9a047fe7.el9 ceph 5.4 M 2024-05-03T21:07:58.948 INFO:teuthology.orchestra.run.smithi067.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-05-03T21:07:58.948 INFO:teuthology.orchestra.run.smithi067.stdout: libzstd-devel x86_64 1.5.1-2.el9 appstream 47 k 2024-05-03T21:07:58.948 INFO:teuthology.orchestra.run.smithi067.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-05-03T21:07:58.948 INFO:teuthology.orchestra.run.smithi067.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-05-03T21:07:58.948 INFO:teuthology.orchestra.run.smithi067.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-05-03T21:07:58.948 INFO:teuthology.orchestra.run.smithi067.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-05-03T21:07:58.948 INFO:teuthology.orchestra.run.smithi067.stdout: openblas x86_64 0.3.21-2.el9 appstream 35 k 2024-05-03T21:07:58.948 INFO:teuthology.orchestra.run.smithi067.stdout: openblas-openmp x86_64 0.3.21-2.el9 appstream 4.7 M 2024-05-03T21:07:58.948 INFO:teuthology.orchestra.run.smithi067.stdout: parquet-libs x86_64 9.0.0-11.el9 epel 839 k 2024-05-03T21:07:58.948 INFO:teuthology.orchestra.run.smithi067.stdout: postgresql-private-libs x86_64 13.14-1.el9 appstream 133 k 2024-05-03T21:07:58.948 INFO:teuthology.orchestra.run.smithi067.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-05-03T21:07:58.948 INFO:teuthology.orchestra.run.smithi067.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-05-03T21:07:58.948 INFO:teuthology.orchestra.run.smithi067.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-05-03T21:07:58.949 INFO:teuthology.orchestra.run.smithi067.stdout: python3-attrs noarch 20.3.0-7.el9 appstream 87 k 2024-05-03T21:07:58.949 INFO:teuthology.orchestra.run.smithi067.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-05-03T21:07:58.949 INFO:teuthology.orchestra.run.smithi067.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-05-03T21:07:58.949 INFO:teuthology.orchestra.run.smithi067.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-05-03T21:07:58.949 INFO:teuthology.orchestra.run.smithi067.stdout: python3-ceph-argparse x86_64 2:19.0.0-3450.g9a047fe7.el9 ceph 45 k 2024-05-03T21:07:58.949 INFO:teuthology.orchestra.run.smithi067.stdout: python3-ceph-common x86_64 2:19.0.0-3450.g9a047fe7.el9 ceph 135 k 2024-05-03T21:07:58.949 INFO:teuthology.orchestra.run.smithi067.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-05-03T21:07:58.949 INFO:teuthology.orchestra.run.smithi067.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-05-03T21:07:58.949 INFO:teuthology.orchestra.run.smithi067.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-05-03T21:07:58.949 INFO:teuthology.orchestra.run.smithi067.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-05-03T21:07:58.950 INFO:teuthology.orchestra.run.smithi067.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-05-03T21:07:58.950 INFO:teuthology.orchestra.run.smithi067.stdout: python3-colorama noarch 0.4.4-12.el9 epel 35 k 2024-05-03T21:07:58.950 INFO:teuthology.orchestra.run.smithi067.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-05-03T21:07:58.950 INFO:teuthology.orchestra.run.smithi067.stdout: python3-devel x86_64 3.9.18-3.el9 appstream 244 k 2024-05-03T21:07:58.950 INFO:teuthology.orchestra.run.smithi067.stdout: python3-google-auth noarch 1:2.29.0-1.el9 epel 206 k 2024-05-03T21:07:58.950 INFO:teuthology.orchestra.run.smithi067.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-05-03T21:07:58.950 INFO:teuthology.orchestra.run.smithi067.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-05-03T21:07:58.950 INFO:teuthology.orchestra.run.smithi067.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-05-03T21:07:58.950 INFO:teuthology.orchestra.run.smithi067.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-05-03T21:07:58.950 INFO:teuthology.orchestra.run.smithi067.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-05-03T21:07:58.951 INFO:teuthology.orchestra.run.smithi067.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-05-03T21:07:58.951 INFO:teuthology.orchestra.run.smithi067.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-05-03T21:07:58.951 INFO:teuthology.orchestra.run.smithi067.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-05-03T21:07:58.951 INFO:teuthology.orchestra.run.smithi067.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-05-03T21:07:58.951 INFO:teuthology.orchestra.run.smithi067.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-05-03T21:07:58.951 INFO:teuthology.orchestra.run.smithi067.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-05-03T21:07:58.951 INFO:teuthology.orchestra.run.smithi067.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-05-03T21:07:58.951 INFO:teuthology.orchestra.run.smithi067.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-05-03T21:07:58.951 INFO:teuthology.orchestra.run.smithi067.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-05-03T21:07:58.951 INFO:teuthology.orchestra.run.smithi067.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-05-03T21:07:58.951 INFO:teuthology.orchestra.run.smithi067.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-05-03T21:07:58.951 INFO:teuthology.orchestra.run.smithi067.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-05-03T21:07:58.951 INFO:teuthology.orchestra.run.smithi067.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-05-03T21:07:58.952 INFO:teuthology.orchestra.run.smithi067.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-05-03T21:07:58.952 INFO:teuthology.orchestra.run.smithi067.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-05-03T21:07:58.952 INFO:teuthology.orchestra.run.smithi067.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-05-03T21:07:58.952 INFO:teuthology.orchestra.run.smithi067.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-05-03T21:07:58.952 INFO:teuthology.orchestra.run.smithi067.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-05-03T21:07:58.952 INFO:teuthology.orchestra.run.smithi067.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-05-03T21:07:58.952 INFO:teuthology.orchestra.run.smithi067.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-05-03T21:07:58.952 INFO:teuthology.orchestra.run.smithi067.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-05-03T21:07:58.952 INFO:teuthology.orchestra.run.smithi067.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-05-03T21:07:58.952 INFO:teuthology.orchestra.run.smithi067.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-05-03T21:07:58.952 INFO:teuthology.orchestra.run.smithi067.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-05-03T21:07:58.952 INFO:teuthology.orchestra.run.smithi067.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-05-03T21:07:58.953 INFO:teuthology.orchestra.run.smithi067.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-05-03T21:07:58.953 INFO:teuthology.orchestra.run.smithi067.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-05-03T21:07:58.953 INFO:teuthology.orchestra.run.smithi067.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-05-03T21:07:58.953 INFO:teuthology.orchestra.run.smithi067.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-05-03T21:07:58.953 INFO:teuthology.orchestra.run.smithi067.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-05-03T21:07:58.953 INFO:teuthology.orchestra.run.smithi067.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-05-03T21:07:58.953 INFO:teuthology.orchestra.run.smithi067.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-05-03T21:07:58.953 INFO:teuthology.orchestra.run.smithi067.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-05-03T21:07:58.953 INFO:teuthology.orchestra.run.smithi067.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-05-03T21:07:58.953 INFO:teuthology.orchestra.run.smithi067.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-05-03T21:07:58.953 INFO:teuthology.orchestra.run.smithi067.stdout: python3-urllib3 noarch 1.26.5-5.el9 baseos 215 k 2024-05-03T21:07:58.953 INFO:teuthology.orchestra.run.smithi067.stdout: python3-wcwidth noarch 0.2.5-8.el9 appstream 44 k 2024-05-03T21:07:58.954 INFO:teuthology.orchestra.run.smithi067.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-05-03T21:07:58.954 INFO:teuthology.orchestra.run.smithi067.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-05-03T21:07:58.954 INFO:teuthology.orchestra.run.smithi067.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-05-03T21:07:58.954 INFO:teuthology.orchestra.run.smithi067.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-05-03T21:07:58.954 INFO:teuthology.orchestra.run.smithi067.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-05-03T21:07:58.954 INFO:teuthology.orchestra.run.smithi067.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-05-03T21:07:58.954 INFO:teuthology.orchestra.run.smithi067.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-05-03T21:07:58.954 INFO:teuthology.orchestra.run.smithi067.stdout: socat x86_64 1.7.4.1-5.el9 appstream 305 k 2024-05-03T21:07:58.954 INFO:teuthology.orchestra.run.smithi067.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-05-03T21:07:58.954 INFO:teuthology.orchestra.run.smithi067.stdout: uuid x86_64 1.6.2-55.el9 appstream 58 k 2024-05-03T21:07:58.954 INFO:teuthology.orchestra.run.smithi067.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-05-03T21:07:58.954 INFO:teuthology.orchestra.run.smithi067.stdout: zlib-devel x86_64 1.2.11-41.el9 appstream 45 k 2024-05-03T21:07:58.954 INFO:teuthology.orchestra.run.smithi067.stdout:Installing weak dependencies: 2024-05-03T21:07:58.955 INFO:teuthology.orchestra.run.smithi067.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-05-03T21:07:58.955 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-05-03T21:07:58.955 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction Summary 2024-05-03T21:07:58.955 INFO:teuthology.orchestra.run.smithi067.stdout:======================================================================================= 2024-05-03T21:07:58.955 INFO:teuthology.orchestra.run.smithi067.stdout:Install 148 Packages 2024-05-03T21:07:58.955 INFO:teuthology.orchestra.run.smithi067.stdout:Upgrade 4 Packages 2024-05-03T21:07:58.955 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-05-03T21:07:58.956 INFO:teuthology.orchestra.run.smithi067.stdout:Total download size: 222 M 2024-05-03T21:07:58.956 INFO:teuthology.orchestra.run.smithi067.stdout:Downloading Packages: 2024-05-03T21:07:59.664 INFO:teuthology.orchestra.run.smithi067.stdout:(1/152): ceph-19.0.0-3450.g9a047fe7.el9.x86_64. 30 kB/s | 6.6 kB 00:00 2024-05-03T21:07:59.689 INFO:teuthology.orchestra.run.smithi082.stdout:(1/152): ceph-19.0.0-3450.g9a047fe7.el9.x86_64. 29 kB/s | 6.6 kB 00:00 2024-05-03T21:08:00.523 INFO:teuthology.orchestra.run.smithi082.stdout:(2/152): ceph-fuse-19.0.0-3450.g9a047fe7.el9.x8 1.0 MB/s | 891 kB 00:00 2024-05-03T21:08:00.631 INFO:teuthology.orchestra.run.smithi067.stdout:(2/152): ceph-fuse-19.0.0-3450.g9a047fe7.el9.x8 922 kB/s | 891 kB 00:00 2024-05-03T21:08:00.673 INFO:teuthology.orchestra.run.smithi082.stdout:(3/152): ceph-immutable-object-cache-19.0.0-345 965 kB/s | 145 kB 00:00 2024-05-03T21:08:00.723 INFO:teuthology.orchestra.run.smithi067.stdout:(3/152): ceph-immutable-object-cache-19.0.0-345 1.5 MB/s | 145 kB 00:00 2024-05-03T21:08:01.211 INFO:teuthology.orchestra.run.smithi138.stdout:lab-extras 13 kB/s | 1.7 kB 00:00 2024-05-03T21:08:01.565 INFO:teuthology.orchestra.run.smithi082.stdout:(4/152): ceph-mds-19.0.0-3450.g9a047fe7.el9.x86 2.7 MB/s | 2.4 MB 00:00 2024-05-03T21:08:01.765 INFO:teuthology.orchestra.run.smithi067.stdout:(4/152): ceph-mds-19.0.0-3450.g9a047fe7.el9.x86 2.3 MB/s | 2.4 MB 00:01 2024-05-03T21:08:01.882 INFO:teuthology.orchestra.run.smithi082.stdout:(5/152): ceph-base-19.0.0-3450.g9a047fe7.el9.x8 2.2 MB/s | 5.4 MB 00:02 2024-05-03T21:08:02.207 INFO:teuthology.orchestra.run.smithi067.stdout:(5/152): ceph-mgr-19.0.0-3450.g9a047fe7.el9.x86 3.4 MB/s | 1.5 MB 00:00 2024-05-03T21:08:02.265 INFO:teuthology.orchestra.run.smithi082.stdout:(6/152): ceph-mgr-19.0.0-3450.g9a047fe7.el9.x86 2.2 MB/s | 1.5 MB 00:00 2024-05-03T21:08:02.707 INFO:teuthology.orchestra.run.smithi082.stdout:(7/152): ceph-mon-19.0.0-3450.g9a047fe7.el9.x86 5.7 MB/s | 4.7 MB 00:00 2024-05-03T21:08:03.363 INFO:teuthology.orchestra.run.smithi138.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-05-03T21:08:03.364 INFO:teuthology.orchestra.run.smithi138.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-05-03T21:08:03.374 INFO:teuthology.orchestra.run.smithi138.stdout:Package NetworkManager-1:1.47.4-1.el9.x86_64 is already installed. 2024-05-03T21:08:03.375 INFO:teuthology.orchestra.run.smithi138.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-05-03T21:08:03.376 INFO:teuthology.orchestra.run.smithi138.stdout:Package util-linux-2.37.4-18.el9.x86_64 is already installed. 2024-05-03T21:08:03.376 INFO:teuthology.orchestra.run.smithi138.stdout:Package libacl-devel-2.3.1-4.el9.x86_64 is already installed. 2024-05-03T21:08:03.378 INFO:teuthology.orchestra.run.smithi138.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-05-03T21:08:03.380 INFO:teuthology.orchestra.run.smithi138.stdout:Package xfsdump-3.1.12-4.el9.x86_64 is already installed. 2024-05-03T21:08:03.381 INFO:teuthology.orchestra.run.smithi138.stdout:Package xfsprogs-6.3.0-1.el9.x86_64 is already installed. 2024-05-03T21:08:03.441 INFO:teuthology.orchestra.run.smithi067.stdout:(6/152): ceph-mon-19.0.0-3450.g9a047fe7.el9.x86 3.8 MB/s | 4.7 MB 00:01 2024-05-03T21:08:03.442 INFO:teuthology.orchestra.run.smithi138.stdout:Dependencies resolved. 2024-05-03T21:08:03.460 INFO:teuthology.orchestra.run.smithi138.stdout:======================================================================================= 2024-05-03T21:08:03.460 INFO:teuthology.orchestra.run.smithi138.stdout: Package Arch Version Repository Size 2024-05-03T21:08:03.461 INFO:teuthology.orchestra.run.smithi138.stdout:======================================================================================= 2024-05-03T21:08:03.461 INFO:teuthology.orchestra.run.smithi138.stdout:Installing: 2024-05-03T21:08:03.461 INFO:teuthology.orchestra.run.smithi138.stdout: bison x86_64 3.7.4-5.el9 appstream 943 k 2024-05-03T21:08:03.461 INFO:teuthology.orchestra.run.smithi138.stdout: ceph x86_64 2:19.0.0-3450.g9a047fe7.el9 ceph 6.6 k 2024-05-03T21:08:03.461 INFO:teuthology.orchestra.run.smithi138.stdout: ceph-base x86_64 2:19.0.0-3450.g9a047fe7.el9 ceph 5.4 M 2024-05-03T21:08:03.461 INFO:teuthology.orchestra.run.smithi138.stdout: ceph-fuse x86_64 2:19.0.0-3450.g9a047fe7.el9 ceph 891 k 2024-05-03T21:08:03.461 INFO:teuthology.orchestra.run.smithi138.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-3450.g9a047fe7.el9 ceph 145 k 2024-05-03T21:08:03.461 INFO:teuthology.orchestra.run.smithi138.stdout: ceph-mgr x86_64 2:19.0.0-3450.g9a047fe7.el9 ceph 1.5 M 2024-05-03T21:08:03.461 INFO:teuthology.orchestra.run.smithi138.stdout: ceph-mgr-cephadm noarch 2:19.0.0-3450.g9a047fe7.el9 ceph-noarch 142 k 2024-05-03T21:08:03.461 INFO:teuthology.orchestra.run.smithi138.stdout: ceph-mgr-dashboard noarch 2:19.0.0-3450.g9a047fe7.el9 ceph-noarch 3.5 M 2024-05-03T21:08:03.461 INFO:teuthology.orchestra.run.smithi138.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-3450.g9a047fe7.el9 ceph-noarch 7.4 M 2024-05-03T21:08:03.461 INFO:teuthology.orchestra.run.smithi138.stdout: ceph-mgr-rook noarch 2:19.0.0-3450.g9a047fe7.el9 ceph-noarch 49 k 2024-05-03T21:08:03.461 INFO:teuthology.orchestra.run.smithi138.stdout: ceph-radosgw x86_64 2:19.0.0-3450.g9a047fe7.el9 ceph 11 M 2024-05-03T21:08:03.461 INFO:teuthology.orchestra.run.smithi138.stdout: ceph-test x86_64 2:19.0.0-3450.g9a047fe7.el9 ceph 49 M 2024-05-03T21:08:03.462 INFO:teuthology.orchestra.run.smithi138.stdout: ceph-volume noarch 2:19.0.0-3450.g9a047fe7.el9 ceph-noarch 273 k 2024-05-03T21:08:03.462 INFO:teuthology.orchestra.run.smithi138.stdout: cephadm noarch 2:19.0.0-3450.g9a047fe7.el9 ceph-noarch 763 k 2024-05-03T21:08:03.462 INFO:teuthology.orchestra.run.smithi138.stdout: cephfs-mirror x86_64 2:19.0.0-3450.g9a047fe7.el9 ceph 225 k 2024-05-03T21:08:03.462 INFO:teuthology.orchestra.run.smithi138.stdout: cephfs-shell x86_64 2:19.0.0-3450.g9a047fe7.el9 ceph 27 k 2024-05-03T21:08:03.462 INFO:teuthology.orchestra.run.smithi138.stdout: cephfs-top noarch 2:19.0.0-3450.g9a047fe7.el9 ceph-noarch 18 k 2024-05-03T21:08:03.462 INFO:teuthology.orchestra.run.smithi138.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 appstream 21 k 2024-05-03T21:08:03.462 INFO:teuthology.orchestra.run.smithi138.stdout: flex x86_64 2.6.4-9.el9 appstream 313 k 2024-05-03T21:08:03.462 INFO:teuthology.orchestra.run.smithi138.stdout: libaio-devel x86_64 0.3.111-13.el9 appstream 11 k 2024-05-03T21:08:03.462 INFO:teuthology.orchestra.run.smithi138.stdout: libcephfs-devel x86_64 2:19.0.0-3450.g9a047fe7.el9 ceph 32 k 2024-05-03T21:08:03.462 INFO:teuthology.orchestra.run.smithi138.stdout: libcephfs2 x86_64 2:19.0.0-3450.g9a047fe7.el9 ceph 747 k 2024-05-03T21:08:03.462 INFO:teuthology.orchestra.run.smithi138.stdout: librados-devel x86_64 2:19.0.0-3450.g9a047fe7.el9 ceph 127 k 2024-05-03T21:08:03.462 INFO:teuthology.orchestra.run.smithi138.stdout: libtool x86_64 2.4.6-45.el9 appstream 582 k 2024-05-03T21:08:03.462 INFO:teuthology.orchestra.run.smithi138.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-05-03T21:08:03.463 INFO:teuthology.orchestra.run.smithi138.stdout: openssl-devel x86_64 1:3.2.1-1.el9 appstream 4.4 M 2024-05-03T21:08:03.463 INFO:teuthology.orchestra.run.smithi138.stdout: postgresql x86_64 13.14-1.el9 appstream 1.6 M 2024-05-03T21:08:03.463 INFO:teuthology.orchestra.run.smithi138.stdout: postgresql-contrib x86_64 13.14-1.el9 appstream 911 k 2024-05-03T21:08:03.463 INFO:teuthology.orchestra.run.smithi138.stdout: postgresql-server x86_64 13.14-1.el9 appstream 5.8 M 2024-05-03T21:08:03.463 INFO:teuthology.orchestra.run.smithi138.stdout: python3-cephfs x86_64 2:19.0.0-3450.g9a047fe7.el9 ceph 161 k 2024-05-03T21:08:03.463 INFO:teuthology.orchestra.run.smithi138.stdout: python3-rados x86_64 2:19.0.0-3450.g9a047fe7.el9 ceph 320 k 2024-05-03T21:08:03.463 INFO:teuthology.orchestra.run.smithi138.stdout: python3-rbd x86_64 2:19.0.0-3450.g9a047fe7.el9 ceph 298 k 2024-05-03T21:08:03.465 INFO:teuthology.orchestra.run.smithi138.stdout: python3-rgw x86_64 2:19.0.0-3450.g9a047fe7.el9 ceph 100 k 2024-05-03T21:08:03.465 INFO:teuthology.orchestra.run.smithi138.stdout: rbd-fuse x86_64 2:19.0.0-3450.g9a047fe7.el9 ceph 85 k 2024-05-03T21:08:03.465 INFO:teuthology.orchestra.run.smithi138.stdout: rbd-mirror x86_64 2:19.0.0-3450.g9a047fe7.el9 ceph 3.1 M 2024-05-03T21:08:03.465 INFO:teuthology.orchestra.run.smithi138.stdout: rbd-nbd x86_64 2:19.0.0-3450.g9a047fe7.el9 ceph 170 k 2024-05-03T21:08:03.465 INFO:teuthology.orchestra.run.smithi138.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 appstream 94 k 2024-05-03T21:08:03.465 INFO:teuthology.orchestra.run.smithi138.stdout:Upgrading: 2024-05-03T21:08:03.466 INFO:teuthology.orchestra.run.smithi138.stdout: librados2 x86_64 2:19.0.0-3450.g9a047fe7.el9 ceph 3.4 M 2024-05-03T21:08:03.466 INFO:teuthology.orchestra.run.smithi138.stdout: librbd1 x86_64 2:19.0.0-3450.g9a047fe7.el9 ceph 3.1 M 2024-05-03T21:08:03.466 INFO:teuthology.orchestra.run.smithi138.stdout: openssl x86_64 1:3.2.1-1.el9 baseos 1.3 M 2024-05-03T21:08:03.466 INFO:teuthology.orchestra.run.smithi138.stdout: openssl-libs x86_64 1:3.2.1-1.el9 baseos 2.4 M 2024-05-03T21:08:03.466 INFO:teuthology.orchestra.run.smithi138.stdout:Installing dependencies: 2024-05-03T21:08:03.466 INFO:teuthology.orchestra.run.smithi138.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-05-03T21:08:03.466 INFO:teuthology.orchestra.run.smithi138.stdout: automake noarch 1.16.2-8.el9 appstream 689 k 2024-05-03T21:08:03.466 INFO:teuthology.orchestra.run.smithi138.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-05-03T21:08:03.466 INFO:teuthology.orchestra.run.smithi138.stdout: ceph-common x86_64 2:19.0.0-3450.g9a047fe7.el9 ceph 21 M 2024-05-03T21:08:03.466 INFO:teuthology.orchestra.run.smithi138.stdout: ceph-grafana-dashboards noarch 2:19.0.0-3450.g9a047fe7.el9 ceph-noarch 34 k 2024-05-03T21:08:03.466 INFO:teuthology.orchestra.run.smithi138.stdout: ceph-mds x86_64 2:19.0.0-3450.g9a047fe7.el9 ceph 2.4 M 2024-05-03T21:08:03.467 INFO:teuthology.orchestra.run.smithi138.stdout: ceph-mgr-modules-core noarch 2:19.0.0-3450.g9a047fe7.el9 ceph-noarch 274 k 2024-05-03T21:08:03.467 INFO:teuthology.orchestra.run.smithi138.stdout: ceph-mon x86_64 2:19.0.0-3450.g9a047fe7.el9 ceph 4.7 M 2024-05-03T21:08:03.467 INFO:teuthology.orchestra.run.smithi138.stdout: ceph-osd x86_64 2:19.0.0-3450.g9a047fe7.el9 ceph 17 M 2024-05-03T21:08:03.467 INFO:teuthology.orchestra.run.smithi138.stdout: ceph-prometheus-alerts noarch 2:19.0.0-3450.g9a047fe7.el9 ceph-noarch 16 k 2024-05-03T21:08:03.467 INFO:teuthology.orchestra.run.smithi138.stdout: ceph-selinux x86_64 2:19.0.0-3450.g9a047fe7.el9 ceph 25 k 2024-05-03T21:08:03.467 INFO:teuthology.orchestra.run.smithi138.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-05-03T21:08:03.467 INFO:teuthology.orchestra.run.smithi138.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-05-03T21:08:03.467 INFO:teuthology.orchestra.run.smithi138.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-05-03T21:08:03.467 INFO:teuthology.orchestra.run.smithi138.stdout: fuse x86_64 2.9.9-15.el9 baseos 80 k 2024-05-03T21:08:03.467 INFO:teuthology.orchestra.run.smithi138.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-05-03T21:08:03.467 INFO:teuthology.orchestra.run.smithi138.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-05-03T21:08:03.467 INFO:teuthology.orchestra.run.smithi138.stdout: libarrow x86_64 9.0.0-11.el9 epel 4.4 M 2024-05-03T21:08:03.467 INFO:teuthology.orchestra.run.smithi138.stdout: libarrow-doc noarch 9.0.0-11.el9 epel 26 k 2024-05-03T21:08:03.467 INFO:teuthology.orchestra.run.smithi138.stdout: libcephsqlite x86_64 2:19.0.0-3450.g9a047fe7.el9 ceph 163 k 2024-05-03T21:08:03.468 INFO:teuthology.orchestra.run.smithi138.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-05-03T21:08:03.468 INFO:teuthology.orchestra.run.smithi138.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-05-03T21:08:03.468 INFO:teuthology.orchestra.run.smithi138.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-05-03T21:08:03.468 INFO:teuthology.orchestra.run.smithi138.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-05-03T21:08:03.468 INFO:teuthology.orchestra.run.smithi138.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-05-03T21:08:03.468 INFO:teuthology.orchestra.run.smithi138.stdout: libradosstriper1 x86_64 2:19.0.0-3450.g9a047fe7.el9 ceph 500 k 2024-05-03T21:08:03.468 INFO:teuthology.orchestra.run.smithi138.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-05-03T21:08:03.468 INFO:teuthology.orchestra.run.smithi138.stdout: librgw2 x86_64 2:19.0.0-3450.g9a047fe7.el9 ceph 5.4 M 2024-05-03T21:08:03.468 INFO:teuthology.orchestra.run.smithi138.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-05-03T21:08:03.468 INFO:teuthology.orchestra.run.smithi138.stdout: libzstd-devel x86_64 1.5.1-2.el9 appstream 47 k 2024-05-03T21:08:03.468 INFO:teuthology.orchestra.run.smithi138.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-05-03T21:08:03.469 INFO:teuthology.orchestra.run.smithi138.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-05-03T21:08:03.469 INFO:teuthology.orchestra.run.smithi138.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-05-03T21:08:03.469 INFO:teuthology.orchestra.run.smithi138.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-05-03T21:08:03.469 INFO:teuthology.orchestra.run.smithi138.stdout: openblas x86_64 0.3.21-2.el9 appstream 35 k 2024-05-03T21:08:03.469 INFO:teuthology.orchestra.run.smithi138.stdout: openblas-openmp x86_64 0.3.21-2.el9 appstream 4.7 M 2024-05-03T21:08:03.469 INFO:teuthology.orchestra.run.smithi138.stdout: parquet-libs x86_64 9.0.0-11.el9 epel 839 k 2024-05-03T21:08:03.469 INFO:teuthology.orchestra.run.smithi138.stdout: postgresql-private-libs x86_64 13.14-1.el9 appstream 133 k 2024-05-03T21:08:03.469 INFO:teuthology.orchestra.run.smithi138.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-05-03T21:08:03.469 INFO:teuthology.orchestra.run.smithi138.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-05-03T21:08:03.469 INFO:teuthology.orchestra.run.smithi138.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-05-03T21:08:03.469 INFO:teuthology.orchestra.run.smithi138.stdout: python3-attrs noarch 20.3.0-7.el9 appstream 87 k 2024-05-03T21:08:03.469 INFO:teuthology.orchestra.run.smithi138.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-05-03T21:08:03.469 INFO:teuthology.orchestra.run.smithi138.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-05-03T21:08:03.469 INFO:teuthology.orchestra.run.smithi138.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-05-03T21:08:03.470 INFO:teuthology.orchestra.run.smithi138.stdout: python3-ceph-argparse x86_64 2:19.0.0-3450.g9a047fe7.el9 ceph 45 k 2024-05-03T21:08:03.470 INFO:teuthology.orchestra.run.smithi138.stdout: python3-ceph-common x86_64 2:19.0.0-3450.g9a047fe7.el9 ceph 135 k 2024-05-03T21:08:03.470 INFO:teuthology.orchestra.run.smithi138.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-05-03T21:08:03.470 INFO:teuthology.orchestra.run.smithi138.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-05-03T21:08:03.470 INFO:teuthology.orchestra.run.smithi138.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-05-03T21:08:03.470 INFO:teuthology.orchestra.run.smithi138.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-05-03T21:08:03.470 INFO:teuthology.orchestra.run.smithi138.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-05-03T21:08:03.470 INFO:teuthology.orchestra.run.smithi138.stdout: python3-colorama noarch 0.4.4-12.el9 epel 35 k 2024-05-03T21:08:03.470 INFO:teuthology.orchestra.run.smithi138.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-05-03T21:08:03.470 INFO:teuthology.orchestra.run.smithi138.stdout: python3-devel x86_64 3.9.18-3.el9 appstream 244 k 2024-05-03T21:08:03.470 INFO:teuthology.orchestra.run.smithi138.stdout: python3-google-auth noarch 1:2.29.0-1.el9 epel 206 k 2024-05-03T21:08:03.470 INFO:teuthology.orchestra.run.smithi138.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-05-03T21:08:03.470 INFO:teuthology.orchestra.run.smithi138.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-05-03T21:08:03.470 INFO:teuthology.orchestra.run.smithi138.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-05-03T21:08:03.471 INFO:teuthology.orchestra.run.smithi138.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-05-03T21:08:03.471 INFO:teuthology.orchestra.run.smithi138.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-05-03T21:08:03.471 INFO:teuthology.orchestra.run.smithi138.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-05-03T21:08:03.471 INFO:teuthology.orchestra.run.smithi138.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-05-03T21:08:03.471 INFO:teuthology.orchestra.run.smithi138.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-05-03T21:08:03.471 INFO:teuthology.orchestra.run.smithi138.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-05-03T21:08:03.471 INFO:teuthology.orchestra.run.smithi138.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-05-03T21:08:03.471 INFO:teuthology.orchestra.run.smithi138.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-05-03T21:08:03.471 INFO:teuthology.orchestra.run.smithi138.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-05-03T21:08:03.471 INFO:teuthology.orchestra.run.smithi138.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-05-03T21:08:03.471 INFO:teuthology.orchestra.run.smithi138.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-05-03T21:08:03.471 INFO:teuthology.orchestra.run.smithi138.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-05-03T21:08:03.471 INFO:teuthology.orchestra.run.smithi138.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-05-03T21:08:03.471 INFO:teuthology.orchestra.run.smithi138.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-05-03T21:08:03.472 INFO:teuthology.orchestra.run.smithi138.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-05-03T21:08:03.472 INFO:teuthology.orchestra.run.smithi138.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-05-03T21:08:03.472 INFO:teuthology.orchestra.run.smithi138.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-05-03T21:08:03.472 INFO:teuthology.orchestra.run.smithi138.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-05-03T21:08:03.472 INFO:teuthology.orchestra.run.smithi138.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-05-03T21:08:03.472 INFO:teuthology.orchestra.run.smithi138.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-05-03T21:08:03.472 INFO:teuthology.orchestra.run.smithi138.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-05-03T21:08:03.472 INFO:teuthology.orchestra.run.smithi138.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-05-03T21:08:03.472 INFO:teuthology.orchestra.run.smithi138.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-05-03T21:08:03.472 INFO:teuthology.orchestra.run.smithi138.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-05-03T21:08:03.472 INFO:teuthology.orchestra.run.smithi138.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-05-03T21:08:03.472 INFO:teuthology.orchestra.run.smithi138.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-05-03T21:08:03.472 INFO:teuthology.orchestra.run.smithi138.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-05-03T21:08:03.472 INFO:teuthology.orchestra.run.smithi138.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-05-03T21:08:03.473 INFO:teuthology.orchestra.run.smithi138.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-05-03T21:08:03.473 INFO:teuthology.orchestra.run.smithi138.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-05-03T21:08:03.473 INFO:teuthology.orchestra.run.smithi138.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-05-03T21:08:03.473 INFO:teuthology.orchestra.run.smithi138.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-05-03T21:08:03.473 INFO:teuthology.orchestra.run.smithi138.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-05-03T21:08:03.473 INFO:teuthology.orchestra.run.smithi138.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-05-03T21:08:03.473 INFO:teuthology.orchestra.run.smithi138.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-05-03T21:08:03.473 INFO:teuthology.orchestra.run.smithi138.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-05-03T21:08:03.473 INFO:teuthology.orchestra.run.smithi138.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-05-03T21:08:03.473 INFO:teuthology.orchestra.run.smithi138.stdout: python3-urllib3 noarch 1.26.5-5.el9 baseos 215 k 2024-05-03T21:08:03.473 INFO:teuthology.orchestra.run.smithi138.stdout: python3-wcwidth noarch 0.2.5-8.el9 appstream 44 k 2024-05-03T21:08:03.473 INFO:teuthology.orchestra.run.smithi138.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-05-03T21:08:03.473 INFO:teuthology.orchestra.run.smithi138.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-05-03T21:08:03.473 INFO:teuthology.orchestra.run.smithi138.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-05-03T21:08:03.474 INFO:teuthology.orchestra.run.smithi138.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-05-03T21:08:03.474 INFO:teuthology.orchestra.run.smithi138.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-05-03T21:08:03.474 INFO:teuthology.orchestra.run.smithi138.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-05-03T21:08:03.474 INFO:teuthology.orchestra.run.smithi138.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-05-03T21:08:03.474 INFO:teuthology.orchestra.run.smithi138.stdout: socat x86_64 1.7.4.1-5.el9 appstream 305 k 2024-05-03T21:08:03.474 INFO:teuthology.orchestra.run.smithi138.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-05-03T21:08:03.474 INFO:teuthology.orchestra.run.smithi138.stdout: uuid x86_64 1.6.2-55.el9 appstream 58 k 2024-05-03T21:08:03.474 INFO:teuthology.orchestra.run.smithi138.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-05-03T21:08:03.474 INFO:teuthology.orchestra.run.smithi138.stdout: zlib-devel x86_64 1.2.11-41.el9 appstream 45 k 2024-05-03T21:08:03.474 INFO:teuthology.orchestra.run.smithi138.stdout:Installing weak dependencies: 2024-05-03T21:08:03.474 INFO:teuthology.orchestra.run.smithi138.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-05-03T21:08:03.474 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-05-03T21:08:03.474 INFO:teuthology.orchestra.run.smithi138.stdout:Transaction Summary 2024-05-03T21:08:03.474 INFO:teuthology.orchestra.run.smithi138.stdout:======================================================================================= 2024-05-03T21:08:03.475 INFO:teuthology.orchestra.run.smithi138.stdout:Install 148 Packages 2024-05-03T21:08:03.475 INFO:teuthology.orchestra.run.smithi138.stdout:Upgrade 4 Packages 2024-05-03T21:08:03.475 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-05-03T21:08:03.475 INFO:teuthology.orchestra.run.smithi138.stdout:Total download size: 222 M 2024-05-03T21:08:03.475 INFO:teuthology.orchestra.run.smithi138.stdout:Downloading Packages: 2024-05-03T21:08:03.949 INFO:teuthology.orchestra.run.smithi082.stdout:(8/152): ceph-radosgw-19.0.0-3450.g9a047fe7.el9 8.5 MB/s | 11 MB 00:01 2024-05-03T21:08:04.016 INFO:teuthology.orchestra.run.smithi082.stdout:(9/152): ceph-selinux-19.0.0-3450.g9a047fe7.el9 378 kB/s | 25 kB 00:00 2024-05-03T21:08:04.183 INFO:teuthology.orchestra.run.smithi067.stdout:(7/152): ceph-common-19.0.0-3450.g9a047fe7.el9. 4.5 MB/s | 21 MB 00:04 2024-05-03T21:08:04.257 INFO:teuthology.orchestra.run.smithi138.stdout:(1/152): ceph-19.0.0-3450.g9a047fe7.el9.x86_64. 24 kB/s | 6.6 kB 00:00 2024-05-03T21:08:04.417 INFO:teuthology.orchestra.run.smithi067.stdout:(8/152): ceph-base-19.0.0-3450.g9a047fe7.el9.x8 1.1 MB/s | 5.4 MB 00:04 2024-05-03T21:08:04.517 INFO:teuthology.orchestra.run.smithi067.stdout:(9/152): ceph-selinux-19.0.0-3450.g9a047fe7.el9 251 kB/s | 25 kB 00:00 2024-05-03T21:08:05.384 INFO:teuthology.orchestra.run.smithi067.stdout:(10/152): ceph-radosgw-19.0.0-3450.g9a047fe7.el 8.8 MB/s | 11 MB 00:01 2024-05-03T21:08:05.471 INFO:teuthology.orchestra.run.smithi138.stdout:(2/152): ceph-fuse-19.0.0-3450.g9a047fe7.el9.x8 732 kB/s | 891 kB 00:01 2024-05-03T21:08:05.475 INFO:teuthology.orchestra.run.smithi067.stdout:(11/152): cephfs-mirror-19.0.0-3450.g9a047fe7.e 2.4 MB/s | 225 kB 00:00 2024-05-03T21:08:05.550 INFO:teuthology.orchestra.run.smithi067.stdout:(12/152): cephfs-shell-19.0.0-3450.g9a047fe7.el 359 kB/s | 27 kB 00:00 2024-05-03T21:08:05.605 INFO:teuthology.orchestra.run.smithi138.stdout:(3/152): ceph-immutable-object-cache-19.0.0-345 1.1 MB/s | 145 kB 00:00 2024-05-03T21:08:05.626 INFO:teuthology.orchestra.run.smithi067.stdout:(13/152): libcephfs-devel-19.0.0-3450.g9a047fe7 423 kB/s | 32 kB 00:00 2024-05-03T21:08:05.734 INFO:teuthology.orchestra.run.smithi067.stdout:(14/152): libcephfs2-19.0.0-3450.g9a047fe7.el9. 6.7 MB/s | 747 kB 00:00 2024-05-03T21:08:05.818 INFO:teuthology.orchestra.run.smithi067.stdout:(15/152): libcephsqlite-19.0.0-3450.g9a047fe7.e 1.9 MB/s | 163 kB 00:00 2024-05-03T21:08:05.902 INFO:teuthology.orchestra.run.smithi067.stdout:(16/152): librados-devel-19.0.0-3450.g9a047fe7. 1.5 MB/s | 127 kB 00:00 2024-05-03T21:08:06.002 INFO:teuthology.orchestra.run.smithi067.stdout:(17/152): libradosstriper1-19.0.0-3450.g9a047fe 4.9 MB/s | 500 kB 00:00 2024-05-03T21:08:06.669 INFO:teuthology.orchestra.run.smithi067.stdout:(18/152): librgw2-19.0.0-3450.g9a047fe7.el9.x86 8.1 MB/s | 5.4 MB 00:00 2024-05-03T21:08:06.761 INFO:teuthology.orchestra.run.smithi067.stdout:(19/152): python3-ceph-argparse-19.0.0-3450.g9a 493 kB/s | 45 kB 00:00 2024-05-03T21:08:06.836 INFO:teuthology.orchestra.run.smithi067.stdout:(20/152): python3-ceph-common-19.0.0-3450.g9a04 1.8 MB/s | 135 kB 00:00 2024-05-03T21:08:06.920 INFO:teuthology.orchestra.run.smithi067.stdout:(21/152): python3-cephfs-19.0.0-3450.g9a047fe7. 1.9 MB/s | 161 kB 00:00 2024-05-03T21:08:06.980 INFO:teuthology.orchestra.run.smithi138.stdout:(4/152): ceph-mds-19.0.0-3450.g9a047fe7.el9.x86 1.7 MB/s | 2.4 MB 00:01 2024-05-03T21:08:07.012 INFO:teuthology.orchestra.run.smithi067.stdout:(22/152): python3-rados-19.0.0-3450.g9a047fe7.e 3.4 MB/s | 320 kB 00:00 2024-05-03T21:08:07.112 INFO:teuthology.orchestra.run.smithi067.stdout:(23/152): python3-rbd-19.0.0-3450.g9a047fe7.el9 2.9 MB/s | 298 kB 00:00 2024-05-03T21:08:07.204 INFO:teuthology.orchestra.run.smithi067.stdout:(24/152): python3-rgw-19.0.0-3450.g9a047fe7.el9 1.1 MB/s | 100 kB 00:00 2024-05-03T21:08:07.296 INFO:teuthology.orchestra.run.smithi067.stdout:(25/152): rbd-fuse-19.0.0-3450.g9a047fe7.el9.x8 928 kB/s | 85 kB 00:00 2024-05-03T21:08:07.547 INFO:teuthology.orchestra.run.smithi138.stdout:(5/152): ceph-mgr-19.0.0-3450.g9a047fe7.el9.x86 2.7 MB/s | 1.5 MB 00:00 2024-05-03T21:08:07.738 INFO:teuthology.orchestra.run.smithi067.stdout:(26/152): rbd-mirror-19.0.0-3450.g9a047fe7.el9. 7.0 MB/s | 3.1 MB 00:00 2024-05-03T21:08:07.822 INFO:teuthology.orchestra.run.smithi067.stdout:(27/152): rbd-nbd-19.0.0-3450.g9a047fe7.el9.x86 2.0 MB/s | 170 kB 00:00 2024-05-03T21:08:08.005 INFO:teuthology.orchestra.run.smithi067.stdout:(28/152): ceph-osd-19.0.0-3450.g9a047fe7.el9.x8 3.7 MB/s | 17 MB 00:04 2024-05-03T21:08:08.030 INFO:teuthology.orchestra.run.smithi067.stdout:(29/152): ceph-grafana-dashboards-19.0.0-3450.g 164 kB/s | 34 kB 00:00 2024-05-03T21:08:08.081 INFO:teuthology.orchestra.run.smithi067.stdout:(30/152): ceph-mgr-cephadm-19.0.0-3450.g9a047fe 1.9 MB/s | 142 kB 00:00 2024-05-03T21:08:08.464 INFO:teuthology.orchestra.run.smithi138.stdout:(6/152): ceph-mon-19.0.0-3450.g9a047fe7.el9.x86 5.1 MB/s | 4.7 MB 00:00 2024-05-03T21:08:08.564 INFO:teuthology.orchestra.run.smithi067.stdout:(31/152): ceph-mgr-dashboard-19.0.0-3450.g9a047 6.6 MB/s | 3.5 MB 00:00 2024-05-03T21:08:08.673 INFO:teuthology.orchestra.run.smithi067.stdout:(32/152): ceph-mgr-modules-core-19.0.0-3450.g9a 2.5 MB/s | 274 kB 00:00 2024-05-03T21:08:08.773 INFO:teuthology.orchestra.run.smithi067.stdout:(33/152): ceph-mgr-rook-19.0.0-3450.g9a047fe7.e 492 kB/s | 49 kB 00:00 2024-05-03T21:08:09.157 INFO:teuthology.orchestra.run.smithi067.stdout:(34/152): ceph-prometheus-alerts-19.0.0-3450.g9 43 kB/s | 16 kB 00:00 2024-05-03T21:08:09.524 INFO:teuthology.orchestra.run.smithi067.stdout:(35/152): ceph-volume-19.0.0-3450.g9a047fe7.el9 745 kB/s | 273 kB 00:00 2024-05-03T21:08:09.574 INFO:teuthology.orchestra.run.smithi082.stdout:(10/152): ceph-test-19.0.0-3450.g9a047fe7.el9.x 8.8 MB/s | 49 MB 00:05 2024-05-03T21:08:09.716 INFO:teuthology.orchestra.run.smithi067.stdout:(36/152): cephadm-19.0.0-3450.g9a047fe7.el9.noa 3.9 MB/s | 763 kB 00:00 2024-05-03T21:08:09.732 INFO:teuthology.orchestra.run.smithi082.stdout:(11/152): ceph-osd-19.0.0-3450.g9a047fe7.el9.x8 2.3 MB/s | 17 MB 00:07 2024-05-03T21:08:09.757 INFO:teuthology.orchestra.run.smithi082.stdout:(12/152): cephfs-mirror-19.0.0-3450.g9a047fe7.e 1.2 MB/s | 225 kB 00:00 2024-05-03T21:08:09.816 INFO:teuthology.orchestra.run.smithi067.stdout:(37/152): cephfs-top-19.0.0-3450.g9a047fe7.el9. 182 kB/s | 18 kB 00:00 2024-05-03T21:08:09.824 INFO:teuthology.orchestra.run.smithi082.stdout:(13/152): cephfs-shell-19.0.0-3450.g9a047fe7.el 295 kB/s | 27 kB 00:00 2024-05-03T21:08:09.850 INFO:teuthology.orchestra.run.smithi082.stdout:(14/152): libcephfs-devel-19.0.0-3450.g9a047fe7 345 kB/s | 32 kB 00:00 2024-05-03T21:08:09.925 INFO:teuthology.orchestra.run.smithi082.stdout:(15/152): libcephsqlite-19.0.0-3450.g9a047fe7.e 2.1 MB/s | 163 kB 00:00 2024-05-03T21:08:10.008 INFO:teuthology.orchestra.run.smithi067.stdout:(38/152): ceph-mgr-diskprediction-local-19.0.0- 3.8 MB/s | 7.4 MB 00:01 2024-05-03T21:08:10.033 INFO:teuthology.orchestra.run.smithi067.stdout:(39/152): fuse-2.9.9-15.el9.x86_64.rpm 370 kB/s | 80 kB 00:00 2024-05-03T21:08:10.117 INFO:teuthology.orchestra.run.smithi067.stdout:(40/152): libquadmath-11.4.1-3.el9.x86_64.rpm 2.3 MB/s | 195 kB 00:00 2024-05-03T21:08:10.125 INFO:teuthology.orchestra.run.smithi082.stdout:(16/152): ceph-common-19.0.0-3450.g9a047fe7.el9 2.0 MB/s | 21 MB 00:10 2024-05-03T21:08:10.131 INFO:teuthology.orchestra.run.smithi138.stdout:(7/152): ceph-osd-19.0.0-3450.g9a047fe7.el9.x86 10 MB/s | 17 MB 00:01 2024-05-03T21:08:10.151 INFO:teuthology.orchestra.run.smithi082.stdout:(17/152): librados-devel-19.0.0-3450.g9a047fe7. 564 kB/s | 127 kB 00:00 2024-05-03T21:08:10.184 INFO:teuthology.orchestra.run.smithi082.stdout:(18/152): libcephfs2-19.0.0-3450.g9a047fe7.el9. 2.0 MB/s | 747 kB 00:00 2024-05-03T21:08:10.200 INFO:teuthology.orchestra.run.smithi067.stdout:(41/152): python3-cffi-1.14.5-5.el9.x86_64.rpm 3.0 MB/s | 253 kB 00:00 2024-05-03T21:08:10.276 INFO:teuthology.orchestra.run.smithi082.stdout:(19/152): python3-ceph-argparse-19.0.0-3450.g9a 493 kB/s | 45 kB 00:00 2024-05-03T21:08:10.326 INFO:teuthology.orchestra.run.smithi067.stdout:(42/152): libgfortran-11.4.1-3.el9.x86_64.rpm 2.5 MB/s | 803 kB 00:00 2024-05-03T21:08:10.368 INFO:teuthology.orchestra.run.smithi082.stdout:(20/152): libradosstriper1-19.0.0-3450.g9a047fe 2.0 MB/s | 500 kB 00:00 2024-05-03T21:08:10.384 INFO:teuthology.orchestra.run.smithi067.stdout:(43/152): python3-ply-3.11-14.el9.noarch.rpm 1.8 MB/s | 106 kB 00:00 2024-05-03T21:08:10.393 INFO:teuthology.orchestra.run.smithi082.stdout:(21/152): python3-ceph-common-19.0.0-3450.g9a04 1.1 MB/s | 135 kB 00:00 2024-05-03T21:08:10.468 INFO:teuthology.orchestra.run.smithi067.stdout:(44/152): python3-pycparser-2.20-6.el9.noarch.r 1.6 MB/s | 135 kB 00:00 2024-05-03T21:08:10.469 INFO:teuthology.orchestra.run.smithi082.stdout:(22/152): python3-cephfs-19.0.0-3450.g9a047fe7. 1.6 MB/s | 161 kB 00:00 2024-05-03T21:08:10.527 INFO:teuthology.orchestra.run.smithi067.stdout:(45/152): python3-requests-2.25.1-8.el9.noarch. 2.1 MB/s | 125 kB 00:00 2024-05-03T21:08:10.560 INFO:teuthology.orchestra.run.smithi067.stdout:(46/152): python3-cryptography-36.0.1-4.el9.x86 3.4 MB/s | 1.2 MB 00:00 2024-05-03T21:08:10.585 INFO:teuthology.orchestra.run.smithi082.stdout:(23/152): python3-rados-19.0.0-3450.g9a047fe7.e 1.6 MB/s | 320 kB 00:00 2024-05-03T21:08:10.610 INFO:teuthology.orchestra.run.smithi067.stdout:(47/152): python3-urllib3-1.26.5-5.el9.noarch.r 2.5 MB/s | 215 kB 00:00 2024-05-03T21:08:10.694 INFO:teuthology.orchestra.run.smithi082.stdout:(24/152): python3-rbd-19.0.0-3450.g9a047fe7.el9 1.3 MB/s | 298 kB 00:00 2024-05-03T21:08:10.719 INFO:teuthology.orchestra.run.smithi082.stdout:(25/152): python3-rgw-19.0.0-3450.g9a047fe7.el9 747 kB/s | 100 kB 00:00 2024-05-03T21:08:10.803 INFO:teuthology.orchestra.run.smithi082.stdout:(26/152): rbd-fuse-19.0.0-3450.g9a047fe7.el9.x8 786 kB/s | 85 kB 00:00 2024-05-03T21:08:10.978 INFO:teuthology.orchestra.run.smithi082.stdout:(27/152): librgw2-19.0.0-3450.g9a047fe7.el9.x86 6.5 MB/s | 5.4 MB 00:00 2024-05-03T21:08:10.981 INFO:teuthology.orchestra.run.smithi138.stdout:(8/152): ceph-radosgw-19.0.0-3450.g9a047fe7.el9 12 MB/s | 11 MB 00:00 2024-05-03T21:08:11.012 INFO:teuthology.orchestra.run.smithi082.stdout:(28/152): rbd-nbd-19.0.0-3450.g9a047fe7.el9.x86 816 kB/s | 170 kB 00:00 2024-05-03T21:08:11.046 INFO:teuthology.orchestra.run.smithi082.stdout:(29/152): ceph-grafana-dashboards-19.0.0-3450.g 509 kB/s | 34 kB 00:00 2024-05-03T21:08:11.047 INFO:teuthology.orchestra.run.smithi138.stdout:(9/152): ceph-selinux-19.0.0-3450.g9a047fe7.el9 379 kB/s | 25 kB 00:00 2024-05-03T21:08:11.079 INFO:teuthology.orchestra.run.smithi082.stdout:(30/152): ceph-mgr-cephadm-19.0.0-3450.g9a047fe 2.1 MB/s | 142 kB 00:00 2024-05-03T21:08:11.511 INFO:teuthology.orchestra.run.smithi067.stdout:(48/152): automake-1.16.2-8.el9.noarch.rpm 725 kB/s | 689 kB 00:00 2024-05-03T21:08:11.561 INFO:teuthology.orchestra.run.smithi067.stdout:(49/152): bison-3.7.4-5.el9.x86_64.rpm 992 kB/s | 943 kB 00:00 2024-05-03T21:08:11.571 INFO:teuthology.orchestra.run.smithi082.stdout:(31/152): ceph-mgr-dashboard-19.0.0-3450.g9a047 6.7 MB/s | 3.5 MB 00:00 2024-05-03T21:08:11.680 INFO:teuthology.orchestra.run.smithi082.stdout:(32/152): ceph-mgr-modules-core-19.0.0-3450.g9a 2.5 MB/s | 274 kB 00:00 2024-05-03T21:08:11.780 INFO:teuthology.orchestra.run.smithi082.stdout:(33/152): ceph-mgr-rook-19.0.0-3450.g9a047fe7.e 494 kB/s | 49 kB 00:00 2024-05-03T21:08:11.872 INFO:teuthology.orchestra.run.smithi082.stdout:(34/152): ceph-prometheus-alerts-19.0.0-3450.g9 179 kB/s | 16 kB 00:00 2024-05-03T21:08:11.997 INFO:teuthology.orchestra.run.smithi082.stdout:(35/152): ceph-volume-19.0.0-3450.g9a047fe7.el9 2.1 MB/s | 273 kB 00:00 2024-05-03T21:08:12.048 INFO:teuthology.orchestra.run.smithi082.stdout:(36/152): rbd-mirror-19.0.0-3450.g9a047fe7.el9. 2.3 MB/s | 3.1 MB 00:01 2024-05-03T21:08:12.148 INFO:teuthology.orchestra.run.smithi082.stdout:(37/152): cephfs-top-19.0.0-3450.g9a047fe7.el9. 182 kB/s | 18 kB 00:00 2024-05-03T21:08:12.153 INFO:teuthology.orchestra.run.smithi067.stdout:(50/152): elfutils-libelf-devel-0.190-2.el9.x86 36 kB/s | 21 kB 00:00 2024-05-03T21:08:12.228 INFO:teuthology.orchestra.run.smithi067.stdout:(51/152): boost-program-options-1.75.0-8.el9.x8 149 kB/s | 107 kB 00:00 2024-05-03T21:08:12.281 INFO:teuthology.orchestra.run.smithi138.stdout:(10/152): ceph-base-19.0.0-3450.g9a047fe7.el9.x 662 kB/s | 5.4 MB 00:08 2024-05-03T21:08:12.315 INFO:teuthology.orchestra.run.smithi082.stdout:(38/152): fuse-2.9.9-15.el9.x86_64.rpm 482 kB/s | 80 kB 00:00 2024-05-03T21:08:12.440 INFO:teuthology.orchestra.run.smithi082.stdout:(39/152): cephadm-19.0.0-3450.g9a047fe7.el9.noa 1.7 MB/s | 763 kB 00:00 2024-05-03T21:08:12.556 INFO:teuthology.orchestra.run.smithi138.stdout:(11/152): cephfs-mirror-19.0.0-3450.g9a047fe7.e 817 kB/s | 225 kB 00:00 2024-05-03T21:08:12.582 INFO:teuthology.orchestra.run.smithi082.stdout:(40/152): libgfortran-11.4.1-3.el9.x86_64.rpm 2.9 MB/s | 803 kB 00:00 2024-05-03T21:08:12.607 INFO:teuthology.orchestra.run.smithi082.stdout:(41/152): libquadmath-11.4.1-3.el9.x86_64.rpm 1.1 MB/s | 195 kB 00:00 2024-05-03T21:08:12.657 INFO:teuthology.orchestra.run.smithi138.stdout:(12/152): cephfs-shell-19.0.0-3450.g9a047fe7.el 271 kB/s | 27 kB 00:00 2024-05-03T21:08:12.666 INFO:teuthology.orchestra.run.smithi082.stdout:(42/152): python3-cffi-1.14.5-5.el9.x86_64.rpm 3.0 MB/s | 253 kB 00:00 2024-05-03T21:08:12.716 INFO:teuthology.orchestra.run.smithi082.stdout:(43/152): python3-ply-3.11-14.el9.noarch.rpm 2.1 MB/s | 106 kB 00:00 2024-05-03T21:08:12.765 INFO:teuthology.orchestra.run.smithi138.stdout:(13/152): libcephfs-devel-19.0.0-3450.g9a047fe7 293 kB/s | 32 kB 00:00 2024-05-03T21:08:12.775 INFO:teuthology.orchestra.run.smithi082.stdout:(44/152): python3-cryptography-36.0.1-4.el9.x86 7.3 MB/s | 1.2 MB 00:00 2024-05-03T21:08:12.800 INFO:teuthology.orchestra.run.smithi082.stdout:(45/152): python3-pycparser-2.20-6.el9.noarch.r 1.6 MB/s | 135 kB 00:00 2024-05-03T21:08:12.820 INFO:teuthology.orchestra.run.smithi067.stdout:(52/152): flexiblas-3.0.4-8.el9.x86_64.rpm 55 kB/s | 33 kB 00:00 2024-05-03T21:08:12.825 INFO:teuthology.orchestra.run.smithi082.stdout:(46/152): python3-requests-2.25.1-8.el9.noarch. 2.4 MB/s | 125 kB 00:00 2024-05-03T21:08:12.851 INFO:teuthology.orchestra.run.smithi082.stdout:(47/152): python3-urllib3-1.26.5-5.el9.noarch.r 4.2 MB/s | 215 kB 00:00 2024-05-03T21:08:12.993 INFO:teuthology.orchestra.run.smithi082.stdout:(48/152): automake-1.16.2-8.el9.noarch.rpm 4.0 MB/s | 689 kB 00:00 2024-05-03T21:08:12.996 INFO:teuthology.orchestra.run.smithi067.stdout:(53/152): flex-2.6.4-9.el9.x86_64.rpm 372 kB/s | 313 kB 00:00 2024-05-03T21:08:13.035 INFO:teuthology.orchestra.run.smithi082.stdout:(49/152): bison-3.7.4-5.el9.x86_64.rpm 5.0 MB/s | 943 kB 00:00 2024-05-03T21:08:13.060 INFO:teuthology.orchestra.run.smithi082.stdout:(50/152): boost-program-options-1.75.0-8.el9.x8 1.6 MB/s | 107 kB 00:00 2024-05-03T21:08:13.085 INFO:teuthology.orchestra.run.smithi082.stdout:(51/152): elfutils-libelf-devel-0.190-2.el9.x86 429 kB/s | 21 kB 00:00 2024-05-03T21:08:13.110 INFO:teuthology.orchestra.run.smithi082.stdout:(52/152): flex-2.6.4-9.el9.x86_64.rpm 6.1 MB/s | 313 kB 00:00 2024-05-03T21:08:13.136 INFO:teuthology.orchestra.run.smithi082.stdout:(53/152): flexiblas-3.0.4-8.el9.x86_64.rpm 653 kB/s | 33 kB 00:00 2024-05-03T21:08:13.178 INFO:teuthology.orchestra.run.smithi082.stdout:(54/152): flexiblas-openblas-openmp-3.0.4-8.el9 439 kB/s | 18 kB 00:00 2024-05-03T21:08:13.228 INFO:teuthology.orchestra.run.smithi082.stdout:(55/152): flexiblas-netlib-3.0.4-8.el9.x86_64.r 25 MB/s | 3.0 MB 00:00 2024-05-03T21:08:13.262 INFO:teuthology.orchestra.run.smithi082.stdout:(56/152): libaio-devel-0.3.111-13.el9.x86_64.rp 133 kB/s | 11 kB 00:00 2024-05-03T21:08:13.289 INFO:teuthology.orchestra.run.smithi082.stdout:(57/152): libpmemobj-1.12.1-1.el9.x86_64.rpm 2.7 MB/s | 160 kB 00:00 2024-05-03T21:08:13.312 INFO:teuthology.orchestra.run.smithi082.stdout:(58/152): librabbitmq-0.11.0-7.el9.x86_64.rpm 905 kB/s | 45 kB 00:00 2024-05-03T21:08:13.347 INFO:teuthology.orchestra.run.smithi082.stdout:(59/152): librdkafka-1.6.1-102.el9.x86_64.rpm 11 MB/s | 662 kB 00:00 2024-05-03T21:08:13.371 INFO:teuthology.orchestra.run.smithi082.stdout:(60/152): libtool-2.4.6-45.el9.x86_64.rpm 9.8 MB/s | 582 kB 00:00 2024-05-03T21:08:13.397 INFO:teuthology.orchestra.run.smithi082.stdout:(61/152): libuuid-devel-2.37.4-18.el9.x86_64.rp 507 kB/s | 25 kB 00:00 2024-05-03T21:08:13.422 INFO:teuthology.orchestra.run.smithi082.stdout:(62/152): libzstd-devel-1.5.1-2.el9.x86_64.rpm 944 kB/s | 47 kB 00:00 2024-05-03T21:08:13.447 INFO:teuthology.orchestra.run.smithi082.stdout:(63/152): lttng-ust-2.12.0-6.el9.x86_64.rpm 5.7 MB/s | 292 kB 00:00 2024-05-03T21:08:13.472 INFO:teuthology.orchestra.run.smithi082.stdout:(64/152): lua-5.4.4-4.el9.x86_64.rpm 3.7 MB/s | 188 kB 00:00 2024-05-03T21:08:13.474 INFO:teuthology.orchestra.run.smithi138.stdout:(14/152): libcephfs2-19.0.0-3450.g9a047fe7.el9. 1.0 MB/s | 747 kB 00:00 2024-05-03T21:08:13.498 INFO:teuthology.orchestra.run.smithi082.stdout:(65/152): openblas-0.3.21-2.el9.x86_64.rpm 703 kB/s | 35 kB 00:00 2024-05-03T21:08:13.554 INFO:teuthology.orchestra.run.smithi067.stdout:(54/152): flexiblas-openblas-openmp-3.0.4-8.el9 33 kB/s | 18 kB 00:00 2024-05-03T21:08:13.590 INFO:teuthology.orchestra.run.smithi082.stdout:(66/152): openblas-openmp-0.3.21-2.el9.x86_64.r 41 MB/s | 4.7 MB 00:00 2024-05-03T21:08:13.665 INFO:teuthology.orchestra.run.smithi082.stdout:(67/152): openssl-devel-3.2.1-1.el9.x86_64.rpm 27 MB/s | 4.4 MB 00:00 2024-05-03T21:08:13.707 INFO:teuthology.orchestra.run.smithi082.stdout:(68/152): postgresql-13.14-1.el9.x86_64.rpm 14 MB/s | 1.6 MB 00:00 2024-05-03T21:08:13.707 INFO:teuthology.orchestra.run.smithi138.stdout:(15/152): libcephsqlite-19.0.0-3450.g9a047fe7.e 701 kB/s | 163 kB 00:00 2024-05-03T21:08:13.749 INFO:teuthology.orchestra.run.smithi082.stdout:(69/152): postgresql-contrib-13.14-1.el9.x86_64 11 MB/s | 911 kB 00:00 2024-05-03T21:08:13.774 INFO:teuthology.orchestra.run.smithi082.stdout:(70/152): postgresql-private-libs-13.14-1.el9.x 2.0 MB/s | 133 kB 00:00 2024-05-03T21:08:13.833 INFO:teuthology.orchestra.run.smithi082.stdout:(71/152): protobuf-3.14.0-13.el9.x86_64.rpm 17 MB/s | 1.0 MB 00:00 2024-05-03T21:08:13.883 INFO:teuthology.orchestra.run.smithi138.stdout:(16/152): librados-devel-19.0.0-3450.g9a047fe7. 725 kB/s | 127 kB 00:00 2024-05-03T21:08:13.908 INFO:teuthology.orchestra.run.smithi082.stdout:(72/152): postgresql-server-13.14-1.el9.x86_64. 36 MB/s | 5.8 MB 00:00 2024-05-03T21:08:13.933 INFO:teuthology.orchestra.run.smithi082.stdout:(73/152): python3-attrs-20.3.0-7.el9.noarch.rpm 875 kB/s | 87 kB 00:00 2024-05-03T21:08:13.963 INFO:teuthology.orchestra.run.smithi067.stdout:(55/152): flexiblas-netlib-3.0.4-8.el9.x86_64.r 2.6 MB/s | 3.0 MB 00:01 2024-05-03T21:08:13.975 INFO:teuthology.orchestra.run.smithi082.stdout:(74/152): python3-devel-3.9.18-3.el9.x86_64.rpm 5.7 MB/s | 244 kB 00:00 2024-05-03T21:08:14.059 INFO:teuthology.orchestra.run.smithi082.stdout:(75/152): python3-babel-2.9.1-2.el9.noarch.rpm 40 MB/s | 6.0 MB 00:00 2024-05-03T21:08:14.080 INFO:teuthology.orchestra.run.smithi067.stdout:(56/152): libaio-devel-0.3.111-13.el9.x86_64.rp 21 kB/s | 11 kB 00:00 2024-05-03T21:08:14.084 INFO:teuthology.orchestra.run.smithi082.stdout:(76/152): python3-jinja2-2.11.3-5.el9.noarch.rp 2.2 MB/s | 248 kB 00:00 2024-05-03T21:08:14.109 INFO:teuthology.orchestra.run.smithi082.stdout:(77/152): python3-jsonpatch-1.21-16.el9.noarch. 513 kB/s | 26 kB 00:00 2024-05-03T21:08:14.135 INFO:teuthology.orchestra.run.smithi082.stdout:(78/152): python3-jsonpointer-2.0-4.el9.noarch. 383 kB/s | 19 kB 00:00 2024-05-03T21:08:14.223 INFO:teuthology.orchestra.run.smithi082.stdout:(79/152): python3-mako-1.1.4-6.el9.noarch.rpm 1.5 MB/s | 172 kB 00:00 2024-05-03T21:08:14.244 INFO:teuthology.orchestra.run.smithi082.stdout:(80/152): python3-markupsafe-1.1.1-12.el9.x86_6 320 kB/s | 35 kB 00:00 2024-05-03T21:08:14.294 INFO:teuthology.orchestra.run.smithi082.stdout:(81/152): python3-numpy-f2py-1.20.1-5.el9.x86_6 4.9 MB/s | 250 kB 00:00 2024-05-03T21:08:14.369 INFO:teuthology.orchestra.run.smithi082.stdout:(82/152): python3-numpy-1.20.1-5.el9.x86_64.rpm 35 MB/s | 5.1 MB 00:00 2024-05-03T21:08:14.391 INFO:teuthology.orchestra.run.smithi138.stdout:(17/152): libradosstriper1-19.0.0-3450.g9a047fe 985 kB/s | 500 kB 00:00 2024-05-03T21:08:14.394 INFO:teuthology.orchestra.run.smithi082.stdout:(83/152): python3-oauthlib-3.1.1-5.el9.noarch.r 2.2 MB/s | 222 kB 00:00 2024-05-03T21:08:14.420 INFO:teuthology.orchestra.run.smithi082.stdout:(84/152): python3-packaging-20.9-5.el9.noarch.r 1.5 MB/s | 77 kB 00:00 2024-05-03T21:08:14.445 INFO:teuthology.orchestra.run.smithi082.stdout:(85/152): python3-prettytable-0.7.2-27.el9.noar 841 kB/s | 42 kB 00:00 2024-05-03T21:08:14.471 INFO:teuthology.orchestra.run.smithi082.stdout:(86/152): python3-protobuf-3.14.0-13.el9.noarch 5.2 MB/s | 269 kB 00:00 2024-05-03T21:08:14.496 INFO:teuthology.orchestra.run.smithi082.stdout:(87/152): python3-pyasn1-0.4.8-6.el9.noarch.rpm 3.1 MB/s | 159 kB 00:00 2024-05-03T21:08:14.521 INFO:teuthology.orchestra.run.smithi082.stdout:(88/152): python3-pyasn1-modules-0.4.8-6.el9.no 5.4 MB/s | 279 kB 00:00 2024-05-03T21:08:14.548 INFO:teuthology.orchestra.run.smithi082.stdout:(89/152): python3-pytz-2021.1-5.el9.noarch.rpm 1.0 MB/s | 51 kB 00:00 2024-05-03T21:08:14.572 INFO:teuthology.orchestra.run.smithi082.stdout:(90/152): python3-requests-oauthlib-1.3.0-12.el 1.0 MB/s | 54 kB 00:00 2024-05-03T21:08:14.614 INFO:teuthology.orchestra.run.smithi082.stdout:(91/152): python3-toml-0.10.2-6.el9.noarch.rpm 1.0 MB/s | 42 kB 00:00 2024-05-03T21:08:14.656 INFO:teuthology.orchestra.run.smithi082.stdout:(92/152): python3-wcwidth-0.2.5-8.el9.noarch.rp 1.0 MB/s | 44 kB 00:00 2024-05-03T21:08:14.697 INFO:teuthology.orchestra.run.smithi082.stdout:(93/152): qatlib-24.02.0-1.el9.x86_64.rpm 5.2 MB/s | 221 kB 00:00 2024-05-03T21:08:14.713 INFO:teuthology.orchestra.run.smithi067.stdout:(57/152): librabbitmq-0.11.0-7.el9.x86_64.rpm 71 kB/s | 45 kB 00:00 2024-05-03T21:08:14.739 INFO:teuthology.orchestra.run.smithi067.stdout:(58/152): libpmemobj-1.12.1-1.el9.x86_64.rpm 207 kB/s | 160 kB 00:00 2024-05-03T21:08:14.739 INFO:teuthology.orchestra.run.smithi082.stdout:(94/152): qatlib-service-24.02.0-1.el9.x86_64.r 863 kB/s | 36 kB 00:00 2024-05-03T21:08:14.875 INFO:teuthology.orchestra.run.smithi138.stdout:(18/152): ceph-test-19.0.0-3450.g9a047fe7.el9.x 13 MB/s | 49 MB 00:03 2024-05-03T21:08:14.898 INFO:teuthology.orchestra.run.smithi082.stdout:(95/152): python3-scipy-1.6.2-8.el9.x86_64.rpm 46 MB/s | 16 MB 00:00 2024-05-03T21:08:14.924 INFO:teuthology.orchestra.run.smithi082.stdout:(96/152): qatzip-libs-1.2.0-1.el9.x86_64.rpm 258 kB/s | 47 kB 00:00 2024-05-03T21:08:14.959 INFO:teuthology.orchestra.run.smithi138.stdout:(19/152): python3-ceph-argparse-19.0.0-3450.g9a 544 kB/s | 45 kB 00:00 2024-05-03T21:08:15.015 INFO:teuthology.orchestra.run.smithi082.stdout:(97/152): ceph-mgr-diskprediction-local-19.0.0- 1.9 MB/s | 7.4 MB 00:03 2024-05-03T21:08:15.034 INFO:teuthology.orchestra.run.smithi138.stdout:(20/152): python3-ceph-common-19.0.0-3450.g9a04 1.8 MB/s | 135 kB 00:00 2024-05-03T21:08:15.041 INFO:teuthology.orchestra.run.smithi082.stdout:(98/152): uuid-1.6.2-55.el9.x86_64.rpm 495 kB/s | 58 kB 00:00 2024-05-03T21:08:15.066 INFO:teuthology.orchestra.run.smithi082.stdout:(99/152): socat-1.7.4.1-5.el9.x86_64.rpm 1.8 MB/s | 305 kB 00:00 2024-05-03T21:08:15.091 INFO:teuthology.orchestra.run.smithi082.stdout:(100/152): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.2 MB/s | 64 kB 00:00 2024-05-03T21:08:15.117 INFO:teuthology.orchestra.run.smithi082.stdout:(101/152): zlib-devel-1.2.11-41.el9.x86_64.rpm 891 kB/s | 45 kB 00:00 2024-05-03T21:08:15.117 INFO:teuthology.orchestra.run.smithi138.stdout:(21/152): python3-cephfs-19.0.0-3450.g9a047fe7. 1.9 MB/s | 161 kB 00:00 2024-05-03T21:08:15.142 INFO:teuthology.orchestra.run.smithi082.stdout:(102/152): xfsprogs-devel-6.3.0-1.el9.x86_64.rp 744 kB/s | 94 kB 00:00 2024-05-03T21:08:15.184 INFO:teuthology.orchestra.run.smithi082.stdout:(103/152): lua-devel-5.4.4-4.el9.x86_64.rpm 242 kB/s | 22 kB 00:00 2024-05-03T21:08:15.218 INFO:teuthology.orchestra.run.smithi138.stdout:(22/152): python3-rados-19.0.0-3450.g9a047fe7.e 3.1 MB/s | 320 kB 00:00 2024-05-03T21:08:15.259 INFO:teuthology.orchestra.run.smithi082.stdout:(104/152): protobuf-compiler-3.14.0-13.el9.x86_ 5.9 MB/s | 863 kB 00:00 2024-05-03T21:08:15.327 INFO:teuthology.orchestra.run.smithi138.stdout:(23/152): python3-rbd-19.0.0-3450.g9a047fe7.el9 2.7 MB/s | 298 kB 00:00 2024-05-03T21:08:15.384 INFO:teuthology.orchestra.run.smithi082.stdout:(105/152): grpc-data-1.46.7-10.el9.noarch.rpm 156 kB/s | 19 kB 00:00 2024-05-03T21:08:15.435 INFO:teuthology.orchestra.run.smithi138.stdout:(24/152): python3-rgw-19.0.0-3450.g9a047fe7.el9 921 kB/s | 100 kB 00:00 2024-05-03T21:08:15.443 INFO:teuthology.orchestra.run.smithi082.stdout:(106/152): abseil-cpp-20211102.0-3.el9.x86_64.r 1.8 MB/s | 548 kB 00:00 2024-05-03T21:08:15.468 INFO:teuthology.orchestra.run.smithi082.stdout:(107/152): gperftools-libs-2.9.1-2.el9.x86_64.r 1.1 MB/s | 309 kB 00:00 2024-05-03T21:08:15.502 INFO:teuthology.orchestra.run.smithi082.stdout:(108/152): libarrow-doc-9.0.0-11.el9.noarch.rpm 443 kB/s | 26 kB 00:00 2024-05-03T21:08:15.535 INFO:teuthology.orchestra.run.smithi082.stdout:(109/152): liboath-2.6.7-2.el9.x86_64.rpm 726 kB/s | 49 kB 00:00 2024-05-03T21:08:15.552 INFO:teuthology.orchestra.run.smithi138.stdout:(25/152): rbd-fuse-19.0.0-3450.g9a047fe7.el9.x8 730 kB/s | 85 kB 00:00 2024-05-03T21:08:15.569 INFO:teuthology.orchestra.run.smithi082.stdout:(110/152): libunwind-1.6.2-1.el9.x86_64.rpm 1.0 MB/s | 67 kB 00:00 2024-05-03T21:08:15.611 INFO:teuthology.orchestra.run.smithi082.stdout:(111/152): luarocks-3.9.2-1.el9.noarch.rpm 2.0 MB/s | 151 kB 00:00 2024-05-03T21:08:15.622 INFO:teuthology.orchestra.run.smithi067.stdout:(59/152): librdkafka-1.6.1-102.el9.x86_64.rpm 729 kB/s | 662 kB 00:00 2024-05-03T21:08:15.656 INFO:teuthology.orchestra.run.smithi067.stdout:(60/152): libtool-2.4.6-45.el9.x86_64.rpm 635 kB/s | 582 kB 00:00 2024-05-03T21:08:15.686 INFO:teuthology.orchestra.run.smithi082.stdout:(112/152): parquet-libs-9.0.0-11.el9.x86_64.rpm 7.0 MB/s | 839 kB 00:00 2024-05-03T21:08:15.754 INFO:teuthology.orchestra.run.smithi082.stdout:(113/152): python3-asyncssh-2.13.2-5.el9.noarch 3.8 MB/s | 548 kB 00:00 2024-05-03T21:08:15.770 INFO:teuthology.orchestra.run.smithi082.stdout:(114/152): python3-bcrypt-3.2.2-1.el9.x86_64.rp 518 kB/s | 43 kB 00:00 2024-05-03T21:08:15.837 INFO:teuthology.orchestra.run.smithi082.stdout:(115/152): libarrow-9.0.0-11.el9.x86_64.rpm 9.8 MB/s | 4.4 MB 00:00 2024-05-03T21:08:15.862 INFO:teuthology.orchestra.run.smithi082.stdout:(116/152): python3-certifi-2021.10.8-2.el9.noar 163 kB/s | 15 kB 00:00 2024-05-03T21:08:15.879 INFO:teuthology.orchestra.run.smithi082.stdout:(117/152): python3-cachetools-4.2.4-1.el9.noarc 256 kB/s | 32 kB 00:00 2024-05-03T21:08:15.905 INFO:teuthology.orchestra.run.smithi082.stdout:(118/152): python3-cheroot-8.6.0-4.el9.noarch.r 2.5 MB/s | 172 kB 00:00 2024-05-03T21:08:15.930 INFO:teuthology.orchestra.run.smithi082.stdout:(119/152): python3-cherrypy-18.6.1-2.el9.noarch 5.2 MB/s | 358 kB 00:00 2024-05-03T21:08:15.955 INFO:teuthology.orchestra.run.smithi082.stdout:(120/152): python3-cmd2-2.4.2-2.el9.noarch.rpm 3.9 MB/s | 303 kB 00:00 2024-05-03T21:08:15.981 INFO:teuthology.orchestra.run.smithi082.stdout:(121/152): python3-colorama-0.4.4-12.el9.noarch 468 kB/s | 35 kB 00:00 2024-05-03T21:08:16.006 INFO:teuthology.orchestra.run.smithi082.stdout:(122/152): python3-google-auth-2.29.0-1.el9.noa 2.7 MB/s | 206 kB 00:00 2024-05-03T21:08:16.065 INFO:teuthology.orchestra.run.smithi082.stdout:(123/152): python3-grpcio-tools-1.46.7-10.el9.x 1.7 MB/s | 144 kB 00:00 2024-05-03T21:08:16.090 INFO:teuthology.orchestra.run.smithi082.stdout:(124/152): python3-jaraco-8.2.1-3.el9.noarch.rp 128 kB/s | 11 kB 00:00 2024-05-03T21:08:16.140 INFO:teuthology.orchestra.run.smithi082.stdout:(125/152): python3-grpcio-1.46.7-10.el9.x86_64. 11 MB/s | 2.0 MB 00:00 2024-05-03T21:08:16.153 INFO:teuthology.orchestra.run.smithi138.stdout:(26/152): rbd-mirror-19.0.0-3450.g9a047fe7.el9. 5.2 MB/s | 3.1 MB 00:00 2024-05-03T21:08:16.165 INFO:teuthology.orchestra.run.smithi082.stdout:(126/152): python3-jaraco-classes-3.2.1-5.el9.n 176 kB/s | 18 kB 00:00 2024-05-03T21:08:16.190 INFO:teuthology.orchestra.run.smithi082.stdout:(127/152): python3-jaraco-collections-3.0.0-8.e 231 kB/s | 23 kB 00:00 2024-05-03T21:08:16.215 INFO:teuthology.orchestra.run.smithi067.stdout:(61/152): libuuid-devel-2.37.4-18.el9.x86_64.rp 43 kB/s | 25 kB 00:00 2024-05-03T21:08:16.216 INFO:teuthology.orchestra.run.smithi082.stdout:(128/152): python3-jaraco-functools-3.5.0-2.el9 258 kB/s | 19 kB 00:00 2024-05-03T21:08:16.220 INFO:teuthology.orchestra.run.smithi138.stdout:(27/152): rbd-nbd-19.0.0-3450.g9a047fe7.el9.x86 2.5 MB/s | 170 kB 00:00 2024-05-03T21:08:16.241 INFO:teuthology.orchestra.run.smithi082.stdout:(129/152): python3-jaraco-text-3.2.0-6.el9.noar 394 kB/s | 20 kB 00:00 2024-05-03T21:08:16.274 INFO:teuthology.orchestra.run.smithi082.stdout:(130/152): python3-kubernetes-26.1.0-2.el9.noar 12 MB/s | 1.0 MB 00:00 2024-05-03T21:08:16.278 INFO:teuthology.orchestra.run.smithi138.stdout:(28/152): ceph-grafana-dashboards-19.0.0-3450.g 586 kB/s | 34 kB 00:00 2024-05-03T21:08:16.290 INFO:teuthology.orchestra.run.smithi067.stdout:(62/152): libzstd-devel-1.5.1-2.el9.x86_64.rpm 75 kB/s | 47 kB 00:00 2024-05-03T21:08:16.300 INFO:teuthology.orchestra.run.smithi082.stdout:(131/152): python3-logutils-0.3.5-21.el9.noarch 552 kB/s | 46 kB 00:00 2024-05-03T21:08:16.325 INFO:teuthology.orchestra.run.smithi082.stdout:(132/152): python3-more-itertools-8.12.0-2.el9. 940 kB/s | 79 kB 00:00 2024-05-03T21:08:16.345 INFO:teuthology.orchestra.run.smithi138.stdout:(29/152): ceph-mgr-cephadm-19.0.0-3450.g9a047fe 2.1 MB/s | 142 kB 00:00 2024-05-03T21:08:16.350 INFO:teuthology.orchestra.run.smithi082.stdout:(133/152): python3-natsort-7.1.1-5.el9.noarch.r 764 kB/s | 58 kB 00:00 2024-05-03T21:08:16.384 INFO:teuthology.orchestra.run.smithi082.stdout:(134/152): python3-pecan-1.4.2-3.el9.noarch.rpm 3.2 MB/s | 272 kB 00:00 2024-05-03T21:08:16.434 INFO:teuthology.orchestra.run.smithi082.stdout:(135/152): python3-portend-3.1.0-2.el9.noarch.r 151 kB/s | 16 kB 00:00 2024-05-03T21:08:16.468 INFO:teuthology.orchestra.run.smithi082.stdout:(136/152): python3-pyOpenSSL-21.0.0-1.el9.noarc 767 kB/s | 90 kB 00:00 2024-05-03T21:08:16.493 INFO:teuthology.orchestra.run.smithi082.stdout:(137/152): python3-pyperclip-1.8.2-2.el9.noarch 259 kB/s | 28 kB 00:00 2024-05-03T21:08:16.518 INFO:teuthology.orchestra.run.smithi082.stdout:(138/152): python3-repoze-lru-0.7-16.el9.noarch 367 kB/s | 31 kB 00:00 2024-05-03T21:08:16.543 INFO:teuthology.orchestra.run.smithi082.stdout:(139/152): python3-routes-2.5.1-5.el9.noarch.rp 2.4 MB/s | 188 kB 00:00 2024-05-03T21:08:16.569 INFO:teuthology.orchestra.run.smithi082.stdout:(140/152): python3-rsa-4.9-2.el9.noarch.rpm 783 kB/s | 59 kB 00:00 2024-05-03T21:08:16.602 INFO:teuthology.orchestra.run.smithi082.stdout:(141/152): python3-tempora-5.0.0-2.el9.noarch.r 426 kB/s | 36 kB 00:00 2024-05-03T21:08:16.628 INFO:teuthology.orchestra.run.smithi082.stdout:(142/152): python3-typing-extensions-4.4.0-2.el 608 kB/s | 51 kB 00:00 2024-05-03T21:08:16.653 INFO:teuthology.orchestra.run.smithi082.stdout:(143/152): python3-webob-1.8.7-6.el9.noarch.rpm 2.7 MB/s | 230 kB 00:00 2024-05-03T21:08:16.687 INFO:teuthology.orchestra.run.smithi082.stdout:(144/152): python3-websocket-client-1.2.3-2.el9 1.5 MB/s | 90 kB 00:00 2024-05-03T21:08:16.696 INFO:teuthology.orchestra.run.smithi138.stdout:(30/152): ceph-mgr-dashboard-19.0.0-3450.g9a047 10 MB/s | 3.5 MB 00:00 2024-05-03T21:08:16.799 INFO:teuthology.orchestra.run.smithi082.stdout:(145/152): python3-werkzeug-2.0.3-3.el9.1.noarc 2.4 MB/s | 427 kB 00:00 2024-05-03T21:08:16.862 INFO:teuthology.orchestra.run.smithi082.stdout:(146/152): python3-zc-lockfile-2.0-10.el9.noarc 95 kB/s | 20 kB 00:00 2024-05-03T21:08:16.888 INFO:teuthology.orchestra.run.smithi082.stdout:(147/152): re2-20211101-3.el9.x86_64.rpm 954 kB/s | 192 kB 00:00 2024-05-03T21:08:16.930 INFO:teuthology.orchestra.run.smithi082.stdout:(148/152): thrift-0.15.0-2.el9.x86_64.rpm 12 MB/s | 1.6 MB 00:00 2024-05-03T21:08:17.114 INFO:teuthology.orchestra.run.smithi082.stdout:(149/152): openssl-3.2.1-1.el9.x86_64.rpm 7.4 MB/s | 1.3 MB 00:00 2024-05-03T21:08:17.115 INFO:teuthology.orchestra.run.smithi067.stdout:(63/152): lttng-ust-2.12.0-6.el9.x86_64.rpm 325 kB/s | 292 kB 00:00 2024-05-03T21:08:17.140 INFO:teuthology.orchestra.run.smithi067.stdout:(64/152): lua-5.4.4-4.el9.x86_64.rpm 222 kB/s | 188 kB 00:00 2024-05-03T21:08:17.464 INFO:teuthology.orchestra.run.smithi082.stdout:(150/152): openssl-libs-3.2.1-1.el9.x86_64.rpm 6.8 MB/s | 2.4 MB 00:00 2024-05-03T21:08:17.471 INFO:teuthology.orchestra.run.smithi138.stdout:(31/152): ceph-common-19.0.0-3450.g9a047fe7.el9 1.6 MB/s | 21 MB 00:13 2024-05-03T21:08:17.741 INFO:teuthology.orchestra.run.smithi067.stdout:(65/152): openblas-0.3.21-2.el9.x86_64.rpm 56 kB/s | 35 kB 00:00 2024-05-03T21:08:17.754 INFO:teuthology.orchestra.run.smithi138.stdout:(32/152): ceph-mgr-diskprediction-local-19.0.0- 7.0 MB/s | 7.4 MB 00:01 2024-05-03T21:08:17.780 INFO:teuthology.orchestra.run.smithi138.stdout:(33/152): ceph-mgr-modules-core-19.0.0-3450.g9a 889 kB/s | 274 kB 00:00 2024-05-03T21:08:17.797 INFO:teuthology.orchestra.run.smithi082.stdout:(151/152): librbd1-19.0.0-3450.g9a047fe7.el9.x8 3.5 MB/s | 3.1 MB 00:00 2024-05-03T21:08:17.822 INFO:teuthology.orchestra.run.smithi138.stdout:(34/152): ceph-mgr-rook-19.0.0-3450.g9a047fe7.e 739 kB/s | 49 kB 00:00 2024-05-03T21:08:17.847 INFO:teuthology.orchestra.run.smithi138.stdout:(35/152): ceph-prometheus-alerts-19.0.0-3450.g9 246 kB/s | 16 kB 00:00 2024-05-03T21:08:17.897 INFO:teuthology.orchestra.run.smithi138.stdout:(36/152): ceph-volume-19.0.0-3450.g9a047fe7.el9 3.6 MB/s | 273 kB 00:00 2024-05-03T21:08:17.973 INFO:teuthology.orchestra.run.smithi138.stdout:(37/152): cephfs-top-19.0.0-3450.g9a047fe7.el9. 243 kB/s | 18 kB 00:00 2024-05-03T21:08:18.140 INFO:teuthology.orchestra.run.smithi138.stdout:(38/152): librgw2-19.0.0-3450.g9a047fe7.el9.x86 1.4 MB/s | 5.4 MB 00:03 2024-05-03T21:08:18.165 INFO:teuthology.orchestra.run.smithi138.stdout:(39/152): fuse-2.9.9-15.el9.x86_64.rpm 419 kB/s | 80 kB 00:00 2024-05-03T21:08:18.207 INFO:teuthology.orchestra.run.smithi138.stdout:(40/152): cephadm-19.0.0-3450.g9a047fe7.el9.noa 2.1 MB/s | 763 kB 00:00 2024-05-03T21:08:18.231 INFO:teuthology.orchestra.run.smithi082.stdout:(152/152): librados2-19.0.0-3450.g9a047fe7.el9. 2.5 MB/s | 3.4 MB 00:01 2024-05-03T21:08:18.234 INFO:teuthology.orchestra.run.smithi082.stdout:-------------------------------------------------------------------------------- 2024-05-03T21:08:18.234 INFO:teuthology.orchestra.run.smithi082.stdout:Total 11 MB/s | 222 MB 00:19 2024-05-03T21:08:18.274 INFO:teuthology.orchestra.run.smithi138.stdout:(41/152): libquadmath-11.4.1-3.el9.x86_64.rpm 1.8 MB/s | 195 kB 00:00 2024-05-03T21:08:18.341 INFO:teuthology.orchestra.run.smithi067.stdout:(66/152): openblas-openmp-0.3.21-2.el9.x86_64.r 3.9 MB/s | 4.7 MB 00:01 2024-05-03T21:08:18.416 INFO:teuthology.orchestra.run.smithi138.stdout:(42/152): python3-cffi-1.14.5-5.el9.x86_64.rpm 1.2 MB/s | 253 kB 00:00 2024-05-03T21:08:18.483 INFO:teuthology.orchestra.run.smithi138.stdout:(43/152): python3-cryptography-36.0.1-4.el9.x86 5.9 MB/s | 1.2 MB 00:00 2024-05-03T21:08:18.516 INFO:teuthology.orchestra.run.smithi138.stdout:(44/152): libgfortran-11.4.1-3.el9.x86_64.rpm 2.1 MB/s | 803 kB 00:00 2024-05-03T21:08:18.542 INFO:teuthology.orchestra.run.smithi138.stdout:(45/152): python3-ply-3.11-14.el9.noarch.rpm 849 kB/s | 106 kB 00:00 2024-05-03T21:08:18.567 INFO:teuthology.orchestra.run.smithi138.stdout:(46/152): python3-pycparser-2.20-6.el9.noarch.r 1.6 MB/s | 135 kB 00:00 2024-05-03T21:08:18.592 INFO:teuthology.orchestra.run.smithi138.stdout:(47/152): python3-requests-2.25.1-8.el9.noarch. 1.6 MB/s | 125 kB 00:00 2024-05-03T21:08:18.626 INFO:teuthology.orchestra.run.smithi138.stdout:(48/152): python3-urllib3-1.26.5-5.el9.noarch.r 2.5 MB/s | 215 kB 00:00 2024-05-03T21:08:18.735 INFO:teuthology.orchestra.run.smithi138.stdout:(49/152): boost-program-options-1.75.0-8.el9.x8 987 kB/s | 107 kB 00:00 2024-05-03T21:08:18.777 INFO:teuthology.orchestra.run.smithi138.stdout:(50/152): automake-1.16.2-8.el9.noarch.rpm 3.2 MB/s | 689 kB 00:00 2024-05-03T21:08:18.802 INFO:teuthology.orchestra.run.smithi138.stdout:(51/152): elfutils-libelf-devel-0.190-2.el9.x86 321 kB/s | 21 kB 00:00 2024-05-03T21:08:18.836 INFO:teuthology.orchestra.run.smithi138.stdout:(52/152): bison-3.7.4-5.el9.x86_64.rpm 3.8 MB/s | 943 kB 00:00 2024-05-03T21:08:18.861 INFO:teuthology.orchestra.run.smithi138.stdout:(53/152): flexiblas-3.0.4-8.el9.x86_64.rpm 559 kB/s | 33 kB 00:00 2024-05-03T21:08:18.886 INFO:teuthology.orchestra.run.smithi138.stdout:(54/152): flex-2.6.4-9.el9.x86_64.rpm 2.8 MB/s | 313 kB 00:00 2024-05-03T21:08:18.903 INFO:teuthology.orchestra.run.smithi138.stdout:(55/152): flexiblas-openblas-openmp-3.0.4-8.el9 434 kB/s | 18 kB 00:00 2024-05-03T21:08:18.920 INFO:teuthology.orchestra.run.smithi138.stdout:(56/152): libaio-devel-0.3.111-13.el9.x86_64.rp 333 kB/s | 11 kB 00:00 2024-05-03T21:08:18.946 INFO:teuthology.orchestra.run.smithi138.stdout:(57/152): libpmemobj-1.12.1-1.el9.x86_64.rpm 3.7 MB/s | 160 kB 00:00 2024-05-03T21:08:18.962 INFO:teuthology.orchestra.run.smithi138.stdout:(58/152): librabbitmq-0.11.0-7.el9.x86_64.rpm 1.1 MB/s | 45 kB 00:00 2024-05-03T21:08:18.966 INFO:teuthology.orchestra.run.smithi067.stdout:(67/152): openssl-devel-3.2.1-1.el9.x86_64.rpm 3.6 MB/s | 4.4 MB 00:01 2024-05-03T21:08:18.996 INFO:teuthology.orchestra.run.smithi138.stdout:(59/152): librdkafka-1.6.1-102.el9.x86_64.rpm 13 MB/s | 662 kB 00:00 2024-05-03T21:08:19.046 INFO:teuthology.orchestra.run.smithi138.stdout:(60/152): flexiblas-netlib-3.0.4-8.el9.x86_64.r 14 MB/s | 3.0 MB 00:00 2024-05-03T21:08:19.072 INFO:teuthology.orchestra.run.smithi138.stdout:(61/152): libuuid-devel-2.37.4-18.el9.x86_64.rp 338 kB/s | 25 kB 00:00 2024-05-03T21:08:19.097 INFO:teuthology.orchestra.run.smithi138.stdout:(62/152): libzstd-devel-1.5.1-2.el9.x86_64.rpm 948 kB/s | 47 kB 00:00 2024-05-03T21:08:19.122 INFO:teuthology.orchestra.run.smithi138.stdout:(63/152): lttng-ust-2.12.0-6.el9.x86_64.rpm 5.7 MB/s | 292 kB 00:00 2024-05-03T21:08:19.156 INFO:teuthology.orchestra.run.smithi138.stdout:(64/152): libtool-2.4.6-45.el9.x86_64.rpm 2.9 MB/s | 582 kB 00:00 2024-05-03T21:08:19.181 INFO:teuthology.orchestra.run.smithi138.stdout:(65/152): openblas-0.3.21-2.el9.x86_64.rpm 600 kB/s | 35 kB 00:00 2024-05-03T21:08:19.206 INFO:teuthology.orchestra.run.smithi138.stdout:(66/152): lua-5.4.4-4.el9.x86_64.rpm 1.7 MB/s | 188 kB 00:00 2024-05-03T21:08:19.274 INFO:teuthology.orchestra.run.smithi138.stdout:(67/152): postgresql-13.14-1.el9.x86_64.rpm 24 MB/s | 1.6 MB 00:00 2024-05-03T21:08:19.341 INFO:teuthology.orchestra.run.smithi138.stdout:(68/152): openssl-devel-3.2.1-1.el9.x86_64.rpm 28 MB/s | 4.4 MB 00:00 2024-05-03T21:08:19.342 INFO:teuthology.orchestra.run.smithi082.stdout:Running transaction check 2024-05-03T21:08:19.408 INFO:teuthology.orchestra.run.smithi138.stdout:(69/152): openblas-openmp-0.3.21-2.el9.x86_64.r 19 MB/s | 4.7 MB 00:00 2024-05-03T21:08:19.434 INFO:teuthology.orchestra.run.smithi138.stdout:(70/152): postgresql-private-libs-13.14-1.el9.x 1.4 MB/s | 133 kB 00:00 2024-05-03T21:08:19.475 INFO:teuthology.orchestra.run.smithi138.stdout:(71/152): postgresql-contrib-13.14-1.el9.x86_64 4.4 MB/s | 911 kB 00:00 2024-05-03T21:08:19.483 INFO:teuthology.orchestra.run.smithi082.stdout:Transaction check succeeded. 2024-05-03T21:08:19.483 INFO:teuthology.orchestra.run.smithi082.stdout:Running transaction test 2024-05-03T21:08:19.508 INFO:teuthology.orchestra.run.smithi067.stdout:(68/152): ceph-test-19.0.0-3450.g9a047fe7.el9.x 3.3 MB/s | 49 MB 00:14 2024-05-03T21:08:19.517 INFO:teuthology.orchestra.run.smithi138.stdout:(72/152): protobuf-3.14.0-13.el9.x86_64.rpm 12 MB/s | 1.0 MB 00:00 2024-05-03T21:08:19.542 INFO:teuthology.orchestra.run.smithi138.stdout:(73/152): python3-attrs-20.3.0-7.el9.noarch.rpm 1.3 MB/s | 87 kB 00:00 2024-05-03T21:08:19.584 INFO:teuthology.orchestra.run.smithi138.stdout:(74/152): python3-devel-3.9.18-3.el9.x86_64.rpm 5.7 MB/s | 244 kB 00:00 2024-05-03T21:08:19.661 INFO:teuthology.orchestra.run.smithi138.stdout:(75/152): postgresql-server-13.14-1.el9.x86_64. 23 MB/s | 5.8 MB 00:00 2024-05-03T21:08:19.685 INFO:teuthology.orchestra.run.smithi138.stdout:(76/152): python3-jinja2-2.11.3-5.el9.noarch.rp 2.4 MB/s | 248 kB 00:00 2024-05-03T21:08:19.710 INFO:teuthology.orchestra.run.smithi138.stdout:(77/152): python3-jsonpatch-1.21-16.el9.noarch. 509 kB/s | 26 kB 00:00 2024-05-03T21:08:19.735 INFO:teuthology.orchestra.run.smithi138.stdout:(78/152): python3-jsonpointer-2.0-4.el9.noarch. 382 kB/s | 19 kB 00:00 2024-05-03T21:08:19.761 INFO:teuthology.orchestra.run.smithi138.stdout:(79/152): python3-mako-1.1.4-6.el9.noarch.rpm 3.3 MB/s | 172 kB 00:00 2024-05-03T21:08:19.786 INFO:teuthology.orchestra.run.smithi138.stdout:(80/152): python3-markupsafe-1.1.1-12.el9.x86_6 692 kB/s | 35 kB 00:00 2024-05-03T21:08:19.833 INFO:teuthology.orchestra.run.smithi067.stdout:(69/152): postgresql-13.14-1.el9.x86_64.rpm 1.1 MB/s | 1.6 MB 00:01 2024-05-03T21:08:19.870 INFO:teuthology.orchestra.run.smithi138.stdout:(81/152): python3-babel-2.9.1-2.el9.noarch.rpm 17 MB/s | 6.0 MB 00:00 2024-05-03T21:08:19.895 INFO:teuthology.orchestra.run.smithi138.stdout:(82/152): python3-numpy-f2py-1.20.1-5.el9.x86_6 2.3 MB/s | 250 kB 00:00 2024-05-03T21:08:19.920 INFO:teuthology.orchestra.run.smithi138.stdout:(83/152): python3-oauthlib-3.1.1-5.el9.noarch.r 4.3 MB/s | 222 kB 00:00 2024-05-03T21:08:19.945 INFO:teuthology.orchestra.run.smithi138.stdout:(84/152): python3-packaging-20.9-5.el9.noarch.r 1.5 MB/s | 77 kB 00:00 2024-05-03T21:08:19.971 INFO:teuthology.orchestra.run.smithi138.stdout:(85/152): python3-prettytable-0.7.2-27.el9.noar 843 kB/s | 42 kB 00:00 2024-05-03T21:08:19.996 INFO:teuthology.orchestra.run.smithi138.stdout:(86/152): python3-protobuf-3.14.0-13.el9.noarch 5.3 MB/s | 269 kB 00:00 2024-05-03T21:08:20.021 INFO:teuthology.orchestra.run.smithi138.stdout:(87/152): python3-pyasn1-0.4.8-6.el9.noarch.rpm 3.1 MB/s | 159 kB 00:00 2024-05-03T21:08:20.047 INFO:teuthology.orchestra.run.smithi138.stdout:(88/152): python3-pyasn1-modules-0.4.8-6.el9.no 5.4 MB/s | 279 kB 00:00 2024-05-03T21:08:20.072 INFO:teuthology.orchestra.run.smithi138.stdout:(89/152): python3-pytz-2021.1-5.el9.noarch.rpm 1.0 MB/s | 51 kB 00:00 2024-05-03T21:08:20.089 INFO:teuthology.orchestra.run.smithi138.stdout:(90/152): python3-requests-oauthlib-1.3.0-12.el 1.3 MB/s | 54 kB 00:00 2024-05-03T21:08:20.164 INFO:teuthology.orchestra.run.smithi138.stdout:(91/152): python3-numpy-1.20.1-5.el9.x86_64.rpm 13 MB/s | 5.1 MB 00:00 2024-05-03T21:08:20.181 INFO:teuthology.orchestra.run.smithi138.stdout:(92/152): python3-toml-0.10.2-6.el9.noarch.rpm 453 kB/s | 42 kB 00:00 2024-05-03T21:08:20.198 INFO:teuthology.orchestra.run.smithi138.stdout:(93/152): python3-wcwidth-0.2.5-8.el9.noarch.rp 1.3 MB/s | 44 kB 00:00 2024-05-03T21:08:20.223 INFO:teuthology.orchestra.run.smithi138.stdout:(94/152): qatlib-24.02.0-1.el9.x86_64.rpm 5.1 MB/s | 221 kB 00:00 2024-05-03T21:08:20.249 INFO:teuthology.orchestra.run.smithi138.stdout:(95/152): qatlib-service-24.02.0-1.el9.x86_64.r 717 kB/s | 36 kB 00:00 2024-05-03T21:08:20.258 INFO:teuthology.orchestra.run.smithi067.stdout:(70/152): postgresql-private-libs-13.14-1.el9.x 178 kB/s | 133 kB 00:00 2024-05-03T21:08:20.274 INFO:teuthology.orchestra.run.smithi138.stdout:(96/152): qatzip-libs-1.2.0-1.el9.x86_64.rpm 943 kB/s | 47 kB 00:00 2024-05-03T21:08:20.299 INFO:teuthology.orchestra.run.smithi138.stdout:(97/152): socat-1.7.4.1-5.el9.x86_64.rpm 5.9 MB/s | 305 kB 00:00 2024-05-03T21:08:20.325 INFO:teuthology.orchestra.run.smithi138.stdout:(98/152): uuid-1.6.2-55.el9.x86_64.rpm 1.1 MB/s | 58 kB 00:00 2024-05-03T21:08:20.350 INFO:teuthology.orchestra.run.smithi138.stdout:(99/152): xfsprogs-devel-6.3.0-1.el9.x86_64.rpm 1.8 MB/s | 94 kB 00:00 2024-05-03T21:08:20.367 INFO:teuthology.orchestra.run.smithi067.stdout:(71/152): postgresql-contrib-13.14-1.el9.x86_64 651 kB/s | 911 kB 00:01 2024-05-03T21:08:20.375 INFO:teuthology.orchestra.run.smithi138.stdout:(100/152): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.2 MB/s | 64 kB 00:00 2024-05-03T21:08:20.401 INFO:teuthology.orchestra.run.smithi138.stdout:(101/152): zlib-devel-1.2.11-41.el9.x86_64.rpm 895 kB/s | 45 kB 00:00 2024-05-03T21:08:20.643 INFO:teuthology.orchestra.run.smithi138.stdout:(102/152): python3-scipy-1.6.2-8.el9.x86_64.rpm 29 MB/s | 16 MB 00:00 2024-05-03T21:08:20.668 INFO:teuthology.orchestra.run.smithi138.stdout:(103/152): lua-devel-5.4.4-4.el9.x86_64.rpm 76 kB/s | 22 kB 00:00 2024-05-03T21:08:20.777 INFO:teuthology.orchestra.run.smithi138.stdout:(104/152): gperftools-libs-2.9.1-2.el9.x86_64.r 2.8 MB/s | 309 kB 00:00 2024-05-03T21:08:20.802 INFO:teuthology.orchestra.run.smithi138.stdout:(105/152): protobuf-compiler-3.14.0-13.el9.x86_ 2.1 MB/s | 863 kB 00:00 2024-05-03T21:08:20.827 INFO:teuthology.orchestra.run.smithi138.stdout:(106/152): grpc-data-1.46.7-10.el9.noarch.rpm 386 kB/s | 19 kB 00:00 2024-05-03T21:08:20.853 INFO:teuthology.orchestra.run.smithi138.stdout:(107/152): abseil-cpp-20211102.0-3.el9.x86_64.r 2.6 MB/s | 548 kB 00:00 2024-05-03T21:08:20.878 INFO:teuthology.orchestra.run.smithi138.stdout:(108/152): libarrow-doc-9.0.0-11.el9.noarch.rpm 518 kB/s | 26 kB 00:00 2024-05-03T21:08:20.903 INFO:teuthology.orchestra.run.smithi138.stdout:(109/152): liboath-2.6.7-2.el9.x86_64.rpm 970 kB/s | 49 kB 00:00 2024-05-03T21:08:20.928 INFO:teuthology.orchestra.run.smithi138.stdout:(110/152): libunwind-1.6.2-1.el9.x86_64.rpm 1.3 MB/s | 67 kB 00:00 2024-05-03T21:08:20.954 INFO:teuthology.orchestra.run.smithi138.stdout:(111/152): luarocks-3.9.2-1.el9.noarch.rpm 2.9 MB/s | 151 kB 00:00 2024-05-03T21:08:20.996 INFO:teuthology.orchestra.run.smithi138.stdout:(112/152): parquet-libs-9.0.0-11.el9.x86_64.rpm 12 MB/s | 839 kB 00:00 2024-05-03T21:08:21.017 INFO:teuthology.orchestra.run.smithi067.stdout:(72/152): postgresql-server-13.14-1.el9.x86_64. 4.9 MB/s | 5.8 MB 00:01 2024-05-03T21:08:21.038 INFO:teuthology.orchestra.run.smithi138.stdout:(113/152): python3-asyncssh-2.13.2-5.el9.noarch 6.4 MB/s | 548 kB 00:00 2024-05-03T21:08:21.059 INFO:teuthology.orchestra.run.smithi067.stdout:(73/152): python3-attrs-20.3.0-7.el9.noarch.rpm 126 kB/s | 87 kB 00:00 2024-05-03T21:08:21.063 INFO:teuthology.orchestra.run.smithi138.stdout:(114/152): python3-bcrypt-3.2.2-1.el9.x86_64.rp 649 kB/s | 43 kB 00:00 2024-05-03T21:08:21.088 INFO:teuthology.orchestra.run.smithi138.stdout:(115/152): python3-cachetools-4.2.4-1.el9.noarc 639 kB/s | 32 kB 00:00 2024-05-03T21:08:21.114 INFO:teuthology.orchestra.run.smithi138.stdout:(116/152): python3-certifi-2021.10.8-2.el9.noar 299 kB/s | 15 kB 00:00 2024-05-03T21:08:21.139 INFO:teuthology.orchestra.run.smithi138.stdout:(117/152): python3-cheroot-8.6.0-4.el9.noarch.r 3.4 MB/s | 172 kB 00:00 2024-05-03T21:08:21.164 INFO:teuthology.orchestra.run.smithi138.stdout:(118/152): python3-cherrypy-18.6.1-2.el9.noarch 7.0 MB/s | 358 kB 00:00 2024-05-03T21:08:21.190 INFO:teuthology.orchestra.run.smithi138.stdout:(119/152): python3-cmd2-2.4.2-2.el9.noarch.rpm 5.9 MB/s | 303 kB 00:00 2024-05-03T21:08:21.215 INFO:teuthology.orchestra.run.smithi138.stdout:(120/152): python3-colorama-0.4.4-12.el9.noarch 702 kB/s | 35 kB 00:00 2024-05-03T21:08:21.240 INFO:teuthology.orchestra.run.smithi138.stdout:(121/152): python3-google-auth-2.29.0-1.el9.noa 4.0 MB/s | 206 kB 00:00 2024-05-03T21:08:21.245 INFO:teuthology.orchestra.run.smithi082.stdout:Transaction test succeeded. 2024-05-03T21:08:21.246 INFO:teuthology.orchestra.run.smithi082.stdout:Running transaction 2024-05-03T21:08:21.259 INFO:teuthology.orchestra.run.smithi067.stdout:(74/152): protobuf-3.14.0-13.el9.x86_64.rpm 1.0 MB/s | 1.0 MB 00:01 2024-05-03T21:08:21.316 INFO:teuthology.orchestra.run.smithi138.stdout:(122/152): libarrow-9.0.0-11.el9.x86_64.rpm 8.6 MB/s | 4.4 MB 00:00 2024-05-03T21:08:21.341 INFO:teuthology.orchestra.run.smithi138.stdout:(123/152): python3-grpcio-tools-1.46.7-10.el9.x 1.4 MB/s | 144 kB 00:00 2024-05-03T21:08:21.366 INFO:teuthology.orchestra.run.smithi138.stdout:(124/152): python3-jaraco-8.2.1-3.el9.noarch.rp 213 kB/s | 11 kB 00:00 2024-05-03T21:08:21.392 INFO:teuthology.orchestra.run.smithi138.stdout:(125/152): python3-jaraco-classes-3.2.1-5.el9.n 351 kB/s | 18 kB 00:00 2024-05-03T21:08:21.417 INFO:teuthology.orchestra.run.smithi138.stdout:(126/152): python3-jaraco-collections-3.0.0-8.e 460 kB/s | 23 kB 00:00 2024-05-03T21:08:21.442 INFO:teuthology.orchestra.run.smithi138.stdout:(127/152): python3-jaraco-functools-3.5.0-2.el9 386 kB/s | 19 kB 00:00 2024-05-03T21:08:21.467 INFO:teuthology.orchestra.run.smithi138.stdout:(128/152): python3-jaraco-text-3.2.0-6.el9.noar 393 kB/s | 20 kB 00:00 2024-05-03T21:08:21.518 INFO:teuthology.orchestra.run.smithi138.stdout:(129/152): python3-grpcio-1.46.7-10.el9.x86_64. 6.7 MB/s | 2.0 MB 00:00 2024-05-03T21:08:21.543 INFO:teuthology.orchestra.run.smithi138.stdout:(130/152): python3-logutils-0.3.5-21.el9.noarch 612 kB/s | 46 kB 00:00 2024-05-03T21:08:21.569 INFO:teuthology.orchestra.run.smithi138.stdout:(131/152): python3-more-itertools-8.12.0-2.el9. 1.5 MB/s | 79 kB 00:00 2024-05-03T21:08:21.585 INFO:teuthology.orchestra.run.smithi138.stdout:(132/152): python3-natsort-7.1.1-5.el9.noarch.r 1.4 MB/s | 58 kB 00:00 2024-05-03T21:08:21.619 INFO:teuthology.orchestra.run.smithi138.stdout:(133/152): python3-pecan-1.4.2-3.el9.noarch.rpm 5.3 MB/s | 272 kB 00:00 2024-05-03T21:08:21.636 INFO:teuthology.orchestra.run.smithi138.stdout:(134/152): python3-portend-3.1.0-2.el9.noarch.r 326 kB/s | 16 kB 00:00 2024-05-03T21:08:21.662 INFO:teuthology.orchestra.run.smithi138.stdout:(135/152): python3-pyOpenSSL-21.0.0-1.el9.noarc 2.1 MB/s | 90 kB 00:00 2024-05-03T21:08:21.679 INFO:teuthology.orchestra.run.smithi138.stdout:(136/152): python3-pyperclip-1.8.2-2.el9.noarch 673 kB/s | 28 kB 00:00 2024-05-03T21:08:21.704 INFO:teuthology.orchestra.run.smithi138.stdout:(137/152): python3-repoze-lru-0.7-16.el9.noarch 734 kB/s | 31 kB 00:00 2024-05-03T21:08:21.738 INFO:teuthology.orchestra.run.smithi138.stdout:(138/152): python3-kubernetes-26.1.0-2.el9.noar 3.5 MB/s | 1.0 MB 00:00 2024-05-03T21:08:21.754 INFO:teuthology.orchestra.run.smithi138.stdout:(139/152): python3-rsa-4.9-2.el9.noarch.rpm 1.2 MB/s | 59 kB 00:00 2024-05-03T21:08:21.771 INFO:teuthology.orchestra.run.smithi138.stdout:(140/152): python3-tempora-5.0.0-2.el9.noarch.r 1.0 MB/s | 36 kB 00:00 2024-05-03T21:08:21.797 INFO:teuthology.orchestra.run.smithi138.stdout:(141/152): python3-routes-2.5.1-5.el9.noarch.rp 1.6 MB/s | 188 kB 00:00 2024-05-03T21:08:21.814 INFO:teuthology.orchestra.run.smithi138.stdout:(142/152): python3-typing-extensions-4.4.0-2.el 867 kB/s | 51 kB 00:00 2024-05-03T21:08:21.839 INFO:teuthology.orchestra.run.smithi138.stdout:(143/152): python3-websocket-client-1.2.3-2.el9 2.1 MB/s | 90 kB 00:00 2024-05-03T21:08:21.864 INFO:teuthology.orchestra.run.smithi138.stdout:(144/152): python3-webob-1.8.7-6.el9.noarch.rpm 2.4 MB/s | 230 kB 00:00 2024-05-03T21:08:21.890 INFO:teuthology.orchestra.run.smithi138.stdout:(145/152): python3-zc-lockfile-2.0-10.el9.noarc 397 kB/s | 20 kB 00:00 2024-05-03T21:08:21.901 INFO:teuthology.orchestra.run.smithi067.stdout:(75/152): python3-devel-3.9.18-3.el9.x86_64.rpm 290 kB/s | 244 kB 00:00 2024-05-03T21:08:21.915 INFO:teuthology.orchestra.run.smithi138.stdout:(146/152): re2-20211101-3.el9.x86_64.rpm 3.7 MB/s | 192 kB 00:00 2024-05-03T21:08:21.940 INFO:teuthology.orchestra.run.smithi138.stdout:(147/152): python3-werkzeug-2.0.3-3.el9.1.noarc 3.3 MB/s | 427 kB 00:00 2024-05-03T21:08:22.015 INFO:teuthology.orchestra.run.smithi138.stdout:(148/152): thrift-0.15.0-2.el9.x86_64.rpm 13 MB/s | 1.6 MB 00:00 2024-05-03T21:08:22.118 INFO:teuthology.orchestra.run.smithi067.stdout:(76/152): python3-jinja2-2.11.3-5.el9.noarch.rp 309 kB/s | 248 kB 00:00 2024-05-03T21:08:22.224 INFO:teuthology.orchestra.run.smithi138.stdout:(149/152): openssl-3.2.1-1.el9.x86_64.rpm 6.5 MB/s | 1.3 MB 00:00 2024-05-03T21:08:22.310 INFO:teuthology.orchestra.run.smithi067.stdout:(77/152): python3-babel-2.9.1-2.el9.noarch.rpm 4.6 MB/s | 6.0 MB 00:01 2024-05-03T21:08:22.425 INFO:teuthology.orchestra.run.smithi138.stdout:(150/152): librados2-19.0.0-3450.g9a047fe7.el9. 6.7 MB/s | 3.4 MB 00:00 2024-05-03T21:08:22.525 INFO:teuthology.orchestra.run.smithi138.stdout:(151/152): openssl-libs-3.2.1-1.el9.x86_64.rpm 8.0 MB/s | 2.4 MB 00:00 2024-05-03T21:08:22.577 INFO:teuthology.orchestra.run.smithi067.stdout:(78/152): python3-jsonpatch-1.21-16.el9.noarch. 38 kB/s | 26 kB 00:00 2024-05-03T21:08:22.711 INFO:teuthology.orchestra.run.smithi067.stdout:(79/152): python3-jsonpointer-2.0-4.el9.noarch. 32 kB/s | 19 kB 00:00 2024-05-03T21:08:23.083 INFO:teuthology.orchestra.run.smithi138.stdout:(152/152): librbd1-19.0.0-3450.g9a047fe7.el9.x8 2.7 MB/s | 3.1 MB 00:01 2024-05-03T21:08:23.086 INFO:teuthology.orchestra.run.smithi138.stdout:-------------------------------------------------------------------------------- 2024-05-03T21:08:23.086 INFO:teuthology.orchestra.run.smithi138.stdout:Total 11 MB/s | 222 MB 00:19 2024-05-03T21:08:23.344 INFO:teuthology.orchestra.run.smithi067.stdout:(80/152): python3-mako-1.1.4-6.el9.noarch.rpm 167 kB/s | 172 kB 00:01 2024-05-03T21:08:23.370 INFO:teuthology.orchestra.run.smithi067.stdout:(81/152): python3-markupsafe-1.1.1-12.el9.x86_6 45 kB/s | 35 kB 00:00 2024-05-03T21:08:24.037 INFO:teuthology.orchestra.run.smithi067.stdout:(82/152): python3-numpy-1.20.1-5.el9.x86_64.rpm 3.8 MB/s | 5.1 MB 00:01 2024-05-03T21:08:24.162 INFO:teuthology.orchestra.run.smithi067.stdout:(83/152): python3-oauthlib-3.1.1-5.el9.noarch.r 281 kB/s | 222 kB 00:00 2024-05-03T21:08:24.187 INFO:teuthology.orchestra.run.smithi067.stdout:(84/152): python3-numpy-f2py-1.20.1-5.el9.x86_6 298 kB/s | 250 kB 00:00 2024-05-03T21:08:24.205 INFO:teuthology.orchestra.run.smithi138.stdout:Running transaction check 2024-05-03T21:08:24.346 INFO:teuthology.orchestra.run.smithi138.stdout:Transaction check succeeded. 2024-05-03T21:08:24.346 INFO:teuthology.orchestra.run.smithi138.stdout:Running transaction test 2024-05-03T21:08:24.737 INFO:teuthology.orchestra.run.smithi067.stdout:(85/152): python3-packaging-20.9-5.el9.noarch.r 110 kB/s | 77 kB 00:00 2024-05-03T21:08:24.780 INFO:teuthology.orchestra.run.smithi067.stdout:(86/152): python3-prettytable-0.7.2-27.el9.noar 68 kB/s | 42 kB 00:00 2024-05-03T21:08:25.013 INFO:teuthology.orchestra.run.smithi067.stdout:(87/152): python3-protobuf-3.14.0-13.el9.noarch 326 kB/s | 269 kB 00:00 2024-05-03T21:08:25.538 INFO:teuthology.orchestra.run.smithi067.stdout:(88/152): python3-pyasn1-0.4.8-6.el9.noarch.rpm 198 kB/s | 159 kB 00:00 2024-05-03T21:08:25.697 INFO:teuthology.orchestra.run.smithi067.stdout:(89/152): python3-pytz-2021.1-5.el9.noarch.rpm 75 kB/s | 51 kB 00:00 2024-05-03T21:08:25.722 INFO:teuthology.orchestra.run.smithi067.stdout:(90/152): python3-pyasn1-modules-0.4.8-6.el9.no 296 kB/s | 279 kB 00:00 2024-05-03T21:08:26.105 INFO:teuthology.orchestra.run.smithi138.stdout:Transaction test succeeded. 2024-05-03T21:08:26.105 INFO:teuthology.orchestra.run.smithi138.stdout:Running transaction 2024-05-03T21:08:26.256 INFO:teuthology.orchestra.run.smithi067.stdout:(91/152): python3-requests-oauthlib-1.3.0-12.el 75 kB/s | 54 kB 00:00 2024-05-03T21:08:26.398 INFO:teuthology.orchestra.run.smithi067.stdout:(92/152): python3-toml-0.10.2-6.el9.noarch.rpm 62 kB/s | 42 kB 00:00 2024-05-03T21:08:26.915 INFO:teuthology.orchestra.run.smithi067.stdout:(93/152): python3-wcwidth-0.2.5-8.el9.noarch.rp 67 kB/s | 44 kB 00:00 2024-05-03T21:08:27.265 INFO:teuthology.orchestra.run.smithi067.stdout:(94/152): python3-scipy-1.6.2-8.el9.x86_64.rpm 10 MB/s | 16 MB 00:01 2024-05-03T21:08:27.315 INFO:teuthology.orchestra.run.smithi067.stdout:(95/152): qatlib-24.02.0-1.el9.x86_64.rpm 241 kB/s | 221 kB 00:00 2024-05-03T21:08:27.376 INFO:teuthology.orchestra.run.smithi082.stdout: Preparing : 1/1 2024-05-03T21:08:27.451 INFO:teuthology.orchestra.run.smithi082.stdout: Upgrading : openssl-libs-1:3.2.1-1.el9.x86_64 1/156 2024-05-03T21:08:27.489 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 2/156 2024-05-03T21:08:27.543 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 3/156 2024-05-03T21:08:27.585 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : thrift-0.15.0-2.el9.x86_64 4/156 2024-05-03T21:08:27.610 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 5/156 2024-05-03T21:08:27.653 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 6/156 2024-05-03T21:08:27.682 INFO:teuthology.orchestra.run.smithi067.stdout:(96/152): qatlib-service-24.02.0-1.el9.x86_64.r 47 kB/s | 36 kB 00:00 2024-05-03T21:08:27.957 INFO:teuthology.orchestra.run.smithi067.stdout:(97/152): qatzip-libs-1.2.0-1.el9.x86_64.rpm 71 kB/s | 47 kB 00:00 2024-05-03T21:08:27.981 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 7/156 2024-05-03T21:08:28.078 INFO:teuthology.orchestra.run.smithi082.stdout: Upgrading : librados2-2:19.0.0-3450.g9a047fe7.el9.x86_64 8/156 2024-05-03T21:08:28.149 INFO:teuthology.orchestra.run.smithi067.stdout:(98/152): socat-1.7.4.1-5.el9.x86_64.rpm 366 kB/s | 305 kB 00:00 2024-05-03T21:08:28.199 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: librados2-2:19.0.0-3450.g9a047fe7.el9.x86_64 8/156 2024-05-03T21:08:28.255 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : libcephfs2-2:19.0.0-3450.g9a047fe7.el9.x86_64 9/156 2024-05-03T21:08:28.337 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: libcephfs2-2:19.0.0-3450.g9a047fe7.el9.x86_64 9/156 2024-05-03T21:08:28.367 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-rados-2:19.0.0-3450.g9a047fe7.el9.x86_64 10/156 2024-05-03T21:08:28.513 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : postgresql-private-libs-13.14-1.el9.x86_64 11/156 2024-05-03T21:08:28.541 INFO:teuthology.orchestra.run.smithi067.stdout:(99/152): uuid-1.6.2-55.el9.x86_64.rpm 67 kB/s | 58 kB 00:00 2024-05-03T21:08:28.572 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : postgresql-13.14-1.el9.x86_64 12/156 2024-05-03T21:08:28.600 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : libcephsqlite-2:19.0.0-3450.g9a047fe7.el9.x86_64 13/156 2024-05-03T21:08:28.703 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: libcephsqlite-2:19.0.0-3450.g9a047fe7.el9.x86_64 13/156 2024-05-03T21:08:28.726 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : libradosstriper1-2:19.0.0-3450.g9a047fe7.el9.x86 14/156 2024-05-03T21:08:28.789 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: libradosstriper1-2:19.0.0-3450.g9a047fe7.el9.x86 14/156 2024-05-03T21:08:28.811 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 15/156 2024-05-03T21:08:28.831 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 15/156 2024-05-03T21:08:28.841 INFO:teuthology.orchestra.run.smithi067.stdout:(100/152): xfsprogs-devel-6.3.0-1.el9.x86_64.rp 106 kB/s | 94 kB 00:00 2024-05-03T21:08:29.000 INFO:teuthology.orchestra.run.smithi067.stdout:(101/152): lua-devel-5.4.4-4.el9.x86_64.rpm 141 kB/s | 22 kB 00:00 2024-05-03T21:08:29.019 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 16/156 2024-05-03T21:08:29.025 INFO:teuthology.orchestra.run.smithi067.stdout:(102/152): xmlstarlet-1.6.1-20.el9.x86_64.rpm 73 kB/s | 64 kB 00:00 2024-05-03T21:08:29.052 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 16/156 2024-05-03T21:08:29.150 INFO:teuthology.orchestra.run.smithi067.stdout:(103/152): protobuf-compiler-3.14.0-13.el9.x86_ 5.7 MB/s | 863 kB 00:00 2024-05-03T21:08:29.158 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : re2-1:20211101-3.el9.x86_64 17/156 2024-05-03T21:08:29.184 INFO:teuthology.orchestra.run.smithi067.stdout:(104/152): zlib-devel-1.2.11-41.el9.x86_64.rpm 73 kB/s | 45 kB 00:00 2024-05-03T21:08:29.254 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : libarrow-9.0.0-11.el9.x86_64 18/156 2024-05-03T21:08:29.285 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 19/156 2024-05-03T21:08:29.319 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : liboath-2.6.7-2.el9.x86_64 20/156 2024-05-03T21:08:29.383 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 21/156 2024-05-03T21:08:29.418 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 22/156 2024-05-03T21:08:29.462 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 23/156 2024-05-03T21:08:29.476 INFO:teuthology.orchestra.run.smithi067.stdout:(105/152): grpc-data-1.46.7-10.el9.noarch.rpm 67 kB/s | 19 kB 00:00 2024-05-03T21:08:29.494 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 24/156 2024-05-03T21:08:29.523 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : lua-5.4.4-4.el9.x86_64 25/156 2024-05-03T21:08:29.591 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 26/156 2024-05-03T21:08:29.644 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 27/156 2024-05-03T21:08:29.675 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 28/156 2024-05-03T21:08:29.711 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 29/156 2024-05-03T21:08:29.786 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 30/156 2024-05-03T21:08:29.818 INFO:teuthology.orchestra.run.smithi067.stdout:(106/152): gperftools-libs-2.9.1-2.el9.x86_64.r 463 kB/s | 309 kB 00:00 2024-05-03T21:08:29.821 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-ceph-common-2:19.0.0-3450.g9a047fe7.el9. 31/156 2024-05-03T21:08:29.865 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-ceph-argparse-2:19.0.0-3450.g9a047fe7.el 32/156 2024-05-03T21:08:29.893 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-cephfs-2:19.0.0-3450.g9a047fe7.el9.x86_6 33/156 2024-05-03T21:08:29.944 INFO:teuthology.orchestra.run.smithi067.stdout:(107/152): libarrow-doc-9.0.0-11.el9.noarch.rpm 207 kB/s | 26 kB 00:00 2024-05-03T21:08:29.962 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 34/156 2024-05-03T21:08:30.008 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : luarocks-3.9.2-1.el9.noarch 35/156 2024-05-03T21:08:30.060 INFO:teuthology.orchestra.run.smithi067.stdout:(108/152): liboath-2.6.7-2.el9.x86_64.rpm 417 kB/s | 49 kB 00:00 2024-05-03T21:08:30.082 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 36/156 2024-05-03T21:08:30.185 INFO:teuthology.orchestra.run.smithi067.stdout:(109/152): libunwind-1.6.2-1.el9.x86_64.rpm 539 kB/s | 67 kB 00:00 2024-05-03T21:08:30.214 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 37/156 2024-05-03T21:08:30.268 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 38/156 2024-05-03T21:08:30.336 INFO:teuthology.orchestra.run.smithi067.stdout:(110/152): libarrow-9.0.0-11.el9.x86_64.rpm 5.1 MB/s | 4.4 MB 00:00 2024-05-03T21:08:30.361 INFO:teuthology.orchestra.run.smithi067.stdout:(111/152): luarocks-3.9.2-1.el9.noarch.rpm 861 kB/s | 151 kB 00:00 2024-05-03T21:08:30.469 INFO:teuthology.orchestra.run.smithi067.stdout:(112/152): parquet-libs-9.0.0-11.el9.x86_64.rpm 6.1 MB/s | 839 kB 00:00 2024-05-03T21:08:30.603 INFO:teuthology.orchestra.run.smithi067.stdout:(113/152): python3-asyncssh-2.13.2-5.el9.noarch 2.2 MB/s | 548 kB 00:00 2024-05-03T21:08:30.628 INFO:teuthology.orchestra.run.smithi067.stdout:(114/152): python3-bcrypt-3.2.2-1.el9.x86_64.rp 274 kB/s | 43 kB 00:00 2024-05-03T21:08:30.720 INFO:teuthology.orchestra.run.smithi067.stdout:(115/152): python3-cachetools-4.2.4-1.el9.noarc 275 kB/s | 32 kB 00:00 2024-05-03T21:08:30.746 INFO:teuthology.orchestra.run.smithi067.stdout:(116/152): python3-certifi-2021.10.8-2.el9.noar 128 kB/s | 15 kB 00:00 2024-05-03T21:08:30.787 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-rsa-4.9-2.el9.noarch 39/156 2024-05-03T21:08:30.837 INFO:teuthology.orchestra.run.smithi067.stdout:(117/152): python3-cheroot-8.6.0-4.el9.noarch.r 1.4 MB/s | 172 kB 00:00 2024-05-03T21:08:30.863 INFO:teuthology.orchestra.run.smithi067.stdout:(118/152): python3-cherrypy-18.6.1-2.el9.noarch 3.0 MB/s | 358 kB 00:00 2024-05-03T21:08:30.864 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 40/156 2024-05-03T21:08:30.902 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 41/156 2024-05-03T21:08:30.936 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 42/156 2024-05-03T21:08:30.963 INFO:teuthology.orchestra.run.smithi067.stdout:(119/152): python3-cmd2-2.4.2-2.el9.noarch.rpm 2.4 MB/s | 303 kB 00:00 2024-05-03T21:08:30.982 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : librados-devel-2:19.0.0-3450.g9a047fe7.el9.x86_6 43/156 2024-05-03T21:08:30.988 INFO:teuthology.orchestra.run.smithi067.stdout:(120/152): python3-colorama-0.4.4-12.el9.noarch 281 kB/s | 35 kB 00:00 2024-05-03T21:08:31.023 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 44/156 2024-05-03T21:08:31.080 INFO:teuthology.orchestra.run.smithi067.stdout:(121/152): python3-google-auth-2.29.0-1.el9.noa 1.7 MB/s | 206 kB 00:00 2024-05-03T21:08:31.114 INFO:teuthology.orchestra.run.smithi067.stdout:(122/152): abseil-cpp-20211102.0-3.el9.x86_64.r 262 kB/s | 548 kB 00:02 2024-05-03T21:08:31.167 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 45/156 2024-05-03T21:08:31.227 INFO:teuthology.orchestra.run.smithi082.stdout: Upgrading : openssl-1:3.2.1-1.el9.x86_64 46/156 2024-05-03T21:08:31.231 INFO:teuthology.orchestra.run.smithi067.stdout:(123/152): python3-jaraco-8.2.1-3.el9.noarch.rp 92 kB/s | 11 kB 00:00 2024-05-03T21:08:31.273 INFO:teuthology.orchestra.run.smithi067.stdout:(124/152): python3-grpcio-1.46.7-10.el9.x86_64. 7.2 MB/s | 2.0 MB 00:00 2024-05-03T21:08:31.282 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 47/156 2024-05-03T21:08:31.348 INFO:teuthology.orchestra.run.smithi067.stdout:(125/152): python3-jaraco-classes-3.2.1-5.el9.n 151 kB/s | 18 kB 00:00 2024-05-03T21:08:31.354 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 48/156 2024-05-03T21:08:31.386 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 49/156 2024-05-03T21:08:31.390 INFO:teuthology.orchestra.run.smithi067.stdout:(126/152): python3-jaraco-collections-3.0.0-8.e 198 kB/s | 23 kB 00:00 2024-05-03T21:08:31.414 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 50/156 2024-05-03T21:08:31.461 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 51/156 2024-05-03T21:08:31.465 INFO:teuthology.orchestra.run.smithi067.stdout:(127/152): python3-jaraco-functools-3.5.0-2.el9 166 kB/s | 19 kB 00:00 2024-05-03T21:08:31.497 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 52/156 2024-05-03T21:08:31.507 INFO:teuthology.orchestra.run.smithi067.stdout:(128/152): python3-jaraco-text-3.2.0-6.el9.noar 168 kB/s | 20 kB 00:00 2024-05-03T21:08:31.539 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 53/156 2024-05-03T21:08:31.582 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 54/156 2024-05-03T21:08:31.633 INFO:teuthology.orchestra.run.smithi067.stdout:(129/152): python3-logutils-0.3.5-21.el9.noarch 371 kB/s | 46 kB 00:00 2024-05-03T21:08:31.666 INFO:teuthology.orchestra.run.smithi067.stdout:(130/152): python3-grpcio-tools-1.46.7-10.el9.x 247 kB/s | 144 kB 00:00 2024-05-03T21:08:31.716 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 55/156 2024-05-03T21:08:31.750 INFO:teuthology.orchestra.run.smithi067.stdout:(131/152): python3-kubernetes-26.1.0-2.el9.noar 3.6 MB/s | 1.0 MB 00:00 2024-05-03T21:08:31.762 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 56/156 2024-05-03T21:08:31.775 INFO:teuthology.orchestra.run.smithi067.stdout:(132/152): python3-more-itertools-8.12.0-2.el9. 554 kB/s | 79 kB 00:00 2024-05-03T21:08:31.793 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-colorama-0.4.4-12.el9.noarch 57/156 2024-05-03T21:08:31.834 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 58/156 2024-05-03T21:08:31.867 INFO:teuthology.orchestra.run.smithi067.stdout:(133/152): python3-pecan-1.4.2-3.el9.noarch.rpm 2.3 MB/s | 272 kB 00:00 2024-05-03T21:08:31.901 INFO:teuthology.orchestra.run.smithi067.stdout:(134/152): python3-portend-3.1.0-2.el9.noarch.r 131 kB/s | 16 kB 00:00 2024-05-03T21:08:31.926 INFO:teuthology.orchestra.run.smithi067.stdout:(135/152): python3-natsort-7.1.1-5.el9.noarch.r 222 kB/s | 58 kB 00:00 2024-05-03T21:08:31.930 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 59/156 2024-05-03T21:08:31.967 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-google-auth-1:2.29.0-1.el9.noarch 60/156 2024-05-03T21:08:31.985 INFO:teuthology.orchestra.run.smithi067.stdout:(136/152): python3-pyOpenSSL-21.0.0-1.el9.noarc 768 kB/s | 90 kB 00:00 2024-05-03T21:08:32.007 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 61/156 2024-05-03T21:08:32.018 INFO:teuthology.orchestra.run.smithi067.stdout:(137/152): python3-pyperclip-1.8.2-2.el9.noarch 241 kB/s | 28 kB 00:00 2024-05-03T21:08:32.033 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 62/156 2024-05-03T21:08:32.043 INFO:teuthology.orchestra.run.smithi067.stdout:(138/152): python3-repoze-lru-0.7-16.el9.noarch 262 kB/s | 31 kB 00:00 2024-05-03T21:08:32.058 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : libarrow-doc-9.0.0-11.el9.noarch 63/156 2024-05-03T21:08:32.102 INFO:teuthology.orchestra.run.smithi067.stdout:(139/152): python3-routes-2.5.1-5.el9.noarch.rp 1.6 MB/s | 188 kB 00:00 2024-05-03T21:08:32.136 INFO:teuthology.orchestra.run.smithi067.stdout:(140/152): python3-rsa-4.9-2.el9.noarch.rpm 504 kB/s | 59 kB 00:00 2024-05-03T21:08:32.156 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 64/156 2024-05-03T21:08:32.219 INFO:teuthology.orchestra.run.smithi067.stdout:(141/152): python3-typing-extensions-4.4.0-2.el 436 kB/s | 51 kB 00:00 2024-05-03T21:08:32.253 INFO:teuthology.orchestra.run.smithi067.stdout:(142/152): python3-webob-1.8.7-6.el9.noarch.rpm 1.9 MB/s | 230 kB 00:00 2024-05-03T21:08:32.278 INFO:teuthology.orchestra.run.smithi067.stdout:(143/152): python3-tempora-5.0.0-2.el9.noarch.r 152 kB/s | 36 kB 00:00 2024-05-03T21:08:32.284 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 65/156 2024-05-03T21:08:32.285 INFO:teuthology.orchestra.run.smithi138.stdout: Preparing : 1/1 2024-05-03T21:08:32.315 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 66/156 2024-05-03T21:08:32.335 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : zlib-devel-1.2.11-41.el9.x86_64 67/156 2024-05-03T21:08:32.337 INFO:teuthology.orchestra.run.smithi067.stdout:(144/152): python3-websocket-client-1.2.3-2.el9 766 kB/s | 90 kB 00:00 2024-05-03T21:08:32.369 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 68/156 2024-05-03T21:08:32.371 INFO:teuthology.orchestra.run.smithi138.stdout: Upgrading : openssl-libs-1:3.2.1-1.el9.x86_64 1/156 2024-05-03T21:08:32.387 INFO:teuthology.orchestra.run.smithi067.stdout:(145/152): python3-zc-lockfile-2.0-10.el9.noarc 184 kB/s | 20 kB 00:00 2024-05-03T21:08:32.408 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : uuid-1.6.2-55.el9.x86_64 69/156 2024-05-03T21:08:32.409 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 2/156 2024-05-03T21:08:32.440 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 70/156 2024-05-03T21:08:32.462 INFO:teuthology.orchestra.run.smithi067.stdout:(146/152): re2-20211101-3.el9.x86_64.rpm 1.5 MB/s | 192 kB 00:00 2024-05-03T21:08:32.463 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 3/156 2024-05-03T21:08:32.469 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 71/156 2024-05-03T21:08:32.496 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : thrift-0.15.0-2.el9.x86_64 4/156 2024-05-03T21:08:32.504 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 72/156 2024-05-03T21:08:32.522 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 5/156 2024-05-03T21:08:32.538 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 73/156 2024-05-03T21:08:32.565 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 6/156 2024-05-03T21:08:32.581 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 74/156 2024-05-03T21:08:32.746 INFO:teuthology.orchestra.run.smithi067.stdout:(147/152): thrift-0.15.0-2.el9.x86_64.rpm 4.4 MB/s | 1.6 MB 00:00 2024-05-03T21:08:32.873 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 75/156 2024-05-03T21:08:32.892 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 7/156 2024-05-03T21:08:33.007 INFO:teuthology.orchestra.run.smithi138.stdout: Upgrading : librados2-2:19.0.0-3450.g9a047fe7.el9.x86_64 8/156 2024-05-03T21:08:33.090 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 76/156 2024-05-03T21:08:33.113 INFO:teuthology.orchestra.run.smithi067.stdout:(148/152): librados2-19.0.0-3450.g9a047fe7.el9. 5.2 MB/s | 3.4 MB 00:00 2024-05-03T21:08:33.119 INFO:teuthology.orchestra.run.smithi138.stdout: Running scriptlet: librados2-2:19.0.0-3450.g9a047fe7.el9.x86_64 8/156 2024-05-03T21:08:33.150 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : libcephfs2-2:19.0.0-3450.g9a047fe7.el9.x86_64 9/156 2024-05-03T21:08:33.153 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 77/156 2024-05-03T21:08:33.192 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 78/156 2024-05-03T21:08:33.240 INFO:teuthology.orchestra.run.smithi138.stdout: Running scriptlet: libcephfs2-2:19.0.0-3450.g9a047fe7.el9.x86_64 9/156 2024-05-03T21:08:33.240 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 79/156 2024-05-03T21:08:33.262 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-rados-2:19.0.0-3450.g9a047fe7.el9.x86_64 10/156 2024-05-03T21:08:33.347 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-packaging-20.9-5.el9.noarch 80/156 2024-05-03T21:08:33.402 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 81/156 2024-05-03T21:08:33.409 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : postgresql-private-libs-13.14-1.el9.x86_64 11/156 2024-05-03T21:08:33.463 INFO:teuthology.orchestra.run.smithi067.stdout:(149/152): openssl-3.2.1-1.el9.x86_64.rpm 3.9 MB/s | 1.3 MB 00:00 2024-05-03T21:08:33.467 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : postgresql-13.14-1.el9.x86_64 12/156 2024-05-03T21:08:33.496 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : libcephsqlite-2:19.0.0-3450.g9a047fe7.el9.x86_64 13/156 2024-05-03T21:08:33.530 INFO:teuthology.orchestra.run.smithi067.stdout:(150/152): librbd1-19.0.0-3450.g9a047fe7.el9.x8 4.0 MB/s | 3.1 MB 00:00 2024-05-03T21:08:33.598 INFO:teuthology.orchestra.run.smithi138.stdout: Running scriptlet: libcephsqlite-2:19.0.0-3450.g9a047fe7.el9.x86_64 13/156 2024-05-03T21:08:33.622 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : libradosstriper1-2:19.0.0-3450.g9a047fe7.el9.x86 14/156 2024-05-03T21:08:33.639 INFO:teuthology.orchestra.run.smithi067.stdout:(151/152): python3-werkzeug-2.0.3-3.el9.1.noarc 308 kB/s | 427 kB 00:01 2024-05-03T21:08:33.684 INFO:teuthology.orchestra.run.smithi138.stdout: Running scriptlet: libradosstriper1-2:19.0.0-3450.g9a047fe7.el9.x86 14/156 2024-05-03T21:08:33.706 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 15/156 2024-05-03T21:08:33.714 INFO:teuthology.orchestra.run.smithi067.stdout:(152/152): openssl-libs-3.2.1-1.el9.x86_64.rpm 9.6 MB/s | 2.4 MB 00:00 2024-05-03T21:08:33.719 INFO:teuthology.orchestra.run.smithi067.stdout:-------------------------------------------------------------------------------- 2024-05-03T21:08:33.720 INFO:teuthology.orchestra.run.smithi067.stdout:Total 6.4 MB/s | 222 MB 00:34 2024-05-03T21:08:33.729 INFO:teuthology.orchestra.run.smithi138.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 15/156 2024-05-03T21:08:33.940 INFO:teuthology.orchestra.run.smithi138.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 16/156 2024-05-03T21:08:33.982 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 16/156 2024-05-03T21:08:34.101 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : re2-1:20211101-3.el9.x86_64 17/156 2024-05-03T21:08:34.120 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 82/156 2024-05-03T21:08:34.165 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 83/156 2024-05-03T21:08:34.196 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : libarrow-9.0.0-11.el9.x86_64 18/156 2024-05-03T21:08:34.199 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 84/156 2024-05-03T21:08:34.231 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 19/156 2024-05-03T21:08:34.297 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 85/156 2024-05-03T21:08:34.348 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-devel-3.9.18-3.el9.x86_64 86/156 2024-05-03T21:08:34.400 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : liboath-2.6.7-2.el9.x86_64 20/156 2024-05-03T21:08:34.435 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 87/156 2024-05-03T21:08:34.476 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 88/156 2024-05-03T21:08:34.532 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 21/156 2024-05-03T21:08:34.573 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 22/156 2024-05-03T21:08:34.617 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 23/156 2024-05-03T21:08:34.649 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 24/156 2024-05-03T21:08:34.653 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : openblas-0.3.21-2.el9.x86_64 89/156 2024-05-03T21:08:34.674 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 90/156 2024-05-03T21:08:34.686 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : lua-5.4.4-4.el9.x86_64 25/156 2024-05-03T21:08:34.742 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 91/156 2024-05-03T21:08:34.754 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 26/156 2024-05-03T21:08:34.808 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 27/156 2024-05-03T21:08:34.826 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction check 2024-05-03T21:08:34.847 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 28/156 2024-05-03T21:08:34.882 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 29/156 2024-05-03T21:08:34.958 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 30/156 2024-05-03T21:08:34.986 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction check succeeded. 2024-05-03T21:08:34.986 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction test 2024-05-03T21:08:34.993 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-ceph-common-2:19.0.0-3450.g9a047fe7.el9. 31/156 2024-05-03T21:08:35.036 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-ceph-argparse-2:19.0.0-3450.g9a047fe7.el 32/156 2024-05-03T21:08:35.066 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-cephfs-2:19.0.0-3450.g9a047fe7.el9.x86_6 33/156 2024-05-03T21:08:35.134 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 34/156 2024-05-03T21:08:35.180 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : luarocks-3.9.2-1.el9.noarch 35/156 2024-05-03T21:08:35.262 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 36/156 2024-05-03T21:08:35.381 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 92/156 2024-05-03T21:08:35.403 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 37/156 2024-05-03T21:08:35.465 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 38/156 2024-05-03T21:08:35.505 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 93/156 2024-05-03T21:08:35.990 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-rsa-4.9-2.el9.noarch 39/156 2024-05-03T21:08:36.086 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 40/156 2024-05-03T21:08:36.115 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 41/156 2024-05-03T21:08:36.158 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 42/156 2024-05-03T21:08:36.196 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : librados-devel-2:19.0.0-3450.g9a047fe7.el9.x86_6 43/156 2024-05-03T21:08:36.238 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 44/156 2024-05-03T21:08:36.382 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 45/156 2024-05-03T21:08:36.441 INFO:teuthology.orchestra.run.smithi138.stdout: Upgrading : openssl-1:3.2.1-1.el9.x86_64 46/156 2024-05-03T21:08:36.495 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 47/156 2024-05-03T21:08:36.568 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 48/156 2024-05-03T21:08:36.600 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 49/156 2024-05-03T21:08:36.638 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 50/156 2024-05-03T21:08:36.683 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 51/156 2024-05-03T21:08:36.710 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 94/156 2024-05-03T21:08:36.719 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 52/156 2024-05-03T21:08:36.762 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 53/156 2024-05-03T21:08:36.786 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 95/156 2024-05-03T21:08:36.804 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 54/156 2024-05-03T21:08:36.813 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 96/156 2024-05-03T21:08:36.847 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 97/156 2024-05-03T21:08:36.870 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction test succeeded. 2024-05-03T21:08:36.871 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction 2024-05-03T21:08:36.948 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 55/156 2024-05-03T21:08:37.002 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 56/156 2024-05-03T21:08:37.031 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-colorama-0.4.4-12.el9.noarch 57/156 2024-05-03T21:08:37.072 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 58/156 2024-05-03T21:08:37.158 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 98/156 2024-05-03T21:08:37.167 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 59/156 2024-05-03T21:08:37.199 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-google-auth-1:2.29.0-1.el9.noarch 60/156 2024-05-03T21:08:37.201 INFO:teuthology.orchestra.run.smithi082.stdout: Upgrading : librbd1-2:19.0.0-3450.g9a047fe7.el9.x86_64 99/156 2024-05-03T21:08:37.238 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 61/156 2024-05-03T21:08:37.291 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: librbd1-2:19.0.0-3450.g9a047fe7.el9.x86_64 99/156 2024-05-03T21:08:37.297 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 62/156 2024-05-03T21:08:37.314 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : libarrow-doc-9.0.0-11.el9.noarch 63/156 2024-05-03T21:08:37.322 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-rbd-2:19.0.0-3450.g9a047fe7.el9.x86_64 100/156 2024-05-03T21:08:37.359 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 101/156 2024-05-03T21:08:37.388 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 64/156 2024-05-03T21:08:37.515 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 65/156 2024-05-03T21:08:37.546 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 66/156 2024-05-03T21:08:37.574 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : zlib-devel-1.2.11-41.el9.x86_64 67/156 2024-05-03T21:08:37.608 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 68/156 2024-05-03T21:08:37.648 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : uuid-1.6.2-55.el9.x86_64 69/156 2024-05-03T21:08:37.679 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 70/156 2024-05-03T21:08:37.709 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 71/156 2024-05-03T21:08:37.744 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 72/156 2024-05-03T21:08:37.777 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 73/156 2024-05-03T21:08:37.819 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 74/156 2024-05-03T21:08:37.837 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : parquet-libs-9.0.0-11.el9.x86_64 102/156 2024-05-03T21:08:37.871 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : librgw2-2:19.0.0-3450.g9a047fe7.el9.x86_64 103/156 2024-05-03T21:08:37.942 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: librgw2-2:19.0.0-3450.g9a047fe7.el9.x86_64 103/156 2024-05-03T21:08:37.964 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-rgw-2:19.0.0-3450.g9a047fe7.el9.x86_64 104/156 2024-05-03T21:08:38.047 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 75/156 2024-05-03T21:08:38.135 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 76/156 2024-05-03T21:08:38.184 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 77/156 2024-05-03T21:08:38.215 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 78/156 2024-05-03T21:08:38.262 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 79/156 2024-05-03T21:08:38.360 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-packaging-20.9-5.el9.noarch 80/156 2024-05-03T21:08:38.416 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 81/156 2024-05-03T21:08:39.130 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 82/156 2024-05-03T21:08:39.181 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 83/156 2024-05-03T21:08:39.214 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 84/156 2024-05-03T21:08:39.312 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 85/156 2024-05-03T21:08:39.372 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-devel-3.9.18-3.el9.x86_64 86/156 2024-05-03T21:08:39.459 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 87/156 2024-05-03T21:08:39.500 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 88/156 2024-05-03T21:08:39.678 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : openblas-0.3.21-2.el9.x86_64 89/156 2024-05-03T21:08:39.714 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 90/156 2024-05-03T21:08:39.791 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 91/156 2024-05-03T21:08:40.302 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: ceph-common-2:19.0.0-3450.g9a047fe7.el9.x86_64 105/156 2024-05-03T21:08:40.329 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : ceph-common-2:19.0.0-3450.g9a047fe7.el9.x86_64 105/156 2024-05-03T21:08:40.425 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 92/156 2024-05-03T21:08:40.562 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 93/156 2024-05-03T21:08:40.871 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: ceph-common-2:19.0.0-3450.g9a047fe7.el9.x86_64 105/156 2024-05-03T21:08:40.916 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : ceph-base-2:19.0.0-3450.g9a047fe7.el9.x86_64 106/156 2024-05-03T21:08:41.008 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: ceph-base-2:19.0.0-3450.g9a047fe7.el9.x86_64 106/156 2024-05-03T21:08:41.008 INFO:teuthology.orchestra.run.smithi082.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-05-03T21:08:41.008 INFO:teuthology.orchestra.run.smithi082.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-05-03T21:08:41.008 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-05-03T21:08:41.041 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : ceph-selinux-2:19.0.0-3450.g9a047fe7.el9.x86_64 107/156 2024-05-03T21:08:41.778 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 94/156 2024-05-03T21:08:41.869 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 95/156 2024-05-03T21:08:41.897 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 96/156 2024-05-03T21:08:41.930 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 97/156 2024-05-03T21:08:42.233 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 98/156 2024-05-03T21:08:42.277 INFO:teuthology.orchestra.run.smithi138.stdout: Upgrading : librbd1-2:19.0.0-3450.g9a047fe7.el9.x86_64 99/156 2024-05-03T21:08:42.368 INFO:teuthology.orchestra.run.smithi138.stdout: Running scriptlet: librbd1-2:19.0.0-3450.g9a047fe7.el9.x86_64 99/156 2024-05-03T21:08:42.398 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-rbd-2:19.0.0-3450.g9a047fe7.el9.x86_64 100/156 2024-05-03T21:08:42.435 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 101/156 2024-05-03T21:08:42.911 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : parquet-libs-9.0.0-11.el9.x86_64 102/156 2024-05-03T21:08:42.947 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : librgw2-2:19.0.0-3450.g9a047fe7.el9.x86_64 103/156 2024-05-03T21:08:43.018 INFO:teuthology.orchestra.run.smithi138.stdout: Running scriptlet: librgw2-2:19.0.0-3450.g9a047fe7.el9.x86_64 103/156 2024-05-03T21:08:43.039 INFO:teuthology.orchestra.run.smithi067.stdout: Preparing : 1/1 2024-05-03T21:08:43.040 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-rgw-2:19.0.0-3450.g9a047fe7.el9.x86_64 104/156 2024-05-03T21:08:43.144 INFO:teuthology.orchestra.run.smithi067.stdout: Upgrading : openssl-libs-1:3.2.1-1.el9.x86_64 1/156 2024-05-03T21:08:43.183 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 2/156 2024-05-03T21:08:43.245 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 3/156 2024-05-03T21:08:43.287 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : thrift-0.15.0-2.el9.x86_64 4/156 2024-05-03T21:08:43.312 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 5/156 2024-05-03T21:08:43.356 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 6/156 2024-05-03T21:08:43.701 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 7/156 2024-05-03T21:08:43.804 INFO:teuthology.orchestra.run.smithi067.stdout: Upgrading : librados2-2:19.0.0-3450.g9a047fe7.el9.x86_64 8/156 2024-05-03T21:08:43.928 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: librados2-2:19.0.0-3450.g9a047fe7.el9.x86_64 8/156 2024-05-03T21:08:43.955 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : libcephfs2-2:19.0.0-3450.g9a047fe7.el9.x86_64 9/156 2024-05-03T21:08:44.047 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: libcephfs2-2:19.0.0-3450.g9a047fe7.el9.x86_64 9/156 2024-05-03T21:08:44.076 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-rados-2:19.0.0-3450.g9a047fe7.el9.x86_64 10/156 2024-05-03T21:08:44.233 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : postgresql-private-libs-13.14-1.el9.x86_64 11/156 2024-05-03T21:08:44.282 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : postgresql-13.14-1.el9.x86_64 12/156 2024-05-03T21:08:44.310 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : libcephsqlite-2:19.0.0-3450.g9a047fe7.el9.x86_64 13/156 2024-05-03T21:08:44.414 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: libcephsqlite-2:19.0.0-3450.g9a047fe7.el9.x86_64 13/156 2024-05-03T21:08:44.435 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : libradosstriper1-2:19.0.0-3450.g9a047fe7.el9.x86 14/156 2024-05-03T21:08:44.498 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: libradosstriper1-2:19.0.0-3450.g9a047fe7.el9.x86 14/156 2024-05-03T21:08:44.511 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 15/156 2024-05-03T21:08:44.531 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 15/156 2024-05-03T21:08:44.729 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 16/156 2024-05-03T21:08:44.762 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 16/156 2024-05-03T21:08:44.872 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : re2-1:20211101-3.el9.x86_64 17/156 2024-05-03T21:08:44.969 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : libarrow-9.0.0-11.el9.x86_64 18/156 2024-05-03T21:08:45.002 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 19/156 2024-05-03T21:08:45.038 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : liboath-2.6.7-2.el9.x86_64 20/156 2024-05-03T21:08:45.114 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 21/156 2024-05-03T21:08:45.338 INFO:teuthology.orchestra.run.smithi138.stdout: Running scriptlet: ceph-common-2:19.0.0-3450.g9a047fe7.el9.x86_64 105/156 2024-05-03T21:08:45.370 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : ceph-common-2:19.0.0-3450.g9a047fe7.el9.x86_64 105/156 2024-05-03T21:08:45.398 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 22/156 2024-05-03T21:08:45.439 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 23/156 2024-05-03T21:08:45.471 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 24/156 2024-05-03T21:08:45.499 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : lua-5.4.4-4.el9.x86_64 25/156 2024-05-03T21:08:45.578 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 26/156 2024-05-03T21:08:45.634 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 27/156 2024-05-03T21:08:45.660 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 28/156 2024-05-03T21:08:45.696 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 29/156 2024-05-03T21:08:45.774 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 30/156 2024-05-03T21:08:45.806 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-ceph-common-2:19.0.0-3450.g9a047fe7.el9. 31/156 2024-05-03T21:08:45.850 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-ceph-argparse-2:19.0.0-3450.g9a047fe7.el 32/156 2024-05-03T21:08:45.878 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-cephfs-2:19.0.0-3450.g9a047fe7.el9.x86_6 33/156 2024-05-03T21:08:45.913 INFO:teuthology.orchestra.run.smithi138.stdout: Running scriptlet: ceph-common-2:19.0.0-3450.g9a047fe7.el9.x86_64 105/156 2024-05-03T21:08:45.950 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 34/156 2024-05-03T21:08:45.956 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : ceph-base-2:19.0.0-3450.g9a047fe7.el9.x86_64 106/156 2024-05-03T21:08:46.001 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : luarocks-3.9.2-1.el9.noarch 35/156 2024-05-03T21:08:46.046 INFO:teuthology.orchestra.run.smithi138.stdout: Running scriptlet: ceph-base-2:19.0.0-3450.g9a047fe7.el9.x86_64 106/156 2024-05-03T21:08:46.046 INFO:teuthology.orchestra.run.smithi138.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-05-03T21:08:46.046 INFO:teuthology.orchestra.run.smithi138.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-05-03T21:08:46.047 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-05-03T21:08:46.078 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 36/156 2024-05-03T21:08:46.083 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : ceph-selinux-2:19.0.0-3450.g9a047fe7.el9.x86_64 107/156 2024-05-03T21:08:46.216 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 37/156 2024-05-03T21:08:46.272 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 38/156 2024-05-03T21:08:46.833 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-rsa-4.9-2.el9.noarch 39/156 2024-05-03T21:08:46.919 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 40/156 2024-05-03T21:08:46.953 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 41/156 2024-05-03T21:08:46.996 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 42/156 2024-05-03T21:08:47.034 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : librados-devel-2:19.0.0-3450.g9a047fe7.el9.x86_6 43/156 2024-05-03T21:08:47.076 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 44/156 2024-05-03T21:08:47.238 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 45/156 2024-05-03T21:08:47.295 INFO:teuthology.orchestra.run.smithi067.stdout: Upgrading : openssl-1:3.2.1-1.el9.x86_64 46/156 2024-05-03T21:08:47.351 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 47/156 2024-05-03T21:08:47.424 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 48/156 2024-05-03T21:08:47.463 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 49/156 2024-05-03T21:08:47.499 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 50/156 2024-05-03T21:08:47.546 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 51/156 2024-05-03T21:08:47.581 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 52/156 2024-05-03T21:08:47.625 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 53/156 2024-05-03T21:08:47.667 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 54/156 2024-05-03T21:08:47.810 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 55/156 2024-05-03T21:08:47.856 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 56/156 2024-05-03T21:08:47.877 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-colorama-0.4.4-12.el9.noarch 57/156 2024-05-03T21:08:47.919 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 58/156 2024-05-03T21:08:48.017 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 59/156 2024-05-03T21:08:48.052 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-google-auth-1:2.29.0-1.el9.noarch 60/156 2024-05-03T21:08:48.094 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 61/156 2024-05-03T21:08:48.117 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 62/156 2024-05-03T21:08:48.142 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : libarrow-doc-9.0.0-11.el9.noarch 63/156 2024-05-03T21:08:48.220 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 64/156 2024-05-03T21:08:48.349 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 65/156 2024-05-03T21:08:48.391 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 66/156 2024-05-03T21:08:48.410 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : zlib-devel-1.2.11-41.el9.x86_64 67/156 2024-05-03T21:08:48.445 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 68/156 2024-05-03T21:08:48.485 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : uuid-1.6.2-55.el9.x86_64 69/156 2024-05-03T21:08:48.525 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 70/156 2024-05-03T21:08:48.554 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 71/156 2024-05-03T21:08:48.589 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 72/156 2024-05-03T21:08:48.622 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 73/156 2024-05-03T21:08:48.664 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 74/156 2024-05-03T21:08:48.698 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 75/156 2024-05-03T21:08:48.780 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 76/156 2024-05-03T21:08:48.829 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 77/156 2024-05-03T21:08:48.861 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 78/156 2024-05-03T21:08:48.910 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 79/156 2024-05-03T21:08:49.019 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-packaging-20.9-5.el9.noarch 80/156 2024-05-03T21:08:49.071 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 81/156 2024-05-03T21:08:49.840 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 82/156 2024-05-03T21:08:49.883 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 83/156 2024-05-03T21:08:49.916 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 84/156 2024-05-03T21:08:50.019 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 85/156 2024-05-03T21:08:50.075 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-devel-3.9.18-3.el9.x86_64 86/156 2024-05-03T21:08:50.166 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 87/156 2024-05-03T21:08:50.210 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 88/156 2024-05-03T21:08:50.395 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : openblas-0.3.21-2.el9.x86_64 89/156 2024-05-03T21:08:50.424 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 90/156 2024-05-03T21:08:50.504 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 91/156 2024-05-03T21:08:51.189 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 92/156 2024-05-03T21:08:51.318 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 93/156 2024-05-03T21:08:52.622 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 94/156 2024-05-03T21:08:52.663 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: ceph-selinux-2:19.0.0-3450.g9a047fe7.el9.x86_64 107/156 2024-05-03T21:08:52.663 INFO:teuthology.orchestra.run.smithi082.stdout:skipping the directory /sys 2024-05-03T21:08:52.663 INFO:teuthology.orchestra.run.smithi082.stdout:skipping the directory /proc 2024-05-03T21:08:52.664 INFO:teuthology.orchestra.run.smithi082.stdout:skipping the directory /mnt 2024-05-03T21:08:52.664 INFO:teuthology.orchestra.run.smithi082.stdout:skipping the directory /var/tmp 2024-05-03T21:08:52.664 INFO:teuthology.orchestra.run.smithi082.stdout:skipping the directory /home 2024-05-03T21:08:52.664 INFO:teuthology.orchestra.run.smithi082.stdout:skipping the directory /root 2024-05-03T21:08:52.664 INFO:teuthology.orchestra.run.smithi082.stdout:skipping the directory /tmp 2024-05-03T21:08:52.664 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-05-03T21:08:52.711 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 95/156 2024-05-03T21:08:52.739 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 96/156 2024-05-03T21:08:52.772 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 97/156 2024-05-03T21:08:52.830 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : ceph-volume-2:19.0.0-3450.g9a047fe7.el9.noarch 108/156 2024-05-03T21:08:52.845 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: ceph-volume-2:19.0.0-3450.g9a047fe7.el9.noarch 108/156 2024-05-03T21:08:52.846 INFO:teuthology.orchestra.run.smithi082.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-03T21:08:52.846 INFO:teuthology.orchestra.run.smithi082.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-05-03T21:08:52.846 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-05-03T21:08:53.110 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 98/156 2024-05-03T21:08:53.152 INFO:teuthology.orchestra.run.smithi067.stdout: Upgrading : librbd1-2:19.0.0-3450.g9a047fe7.el9.x86_64 99/156 2024-05-03T21:08:53.243 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: librbd1-2:19.0.0-3450.g9a047fe7.el9.x86_64 99/156 2024-05-03T21:08:53.274 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-rbd-2:19.0.0-3450.g9a047fe7.el9.x86_64 100/156 2024-05-03T21:08:53.310 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 101/156 2024-05-03T21:08:53.816 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : parquet-libs-9.0.0-11.el9.x86_64 102/156 2024-05-03T21:08:53.847 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : librgw2-2:19.0.0-3450.g9a047fe7.el9.x86_64 103/156 2024-05-03T21:08:53.918 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: librgw2-2:19.0.0-3450.g9a047fe7.el9.x86_64 103/156 2024-05-03T21:08:53.956 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-rgw-2:19.0.0-3450.g9a047fe7.el9.x86_64 104/156 2024-05-03T21:08:54.332 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : ceph-osd-2:19.0.0-3450.g9a047fe7.el9.x86_64 109/156 2024-05-03T21:08:54.358 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: ceph-osd-2:19.0.0-3450.g9a047fe7.el9.x86_64 109/156 2024-05-03T21:08:54.358 INFO:teuthology.orchestra.run.smithi082.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-03T21:08:54.358 INFO:teuthology.orchestra.run.smithi082.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-05-03T21:08:54.358 INFO:teuthology.orchestra.run.smithi082.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-05-03T21:08:54.358 INFO:teuthology.orchestra.run.smithi082.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-05-03T21:08:54.358 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-05-03T21:08:54.592 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : ceph-mds-2:19.0.0-3450.g9a047fe7.el9.x86_64 110/156 2024-05-03T21:08:54.614 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: ceph-mds-2:19.0.0-3450.g9a047fe7.el9.x86_64 110/156 2024-05-03T21:08:54.614 INFO:teuthology.orchestra.run.smithi082.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-03T21:08:54.614 INFO:teuthology.orchestra.run.smithi082.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-05-03T21:08:54.614 INFO:teuthology.orchestra.run.smithi082.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-05-03T21:08:54.615 INFO:teuthology.orchestra.run.smithi082.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-05-03T21:08:54.615 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-05-03T21:08:55.059 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : ceph-mon-2:19.0.0-3450.g9a047fe7.el9.x86_64 111/156 2024-05-03T21:08:55.082 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: ceph-mon-2:19.0.0-3450.g9a047fe7.el9.x86_64 111/156 2024-05-03T21:08:55.082 INFO:teuthology.orchestra.run.smithi082.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-03T21:08:55.082 INFO:teuthology.orchestra.run.smithi082.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-05-03T21:08:55.082 INFO:teuthology.orchestra.run.smithi082.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-05-03T21:08:55.083 INFO:teuthology.orchestra.run.smithi082.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-05-03T21:08:55.083 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-05-03T21:08:55.189 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : automake-1.16.2-8.el9.noarch 112/156 2024-05-03T21:08:55.239 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-ply-3.11-14.el9.noarch 113/156 2024-05-03T21:08:55.293 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 114/156 2024-05-03T21:08:55.479 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 115/156 2024-05-03T21:08:55.527 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 116/156 2024-05-03T21:08:55.605 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 117/156 2024-05-03T21:08:55.710 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 118/156 2024-05-03T21:08:55.840 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 119/156 2024-05-03T21:08:55.885 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 120/156 2024-05-03T21:08:55.917 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 121/156 2024-05-03T21:08:55.953 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : fuse-2.9.9-15.el9.x86_64 122/156 2024-05-03T21:08:56.363 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: cephadm-2:19.0.0-3450.g9a047fe7.el9.noarch 123/156 2024-05-03T21:08:56.367 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: ceph-common-2:19.0.0-3450.g9a047fe7.el9.x86_64 105/156 2024-05-03T21:08:56.393 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : cephadm-2:19.0.0-3450.g9a047fe7.el9.noarch 123/156 2024-05-03T21:08:56.403 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : ceph-common-2:19.0.0-3450.g9a047fe7.el9.x86_64 105/156 2024-05-03T21:08:56.425 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-3450.g9a047fe7.e 124/156 2024-05-03T21:08:56.486 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-3450.g9a047fe7. 125/156 2024-05-03T21:08:56.516 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-3450.g9a047fe7.el9.noa 126/156 2024-05-03T21:08:56.974 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: ceph-common-2:19.0.0-3450.g9a047fe7.el9.x86_64 105/156 2024-05-03T21:08:57.015 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : ceph-base-2:19.0.0-3450.g9a047fe7.el9.x86_64 106/156 2024-05-03T21:08:57.105 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: ceph-base-2:19.0.0-3450.g9a047fe7.el9.x86_64 106/156 2024-05-03T21:08:57.105 INFO:teuthology.orchestra.run.smithi067.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-05-03T21:08:57.105 INFO:teuthology.orchestra.run.smithi067.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-05-03T21:08:57.105 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-05-03T21:08:57.132 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : ceph-selinux-2:19.0.0-3450.g9a047fe7.el9.x86_64 107/156 2024-05-03T21:08:57.556 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-3450.g9a047fe7.el9.noa 126/156 2024-05-03T21:08:57.659 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-3450.g9a047fe7.el9.n 127/156 2024-05-03T21:08:57.826 INFO:teuthology.orchestra.run.smithi138.stdout: Running scriptlet: ceph-selinux-2:19.0.0-3450.g9a047fe7.el9.x86_64 107/156 2024-05-03T21:08:57.826 INFO:teuthology.orchestra.run.smithi138.stdout:skipping the directory /sys 2024-05-03T21:08:57.827 INFO:teuthology.orchestra.run.smithi138.stdout:skipping the directory /proc 2024-05-03T21:08:57.827 INFO:teuthology.orchestra.run.smithi138.stdout:skipping the directory /mnt 2024-05-03T21:08:57.827 INFO:teuthology.orchestra.run.smithi138.stdout:skipping the directory /var/tmp 2024-05-03T21:08:57.827 INFO:teuthology.orchestra.run.smithi138.stdout:skipping the directory /home 2024-05-03T21:08:57.827 INFO:teuthology.orchestra.run.smithi138.stdout:skipping the directory /root 2024-05-03T21:08:57.827 INFO:teuthology.orchestra.run.smithi138.stdout:skipping the directory /tmp 2024-05-03T21:08:57.827 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-05-03T21:08:57.999 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : ceph-volume-2:19.0.0-3450.g9a047fe7.el9.noarch 108/156 2024-05-03T21:08:58.015 INFO:teuthology.orchestra.run.smithi138.stdout: Running scriptlet: ceph-volume-2:19.0.0-3450.g9a047fe7.el9.noarch 108/156 2024-05-03T21:08:58.015 INFO:teuthology.orchestra.run.smithi138.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-03T21:08:58.015 INFO:teuthology.orchestra.run.smithi138.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-05-03T21:08:58.015 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-05-03T21:08:58.621 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-3450.g9a047fe7.el9.n 127/156 2024-05-03T21:08:58.651 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-3450.g9a0 128/156 2024-05-03T21:08:58.760 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-3450.g9a0 128/156 2024-05-03T21:08:58.929 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-3450.g9a047fe7.el 129/156 2024-05-03T21:08:58.982 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : ceph-mgr-2:19.0.0-3450.g9a047fe7.el9.x86_64 130/156 2024-05-03T21:08:59.005 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: ceph-mgr-2:19.0.0-3450.g9a047fe7.el9.x86_64 130/156 2024-05-03T21:08:59.005 INFO:teuthology.orchestra.run.smithi082.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-03T21:08:59.005 INFO:teuthology.orchestra.run.smithi082.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-05-03T21:08:59.005 INFO:teuthology.orchestra.run.smithi082.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-05-03T21:08:59.005 INFO:teuthology.orchestra.run.smithi082.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-05-03T21:08:59.005 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-05-03T21:08:59.053 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : ceph-mgr-rook-2:19.0.0-3450.g9a047fe7.el9.noarch 131/156 2024-05-03T21:08:59.065 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-3450.g9a047fe7.el9.noarch 131/156 2024-05-03T21:08:59.162 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : ceph-2:19.0.0-3450.g9a047fe7.el9.x86_64 132/156 2024-05-03T21:08:59.210 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : ceph-fuse-2:19.0.0-3450.g9a047fe7.el9.x86_64 133/156 2024-05-03T21:08:59.512 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : ceph-osd-2:19.0.0-3450.g9a047fe7.el9.x86_64 109/156 2024-05-03T21:08:59.538 INFO:teuthology.orchestra.run.smithi138.stdout: Running scriptlet: ceph-osd-2:19.0.0-3450.g9a047fe7.el9.x86_64 109/156 2024-05-03T21:08:59.538 INFO:teuthology.orchestra.run.smithi138.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-03T21:08:59.538 INFO:teuthology.orchestra.run.smithi138.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-05-03T21:08:59.538 INFO:teuthology.orchestra.run.smithi138.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-05-03T21:08:59.538 INFO:teuthology.orchestra.run.smithi138.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-05-03T21:08:59.539 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-05-03T21:08:59.762 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : ceph-mds-2:19.0.0-3450.g9a047fe7.el9.x86_64 110/156 2024-05-03T21:08:59.785 INFO:teuthology.orchestra.run.smithi138.stdout: Running scriptlet: ceph-mds-2:19.0.0-3450.g9a047fe7.el9.x86_64 110/156 2024-05-03T21:08:59.785 INFO:teuthology.orchestra.run.smithi138.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-03T21:08:59.785 INFO:teuthology.orchestra.run.smithi138.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-05-03T21:08:59.785 INFO:teuthology.orchestra.run.smithi138.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-05-03T21:08:59.785 INFO:teuthology.orchestra.run.smithi138.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-05-03T21:08:59.785 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-05-03T21:09:00.133 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : libtool-2.4.6-45.el9.x86_64 134/156 2024-05-03T21:09:00.156 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : ceph-radosgw-2:19.0.0-3450.g9a047fe7.el9.x86_64 135/156 2024-05-03T21:09:00.178 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-3450.g9a047fe7.el9.x86_64 135/156 2024-05-03T21:09:00.179 INFO:teuthology.orchestra.run.smithi082.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-03T21:09:00.179 INFO:teuthology.orchestra.run.smithi082.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-05-03T21:09:00.179 INFO:teuthology.orchestra.run.smithi082.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-05-03T21:09:00.179 INFO:teuthology.orchestra.run.smithi082.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-05-03T21:09:00.179 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-05-03T21:09:00.223 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-3450.g9a047 136/156 2024-05-03T21:09:00.229 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : ceph-mon-2:19.0.0-3450.g9a047fe7.el9.x86_64 111/156 2024-05-03T21:09:00.245 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-3450.g9a047 136/156 2024-05-03T21:09:00.245 INFO:teuthology.orchestra.run.smithi082.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-03T21:09:00.245 INFO:teuthology.orchestra.run.smithi082.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-05-03T21:09:00.245 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-05-03T21:09:00.252 INFO:teuthology.orchestra.run.smithi138.stdout: Running scriptlet: ceph-mon-2:19.0.0-3450.g9a047fe7.el9.x86_64 111/156 2024-05-03T21:09:00.252 INFO:teuthology.orchestra.run.smithi138.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-03T21:09:00.252 INFO:teuthology.orchestra.run.smithi138.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-05-03T21:09:00.252 INFO:teuthology.orchestra.run.smithi138.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-05-03T21:09:00.252 INFO:teuthology.orchestra.run.smithi138.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-05-03T21:09:00.252 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-05-03T21:09:00.290 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : cephfs-mirror-2:19.0.0-3450.g9a047fe7.el9.x86_64 137/156 2024-05-03T21:09:00.312 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-3450.g9a047fe7.el9.x86_64 137/156 2024-05-03T21:09:00.313 INFO:teuthology.orchestra.run.smithi082.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-03T21:09:00.313 INFO:teuthology.orchestra.run.smithi082.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-05-03T21:09:00.313 INFO:teuthology.orchestra.run.smithi082.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-05-03T21:09:00.313 INFO:teuthology.orchestra.run.smithi082.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-05-03T21:09:00.313 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-05-03T21:09:00.359 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : automake-1.16.2-8.el9.noarch 112/156 2024-05-03T21:09:00.408 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-ply-3.11-14.el9.noarch 113/156 2024-05-03T21:09:00.463 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 114/156 2024-05-03T21:09:00.616 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : rbd-mirror-2:19.0.0-3450.g9a047fe7.el9.x86_64 138/156 2024-05-03T21:09:00.638 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: rbd-mirror-2:19.0.0-3450.g9a047fe7.el9.x86_64 138/156 2024-05-03T21:09:00.638 INFO:teuthology.orchestra.run.smithi082.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-03T21:09:00.638 INFO:teuthology.orchestra.run.smithi082.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-05-03T21:09:00.638 INFO:teuthology.orchestra.run.smithi082.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-05-03T21:09:00.639 INFO:teuthology.orchestra.run.smithi082.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-05-03T21:09:00.639 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-05-03T21:09:00.646 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 115/156 2024-05-03T21:09:00.688 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 116/156 2024-05-03T21:09:00.766 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 117/156 2024-05-03T21:09:00.879 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 118/156 2024-05-03T21:09:01.018 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 119/156 2024-05-03T21:09:01.055 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 120/156 2024-05-03T21:09:01.087 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 121/156 2024-05-03T21:09:01.114 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : fuse-2.9.9-15.el9.x86_64 122/156 2024-05-03T21:09:01.530 INFO:teuthology.orchestra.run.smithi138.stdout: Running scriptlet: cephadm-2:19.0.0-3450.g9a047fe7.el9.noarch 123/156 2024-05-03T21:09:01.563 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : cephadm-2:19.0.0-3450.g9a047fe7.el9.noarch 123/156 2024-05-03T21:09:01.595 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-3450.g9a047fe7.e 124/156 2024-05-03T21:09:01.665 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-3450.g9a047fe7. 125/156 2024-05-03T21:09:01.695 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-3450.g9a047fe7.el9.noa 126/156 2024-05-03T21:09:02.725 INFO:teuthology.orchestra.run.smithi138.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-3450.g9a047fe7.el9.noa 126/156 2024-05-03T21:09:02.763 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-3450.g9a047fe7.el9.n 127/156 2024-05-03T21:09:03.724 INFO:teuthology.orchestra.run.smithi138.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-3450.g9a047fe7.el9.n 127/156 2024-05-03T21:09:03.755 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-3450.g9a0 128/156 2024-05-03T21:09:03.865 INFO:teuthology.orchestra.run.smithi138.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-3450.g9a0 128/156 2024-05-03T21:09:04.033 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-3450.g9a047fe7.el 129/156 2024-05-03T21:09:04.057 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : ceph-mgr-2:19.0.0-3450.g9a047fe7.el9.x86_64 130/156 2024-05-03T21:09:04.081 INFO:teuthology.orchestra.run.smithi138.stdout: Running scriptlet: ceph-mgr-2:19.0.0-3450.g9a047fe7.el9.x86_64 130/156 2024-05-03T21:09:04.081 INFO:teuthology.orchestra.run.smithi138.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-03T21:09:04.081 INFO:teuthology.orchestra.run.smithi138.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-05-03T21:09:04.081 INFO:teuthology.orchestra.run.smithi138.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-05-03T21:09:04.081 INFO:teuthology.orchestra.run.smithi138.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-05-03T21:09:04.081 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-05-03T21:09:04.124 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : ceph-mgr-rook-2:19.0.0-3450.g9a047fe7.el9.noarch 131/156 2024-05-03T21:09:04.136 INFO:teuthology.orchestra.run.smithi138.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-3450.g9a047fe7.el9.noarch 131/156 2024-05-03T21:09:04.234 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : ceph-2:19.0.0-3450.g9a047fe7.el9.x86_64 132/156 2024-05-03T21:09:04.290 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : ceph-fuse-2:19.0.0-3450.g9a047fe7.el9.x86_64 133/156 2024-05-03T21:09:05.209 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : libtool-2.4.6-45.el9.x86_64 134/156 2024-05-03T21:09:05.243 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : ceph-radosgw-2:19.0.0-3450.g9a047fe7.el9.x86_64 135/156 2024-05-03T21:09:05.252 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : ceph-test-2:19.0.0-3450.g9a047fe7.el9.x86_64 139/156 2024-05-03T21:09:05.267 INFO:teuthology.orchestra.run.smithi138.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-3450.g9a047fe7.el9.x86_64 135/156 2024-05-03T21:09:05.267 INFO:teuthology.orchestra.run.smithi138.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-03T21:09:05.267 INFO:teuthology.orchestra.run.smithi138.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-05-03T21:09:05.267 INFO:teuthology.orchestra.run.smithi138.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-05-03T21:09:05.267 INFO:teuthology.orchestra.run.smithi138.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-05-03T21:09:05.267 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-05-03T21:09:05.293 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : rbd-fuse-2:19.0.0-3450.g9a047fe7.el9.x86_64 140/156 2024-05-03T21:09:05.311 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-3450.g9a047 136/156 2024-05-03T21:09:05.333 INFO:teuthology.orchestra.run.smithi138.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-3450.g9a047 136/156 2024-05-03T21:09:05.333 INFO:teuthology.orchestra.run.smithi138.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-03T21:09:05.333 INFO:teuthology.orchestra.run.smithi138.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-05-03T21:09:05.334 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-05-03T21:09:05.334 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : rbd-nbd-2:19.0.0-3450.g9a047fe7.el9.x86_64 141/156 2024-05-03T21:09:05.366 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 142/156 2024-05-03T21:09:05.378 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : cephfs-mirror-2:19.0.0-3450.g9a047fe7.el9.x86_64 137/156 2024-05-03T21:09:05.393 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 143/156 2024-05-03T21:09:05.401 INFO:teuthology.orchestra.run.smithi138.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-3450.g9a047fe7.el9.x86_64 137/156 2024-05-03T21:09:05.401 INFO:teuthology.orchestra.run.smithi138.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-03T21:09:05.401 INFO:teuthology.orchestra.run.smithi138.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-05-03T21:09:05.401 INFO:teuthology.orchestra.run.smithi138.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-05-03T21:09:05.401 INFO:teuthology.orchestra.run.smithi138.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-05-03T21:09:05.401 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-05-03T21:09:05.568 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : cephfs-shell-2:19.0.0-3450.g9a047fe7.el9.x86_64 144/156 2024-05-03T21:09:05.614 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : postgresql-contrib-13.14-1.el9.x86_64 145/156 2024-05-03T21:09:05.644 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : libcephfs-devel-2:19.0.0-3450.g9a047fe7.el9.x86_ 146/156 2024-05-03T21:09:05.704 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : rbd-mirror-2:19.0.0-3450.g9a047fe7.el9.x86_64 138/156 2024-05-03T21:09:05.727 INFO:teuthology.orchestra.run.smithi138.stdout: Running scriptlet: rbd-mirror-2:19.0.0-3450.g9a047fe7.el9.x86_64 138/156 2024-05-03T21:09:05.727 INFO:teuthology.orchestra.run.smithi138.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-03T21:09:05.727 INFO:teuthology.orchestra.run.smithi138.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-05-03T21:09:05.727 INFO:teuthology.orchestra.run.smithi138.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-05-03T21:09:05.727 INFO:teuthology.orchestra.run.smithi138.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-05-03T21:09:05.727 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-05-03T21:09:06.277 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 147/156 2024-05-03T21:09:06.316 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : postgresql-server-13.14-1.el9.x86_64 147/156 2024-05-03T21:09:06.337 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 147/156 2024-05-03T21:09:07.167 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : cephfs-top-2:19.0.0-3450.g9a047fe7.el9.noarch 148/156 2024-05-03T21:09:07.254 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : openssl-devel-1:3.2.1-1.el9.x86_64 149/156 2024-05-03T21:09:07.295 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 150/156 2024-05-03T21:09:07.376 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : flex-2.6.4-9.el9.x86_64 151/156 2024-05-03T21:09:07.416 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : bison-3.7.4-5.el9.x86_64 152/156 2024-05-03T21:09:07.416 INFO:teuthology.orchestra.run.smithi082.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 153/156 2024-05-03T21:09:07.500 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 153/156 2024-05-03T21:09:07.525 INFO:teuthology.orchestra.run.smithi082.stdout: Cleanup : openssl-1:3.0.7-27.el9.x86_64 154/156 2024-05-03T21:09:07.525 INFO:teuthology.orchestra.run.smithi082.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 155/156 2024-05-03T21:09:07.618 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 155/156 2024-05-03T21:09:07.682 INFO:teuthology.orchestra.run.smithi082.stdout: Cleanup : openssl-libs-1:3.0.7-27.el9.x86_64 156/156 2024-05-03T21:09:09.023 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: ceph-selinux-2:19.0.0-3450.g9a047fe7.el9.x86_64 107/156 2024-05-03T21:09:09.023 INFO:teuthology.orchestra.run.smithi067.stdout:skipping the directory /sys 2024-05-03T21:09:09.023 INFO:teuthology.orchestra.run.smithi067.stdout:skipping the directory /proc 2024-05-03T21:09:09.023 INFO:teuthology.orchestra.run.smithi067.stdout:skipping the directory /mnt 2024-05-03T21:09:09.023 INFO:teuthology.orchestra.run.smithi067.stdout:skipping the directory /var/tmp 2024-05-03T21:09:09.023 INFO:teuthology.orchestra.run.smithi067.stdout:skipping the directory /home 2024-05-03T21:09:09.023 INFO:teuthology.orchestra.run.smithi067.stdout:skipping the directory /root 2024-05-03T21:09:09.023 INFO:teuthology.orchestra.run.smithi067.stdout:skipping the directory /tmp 2024-05-03T21:09:09.023 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-05-03T21:09:09.196 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : ceph-volume-2:19.0.0-3450.g9a047fe7.el9.noarch 108/156 2024-05-03T21:09:09.212 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: ceph-volume-2:19.0.0-3450.g9a047fe7.el9.noarch 108/156 2024-05-03T21:09:09.212 INFO:teuthology.orchestra.run.smithi067.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-03T21:09:09.213 INFO:teuthology.orchestra.run.smithi067.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-05-03T21:09:09.213 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-05-03T21:09:10.307 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : ceph-test-2:19.0.0-3450.g9a047fe7.el9.x86_64 139/156 2024-05-03T21:09:10.348 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : rbd-fuse-2:19.0.0-3450.g9a047fe7.el9.x86_64 140/156 2024-05-03T21:09:10.397 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : rbd-nbd-2:19.0.0-3450.g9a047fe7.el9.x86_64 141/156 2024-05-03T21:09:10.429 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 142/156 2024-05-03T21:09:10.457 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 143/156 2024-05-03T21:09:10.639 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : cephfs-shell-2:19.0.0-3450.g9a047fe7.el9.x86_64 144/156 2024-05-03T21:09:10.686 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : postgresql-contrib-13.14-1.el9.x86_64 145/156 2024-05-03T21:09:10.716 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : libcephfs-devel-2:19.0.0-3450.g9a047fe7.el9.x86_ 146/156 2024-05-03T21:09:10.757 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : ceph-osd-2:19.0.0-3450.g9a047fe7.el9.x86_64 109/156 2024-05-03T21:09:10.786 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: ceph-osd-2:19.0.0-3450.g9a047fe7.el9.x86_64 109/156 2024-05-03T21:09:10.786 INFO:teuthology.orchestra.run.smithi067.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-03T21:09:10.786 INFO:teuthology.orchestra.run.smithi067.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-05-03T21:09:10.786 INFO:teuthology.orchestra.run.smithi067.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-05-03T21:09:10.786 INFO:teuthology.orchestra.run.smithi067.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-05-03T21:09:10.786 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-05-03T21:09:11.025 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : ceph-mds-2:19.0.0-3450.g9a047fe7.el9.x86_64 110/156 2024-05-03T21:09:11.048 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: ceph-mds-2:19.0.0-3450.g9a047fe7.el9.x86_64 110/156 2024-05-03T21:09:11.049 INFO:teuthology.orchestra.run.smithi067.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-03T21:09:11.049 INFO:teuthology.orchestra.run.smithi067.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-05-03T21:09:11.049 INFO:teuthology.orchestra.run.smithi067.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-05-03T21:09:11.049 INFO:teuthology.orchestra.run.smithi067.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-05-03T21:09:11.049 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-05-03T21:09:11.319 INFO:teuthology.orchestra.run.smithi138.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 147/156 2024-05-03T21:09:11.363 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : postgresql-server-13.14-1.el9.x86_64 147/156 2024-05-03T21:09:11.384 INFO:teuthology.orchestra.run.smithi138.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 147/156 2024-05-03T21:09:11.508 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : ceph-mon-2:19.0.0-3450.g9a047fe7.el9.x86_64 111/156 2024-05-03T21:09:11.532 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: ceph-mon-2:19.0.0-3450.g9a047fe7.el9.x86_64 111/156 2024-05-03T21:09:11.532 INFO:teuthology.orchestra.run.smithi067.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-03T21:09:11.532 INFO:teuthology.orchestra.run.smithi067.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-05-03T21:09:11.532 INFO:teuthology.orchestra.run.smithi067.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-05-03T21:09:11.532 INFO:teuthology.orchestra.run.smithi067.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-05-03T21:09:11.532 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-05-03T21:09:11.640 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : automake-1.16.2-8.el9.noarch 112/156 2024-05-03T21:09:11.698 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-ply-3.11-14.el9.noarch 113/156 2024-05-03T21:09:11.761 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 114/156 2024-05-03T21:09:11.956 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 115/156 2024-05-03T21:09:12.004 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 116/156 2024-05-03T21:09:12.084 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 117/156 2024-05-03T21:09:12.182 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 118/156 2024-05-03T21:09:12.203 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : cephfs-top-2:19.0.0-3450.g9a047fe7.el9.noarch 148/156 2024-05-03T21:09:12.293 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : openssl-devel-1:3.2.1-1.el9.x86_64 149/156 2024-05-03T21:09:12.324 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 119/156 2024-05-03T21:09:12.334 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 150/156 2024-05-03T21:09:12.370 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 120/156 2024-05-03T21:09:12.402 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 121/156 2024-05-03T21:09:12.415 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : flex-2.6.4-9.el9.x86_64 151/156 2024-05-03T21:09:12.429 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : fuse-2.9.9-15.el9.x86_64 122/156 2024-05-03T21:09:12.447 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : bison-3.7.4-5.el9.x86_64 152/156 2024-05-03T21:09:12.447 INFO:teuthology.orchestra.run.smithi138.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 153/156 2024-05-03T21:09:12.531 INFO:teuthology.orchestra.run.smithi138.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 153/156 2024-05-03T21:09:12.564 INFO:teuthology.orchestra.run.smithi138.stdout: Cleanup : openssl-1:3.0.7-27.el9.x86_64 154/156 2024-05-03T21:09:12.564 INFO:teuthology.orchestra.run.smithi138.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 155/156 2024-05-03T21:09:12.653 INFO:teuthology.orchestra.run.smithi138.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 155/156 2024-05-03T21:09:12.720 INFO:teuthology.orchestra.run.smithi138.stdout: Cleanup : openssl-libs-1:3.0.7-27.el9.x86_64 156/156 2024-05-03T21:09:12.907 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: cephadm-2:19.0.0-3450.g9a047fe7.el9.noarch 123/156 2024-05-03T21:09:12.945 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : cephadm-2:19.0.0-3450.g9a047fe7.el9.noarch 123/156 2024-05-03T21:09:12.977 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-3450.g9a047fe7.e 124/156 2024-05-03T21:09:13.049 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-3450.g9a047fe7. 125/156 2024-05-03T21:09:13.076 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-3450.g9a047fe7.el9.noa 126/156 2024-05-03T21:09:13.259 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: openssl-libs-1:3.0.7-27.el9.x86_64 156/156 2024-05-03T21:09:13.259 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : ceph-2:19.0.0-3450.g9a047fe7.el9.x86_64 1/156 2024-05-03T21:09:13.259 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : ceph-base-2:19.0.0-3450.g9a047fe7.el9.x86_64 2/156 2024-05-03T21:09:13.259 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : ceph-common-2:19.0.0-3450.g9a047fe7.el9.x86_64 3/156 2024-05-03T21:09:13.259 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : ceph-fuse-2:19.0.0-3450.g9a047fe7.el9.x86_64 4/156 2024-05-03T21:09:13.259 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-3450.g9a047 5/156 2024-05-03T21:09:13.260 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : ceph-mds-2:19.0.0-3450.g9a047fe7.el9.x86_64 6/156 2024-05-03T21:09:13.260 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : ceph-mgr-2:19.0.0-3450.g9a047fe7.el9.x86_64 7/156 2024-05-03T21:09:13.260 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : ceph-mon-2:19.0.0-3450.g9a047fe7.el9.x86_64 8/156 2024-05-03T21:09:13.260 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : ceph-osd-2:19.0.0-3450.g9a047fe7.el9.x86_64 9/156 2024-05-03T21:09:13.260 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : ceph-radosgw-2:19.0.0-3450.g9a047fe7.el9.x86_64 10/156 2024-05-03T21:09:13.260 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : ceph-selinux-2:19.0.0-3450.g9a047fe7.el9.x86_64 11/156 2024-05-03T21:09:13.260 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : ceph-test-2:19.0.0-3450.g9a047fe7.el9.x86_64 12/156 2024-05-03T21:09:13.260 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : cephfs-mirror-2:19.0.0-3450.g9a047fe7.el9.x86_64 13/156 2024-05-03T21:09:13.260 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : cephfs-shell-2:19.0.0-3450.g9a047fe7.el9.x86_64 14/156 2024-05-03T21:09:13.260 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : libcephfs-devel-2:19.0.0-3450.g9a047fe7.el9.x86_ 15/156 2024-05-03T21:09:13.260 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : libcephfs2-2:19.0.0-3450.g9a047fe7.el9.x86_64 16/156 2024-05-03T21:09:13.260 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : libcephsqlite-2:19.0.0-3450.g9a047fe7.el9.x86_64 17/156 2024-05-03T21:09:13.260 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : librados-devel-2:19.0.0-3450.g9a047fe7.el9.x86_6 18/156 2024-05-03T21:09:13.260 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : libradosstriper1-2:19.0.0-3450.g9a047fe7.el9.x86 19/156 2024-05-03T21:09:13.261 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : librgw2-2:19.0.0-3450.g9a047fe7.el9.x86_64 20/156 2024-05-03T21:09:13.261 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-ceph-argparse-2:19.0.0-3450.g9a047fe7.el 21/156 2024-05-03T21:09:13.261 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-ceph-common-2:19.0.0-3450.g9a047fe7.el9. 22/156 2024-05-03T21:09:13.261 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-cephfs-2:19.0.0-3450.g9a047fe7.el9.x86_6 23/156 2024-05-03T21:09:13.261 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-rados-2:19.0.0-3450.g9a047fe7.el9.x86_64 24/156 2024-05-03T21:09:13.261 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-rbd-2:19.0.0-3450.g9a047fe7.el9.x86_64 25/156 2024-05-03T21:09:13.261 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-rgw-2:19.0.0-3450.g9a047fe7.el9.x86_64 26/156 2024-05-03T21:09:13.261 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : rbd-fuse-2:19.0.0-3450.g9a047fe7.el9.x86_64 27/156 2024-05-03T21:09:13.276 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : rbd-mirror-2:19.0.0-3450.g9a047fe7.el9.x86_64 28/156 2024-05-03T21:09:13.276 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : rbd-nbd-2:19.0.0-3450.g9a047fe7.el9.x86_64 29/156 2024-05-03T21:09:13.276 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-3450.g9a047fe7. 30/156 2024-05-03T21:09:13.276 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-3450.g9a047fe7.el9.noa 31/156 2024-05-03T21:09:13.276 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-3450.g9a047fe7.el9.n 32/156 2024-05-03T21:09:13.276 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-3450.g9a0 33/156 2024-05-03T21:09:13.276 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-3450.g9a047fe7.el 34/156 2024-05-03T21:09:13.276 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : ceph-mgr-rook-2:19.0.0-3450.g9a047fe7.el9.noarch 35/156 2024-05-03T21:09:13.276 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-3450.g9a047fe7.e 36/156 2024-05-03T21:09:13.276 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : ceph-volume-2:19.0.0-3450.g9a047fe7.el9.noarch 37/156 2024-05-03T21:09:13.276 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : cephadm-2:19.0.0-3450.g9a047fe7.el9.noarch 38/156 2024-05-03T21:09:13.276 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : cephfs-top-2:19.0.0-3450.g9a047fe7.el9.noarch 39/156 2024-05-03T21:09:13.277 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 40/156 2024-05-03T21:09:13.277 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 41/156 2024-05-03T21:09:13.277 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 42/156 2024-05-03T21:09:13.277 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 43/156 2024-05-03T21:09:13.277 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 44/156 2024-05-03T21:09:13.277 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-ply-3.11-14.el9.noarch 45/156 2024-05-03T21:09:13.277 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 46/156 2024-05-03T21:09:13.277 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 47/156 2024-05-03T21:09:13.277 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 48/156 2024-05-03T21:09:13.277 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : automake-1.16.2-8.el9.noarch 49/156 2024-05-03T21:09:13.277 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : bison-3.7.4-5.el9.x86_64 50/156 2024-05-03T21:09:13.277 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 51/156 2024-05-03T21:09:13.278 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 52/156 2024-05-03T21:09:13.278 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : flex-2.6.4-9.el9.x86_64 53/156 2024-05-03T21:09:13.278 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 54/156 2024-05-03T21:09:13.278 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 55/156 2024-05-03T21:09:13.278 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 56/156 2024-05-03T21:09:13.278 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 57/156 2024-05-03T21:09:13.279 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 58/156 2024-05-03T21:09:13.279 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 59/156 2024-05-03T21:09:13.279 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 60/156 2024-05-03T21:09:13.279 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 61/156 2024-05-03T21:09:13.279 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 62/156 2024-05-03T21:09:13.279 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 63/156 2024-05-03T21:09:13.279 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 64/156 2024-05-03T21:09:13.279 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : lua-5.4.4-4.el9.x86_64 65/156 2024-05-03T21:09:13.280 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 66/156 2024-05-03T21:09:13.280 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 67/156 2024-05-03T21:09:13.280 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : openssl-devel-1:3.2.1-1.el9.x86_64 68/156 2024-05-03T21:09:13.280 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : postgresql-13.14-1.el9.x86_64 69/156 2024-05-03T21:09:13.280 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : postgresql-contrib-13.14-1.el9.x86_64 70/156 2024-05-03T21:09:13.280 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : postgresql-private-libs-13.14-1.el9.x86_64 71/156 2024-05-03T21:09:13.280 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : postgresql-server-13.14-1.el9.x86_64 72/156 2024-05-03T21:09:13.280 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 73/156 2024-05-03T21:09:13.281 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 74/156 2024-05-03T21:09:13.281 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 75/156 2024-05-03T21:09:13.281 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-devel-3.9.18-3.el9.x86_64 76/156 2024-05-03T21:09:13.281 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 77/156 2024-05-03T21:09:13.281 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 78/156 2024-05-03T21:09:13.281 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 79/156 2024-05-03T21:09:13.281 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 80/156 2024-05-03T21:09:13.281 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 81/156 2024-05-03T21:09:13.281 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 82/156 2024-05-03T21:09:13.281 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 83/156 2024-05-03T21:09:13.281 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 84/156 2024-05-03T21:09:13.281 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 85/156 2024-05-03T21:09:13.281 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 86/156 2024-05-03T21:09:13.282 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 87/156 2024-05-03T21:09:13.282 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 88/156 2024-05-03T21:09:13.282 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 89/156 2024-05-03T21:09:13.282 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 90/156 2024-05-03T21:09:13.282 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 91/156 2024-05-03T21:09:13.282 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 92/156 2024-05-03T21:09:13.282 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 93/156 2024-05-03T21:09:13.282 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 94/156 2024-05-03T21:09:13.282 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 95/156 2024-05-03T21:09:13.282 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 96/156 2024-05-03T21:09:13.283 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 97/156 2024-05-03T21:09:13.283 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 98/156 2024-05-03T21:09:13.283 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 99/156 2024-05-03T21:09:13.283 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 100/156 2024-05-03T21:09:13.283 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 101/156 2024-05-03T21:09:13.283 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : zlib-devel-1.2.11-41.el9.x86_64 102/156 2024-05-03T21:09:13.283 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 103/156 2024-05-03T21:09:13.283 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 104/156 2024-05-03T21:09:13.283 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 105/156 2024-05-03T21:09:13.283 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 106/156 2024-05-03T21:09:13.283 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 107/156 2024-05-03T21:09:13.283 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : libarrow-9.0.0-11.el9.x86_64 108/156 2024-05-03T21:09:13.283 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : libarrow-doc-9.0.0-11.el9.noarch 109/156 2024-05-03T21:09:13.283 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 110/156 2024-05-03T21:09:13.284 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 111/156 2024-05-03T21:09:13.284 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 112/156 2024-05-03T21:09:13.284 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : parquet-libs-9.0.0-11.el9.x86_64 113/156 2024-05-03T21:09:13.284 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 114/156 2024-05-03T21:09:13.284 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 115/156 2024-05-03T21:09:13.285 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 116/156 2024-05-03T21:09:13.285 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 117/156 2024-05-03T21:09:13.285 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 118/156 2024-05-03T21:09:13.285 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 119/156 2024-05-03T21:09:13.285 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 120/156 2024-05-03T21:09:13.285 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-colorama-0.4.4-12.el9.noarch 121/156 2024-05-03T21:09:13.285 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-google-auth-1:2.29.0-1.el9.noarch 122/156 2024-05-03T21:09:13.285 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 123/156 2024-05-03T21:09:13.285 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 124/156 2024-05-03T21:09:13.285 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 125/156 2024-05-03T21:09:13.285 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 126/156 2024-05-03T21:09:13.285 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 127/156 2024-05-03T21:09:13.285 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 128/156 2024-05-03T21:09:13.286 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 129/156 2024-05-03T21:09:13.286 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 130/156 2024-05-03T21:09:13.286 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 131/156 2024-05-03T21:09:13.286 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 132/156 2024-05-03T21:09:13.286 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 133/156 2024-05-03T21:09:13.286 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 134/156 2024-05-03T21:09:13.286 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 135/156 2024-05-03T21:09:13.286 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 136/156 2024-05-03T21:09:13.286 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 137/156 2024-05-03T21:09:13.286 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 138/156 2024-05-03T21:09:13.286 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 139/156 2024-05-03T21:09:13.286 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 140/156 2024-05-03T21:09:13.286 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 141/156 2024-05-03T21:09:13.287 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 142/156 2024-05-03T21:09:13.287 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 143/156 2024-05-03T21:09:13.287 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 144/156 2024-05-03T21:09:13.287 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 145/156 2024-05-03T21:09:13.287 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 146/156 2024-05-03T21:09:13.287 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : re2-1:20211101-3.el9.x86_64 147/156 2024-05-03T21:09:13.287 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 148/156 2024-05-03T21:09:13.287 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : librados2-2:19.0.0-3450.g9a047fe7.el9.x86_64 149/156 2024-05-03T21:09:13.287 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 150/156 2024-05-03T21:09:13.287 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : librbd1-2:19.0.0-3450.g9a047fe7.el9.x86_64 151/156 2024-05-03T21:09:13.287 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 152/156 2024-05-03T21:09:13.287 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : openssl-1:3.2.1-1.el9.x86_64 153/156 2024-05-03T21:09:13.287 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 154/156 2024-05-03T21:09:13.287 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : openssl-libs-1:3.2.1-1.el9.x86_64 155/156 2024-05-03T21:09:14.168 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-3450.g9a047fe7.el9.noa 126/156 2024-05-03T21:09:14.203 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-3450.g9a047fe7.el9.n 127/156 2024-05-03T21:09:15.216 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-3450.g9a047fe7.el9.n 127/156 2024-05-03T21:09:15.236 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-3450.g9a0 128/156 2024-05-03T21:09:15.356 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-3450.g9a0 128/156 2024-05-03T21:09:15.414 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 156/156 2024-05-03T21:09:15.414 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-05-03T21:09:15.414 INFO:teuthology.orchestra.run.smithi082.stdout:Upgraded: 2024-05-03T21:09:15.414 INFO:teuthology.orchestra.run.smithi082.stdout: librados2-2:19.0.0-3450.g9a047fe7.el9.x86_64 2024-05-03T21:09:15.414 INFO:teuthology.orchestra.run.smithi082.stdout: librbd1-2:19.0.0-3450.g9a047fe7.el9.x86_64 2024-05-03T21:09:15.414 INFO:teuthology.orchestra.run.smithi082.stdout: openssl-1:3.2.1-1.el9.x86_64 2024-05-03T21:09:15.414 INFO:teuthology.orchestra.run.smithi082.stdout: openssl-libs-1:3.2.1-1.el9.x86_64 2024-05-03T21:09:15.414 INFO:teuthology.orchestra.run.smithi082.stdout:Installed: 2024-05-03T21:09:15.414 INFO:teuthology.orchestra.run.smithi082.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-05-03T21:09:15.414 INFO:teuthology.orchestra.run.smithi082.stdout: automake-1.16.2-8.el9.noarch 2024-05-03T21:09:15.414 INFO:teuthology.orchestra.run.smithi082.stdout: bison-3.7.4-5.el9.x86_64 2024-05-03T21:09:15.415 INFO:teuthology.orchestra.run.smithi082.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-05-03T21:09:15.415 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-2:19.0.0-3450.g9a047fe7.el9.x86_64 2024-05-03T21:09:15.415 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-base-2:19.0.0-3450.g9a047fe7.el9.x86_64 2024-05-03T21:09:15.415 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-common-2:19.0.0-3450.g9a047fe7.el9.x86_64 2024-05-03T21:09:15.415 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-fuse-2:19.0.0-3450.g9a047fe7.el9.x86_64 2024-05-03T21:09:15.415 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-grafana-dashboards-2:19.0.0-3450.g9a047fe7.el9.noarch 2024-05-03T21:09:15.415 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-immutable-object-cache-2:19.0.0-3450.g9a047fe7.el9.x86_64 2024-05-03T21:09:15.415 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-mds-2:19.0.0-3450.g9a047fe7.el9.x86_64 2024-05-03T21:09:15.415 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-mgr-2:19.0.0-3450.g9a047fe7.el9.x86_64 2024-05-03T21:09:15.415 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-mgr-cephadm-2:19.0.0-3450.g9a047fe7.el9.noarch 2024-05-03T21:09:15.415 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-mgr-dashboard-2:19.0.0-3450.g9a047fe7.el9.noarch 2024-05-03T21:09:15.415 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-mgr-diskprediction-local-2:19.0.0-3450.g9a047fe7.el9.noarch 2024-05-03T21:09:15.415 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-mgr-modules-core-2:19.0.0-3450.g9a047fe7.el9.noarch 2024-05-03T21:09:15.415 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-mgr-rook-2:19.0.0-3450.g9a047fe7.el9.noarch 2024-05-03T21:09:15.416 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-mon-2:19.0.0-3450.g9a047fe7.el9.x86_64 2024-05-03T21:09:15.416 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-osd-2:19.0.0-3450.g9a047fe7.el9.x86_64 2024-05-03T21:09:15.416 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-prometheus-alerts-2:19.0.0-3450.g9a047fe7.el9.noarch 2024-05-03T21:09:15.416 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-radosgw-2:19.0.0-3450.g9a047fe7.el9.x86_64 2024-05-03T21:09:15.416 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-selinux-2:19.0.0-3450.g9a047fe7.el9.x86_64 2024-05-03T21:09:15.416 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-test-2:19.0.0-3450.g9a047fe7.el9.x86_64 2024-05-03T21:09:15.416 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-volume-2:19.0.0-3450.g9a047fe7.el9.noarch 2024-05-03T21:09:15.416 INFO:teuthology.orchestra.run.smithi082.stdout: cephadm-2:19.0.0-3450.g9a047fe7.el9.noarch 2024-05-03T21:09:15.416 INFO:teuthology.orchestra.run.smithi082.stdout: cephfs-mirror-2:19.0.0-3450.g9a047fe7.el9.x86_64 2024-05-03T21:09:15.416 INFO:teuthology.orchestra.run.smithi082.stdout: cephfs-shell-2:19.0.0-3450.g9a047fe7.el9.x86_64 2024-05-03T21:09:15.416 INFO:teuthology.orchestra.run.smithi082.stdout: cephfs-top-2:19.0.0-3450.g9a047fe7.el9.noarch 2024-05-03T21:09:15.416 INFO:teuthology.orchestra.run.smithi082.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-05-03T21:09:15.416 INFO:teuthology.orchestra.run.smithi082.stdout: flex-2.6.4-9.el9.x86_64 2024-05-03T21:09:15.416 INFO:teuthology.orchestra.run.smithi082.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-05-03T21:09:15.417 INFO:teuthology.orchestra.run.smithi082.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-05-03T21:09:15.417 INFO:teuthology.orchestra.run.smithi082.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-05-03T21:09:15.417 INFO:teuthology.orchestra.run.smithi082.stdout: fuse-2.9.9-15.el9.x86_64 2024-05-03T21:09:15.417 INFO:teuthology.orchestra.run.smithi082.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-05-03T21:09:15.417 INFO:teuthology.orchestra.run.smithi082.stdout: grpc-data-1.46.7-10.el9.noarch 2024-05-03T21:09:15.417 INFO:teuthology.orchestra.run.smithi082.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-05-03T21:09:15.417 INFO:teuthology.orchestra.run.smithi082.stdout: libarrow-9.0.0-11.el9.x86_64 2024-05-03T21:09:15.417 INFO:teuthology.orchestra.run.smithi082.stdout: libarrow-doc-9.0.0-11.el9.noarch 2024-05-03T21:09:15.417 INFO:teuthology.orchestra.run.smithi082.stdout: libcephfs-devel-2:19.0.0-3450.g9a047fe7.el9.x86_64 2024-05-03T21:09:15.417 INFO:teuthology.orchestra.run.smithi082.stdout: libcephfs2-2:19.0.0-3450.g9a047fe7.el9.x86_64 2024-05-03T21:09:15.417 INFO:teuthology.orchestra.run.smithi082.stdout: libcephsqlite-2:19.0.0-3450.g9a047fe7.el9.x86_64 2024-05-03T21:09:15.417 INFO:teuthology.orchestra.run.smithi082.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-05-03T21:09:15.417 INFO:teuthology.orchestra.run.smithi082.stdout: liboath-2.6.7-2.el9.x86_64 2024-05-03T21:09:15.418 INFO:teuthology.orchestra.run.smithi082.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-05-03T21:09:15.418 INFO:teuthology.orchestra.run.smithi082.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-05-03T21:09:15.418 INFO:teuthology.orchestra.run.smithi082.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-05-03T21:09:15.418 INFO:teuthology.orchestra.run.smithi082.stdout: librados-devel-2:19.0.0-3450.g9a047fe7.el9.x86_64 2024-05-03T21:09:15.418 INFO:teuthology.orchestra.run.smithi082.stdout: libradosstriper1-2:19.0.0-3450.g9a047fe7.el9.x86_64 2024-05-03T21:09:15.418 INFO:teuthology.orchestra.run.smithi082.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-05-03T21:09:15.418 INFO:teuthology.orchestra.run.smithi082.stdout: librgw2-2:19.0.0-3450.g9a047fe7.el9.x86_64 2024-05-03T21:09:15.418 INFO:teuthology.orchestra.run.smithi082.stdout: libtool-2.4.6-45.el9.x86_64 2024-05-03T21:09:15.418 INFO:teuthology.orchestra.run.smithi082.stdout: libunwind-1.6.2-1.el9.x86_64 2024-05-03T21:09:15.418 INFO:teuthology.orchestra.run.smithi082.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-05-03T21:09:15.418 INFO:teuthology.orchestra.run.smithi082.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-05-03T21:09:15.418 INFO:teuthology.orchestra.run.smithi082.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-05-03T21:09:15.418 INFO:teuthology.orchestra.run.smithi082.stdout: lua-5.4.4-4.el9.x86_64 2024-05-03T21:09:15.418 INFO:teuthology.orchestra.run.smithi082.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-05-03T21:09:15.419 INFO:teuthology.orchestra.run.smithi082.stdout: luarocks-3.9.2-1.el9.noarch 2024-05-03T21:09:15.419 INFO:teuthology.orchestra.run.smithi082.stdout: openblas-0.3.21-2.el9.x86_64 2024-05-03T21:09:15.419 INFO:teuthology.orchestra.run.smithi082.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-05-03T21:09:15.419 INFO:teuthology.orchestra.run.smithi082.stdout: openssl-devel-1:3.2.1-1.el9.x86_64 2024-05-03T21:09:15.419 INFO:teuthology.orchestra.run.smithi082.stdout: parquet-libs-9.0.0-11.el9.x86_64 2024-05-03T21:09:15.419 INFO:teuthology.orchestra.run.smithi082.stdout: postgresql-13.14-1.el9.x86_64 2024-05-03T21:09:15.419 INFO:teuthology.orchestra.run.smithi082.stdout: postgresql-contrib-13.14-1.el9.x86_64 2024-05-03T21:09:15.419 INFO:teuthology.orchestra.run.smithi082.stdout: postgresql-private-libs-13.14-1.el9.x86_64 2024-05-03T21:09:15.419 INFO:teuthology.orchestra.run.smithi082.stdout: postgresql-server-13.14-1.el9.x86_64 2024-05-03T21:09:15.419 INFO:teuthology.orchestra.run.smithi082.stdout: protobuf-3.14.0-13.el9.x86_64 2024-05-03T21:09:15.419 INFO:teuthology.orchestra.run.smithi082.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-05-03T21:09:15.419 INFO:teuthology.orchestra.run.smithi082.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-05-03T21:09:15.419 INFO:teuthology.orchestra.run.smithi082.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-05-03T21:09:15.420 INFO:teuthology.orchestra.run.smithi082.stdout: python3-babel-2.9.1-2.el9.noarch 2024-05-03T21:09:15.420 INFO:teuthology.orchestra.run.smithi082.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-05-03T21:09:15.420 INFO:teuthology.orchestra.run.smithi082.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-05-03T21:09:15.420 INFO:teuthology.orchestra.run.smithi082.stdout: python3-ceph-argparse-2:19.0.0-3450.g9a047fe7.el9.x86_64 2024-05-03T21:09:15.420 INFO:teuthology.orchestra.run.smithi082.stdout: python3-ceph-common-2:19.0.0-3450.g9a047fe7.el9.x86_64 2024-05-03T21:09:15.420 INFO:teuthology.orchestra.run.smithi082.stdout: python3-cephfs-2:19.0.0-3450.g9a047fe7.el9.x86_64 2024-05-03T21:09:15.420 INFO:teuthology.orchestra.run.smithi082.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-05-03T21:09:15.420 INFO:teuthology.orchestra.run.smithi082.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-05-03T21:09:15.420 INFO:teuthology.orchestra.run.smithi082.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-05-03T21:09:15.420 INFO:teuthology.orchestra.run.smithi082.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-05-03T21:09:15.420 INFO:teuthology.orchestra.run.smithi082.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-05-03T21:09:15.421 INFO:teuthology.orchestra.run.smithi082.stdout: python3-colorama-0.4.4-12.el9.noarch 2024-05-03T21:09:15.421 INFO:teuthology.orchestra.run.smithi082.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-05-03T21:09:15.421 INFO:teuthology.orchestra.run.smithi082.stdout: python3-devel-3.9.18-3.el9.x86_64 2024-05-03T21:09:15.421 INFO:teuthology.orchestra.run.smithi082.stdout: python3-google-auth-1:2.29.0-1.el9.noarch 2024-05-03T21:09:15.421 INFO:teuthology.orchestra.run.smithi082.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-05-03T21:09:15.421 INFO:teuthology.orchestra.run.smithi082.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-05-03T21:09:15.421 INFO:teuthology.orchestra.run.smithi082.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-05-03T21:09:15.421 INFO:teuthology.orchestra.run.smithi082.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-05-03T21:09:15.421 INFO:teuthology.orchestra.run.smithi082.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-05-03T21:09:15.421 INFO:teuthology.orchestra.run.smithi082.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-05-03T21:09:15.421 INFO:teuthology.orchestra.run.smithi082.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-05-03T21:09:15.421 INFO:teuthology.orchestra.run.smithi082.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-05-03T21:09:15.421 INFO:teuthology.orchestra.run.smithi082.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-05-03T21:09:15.421 INFO:teuthology.orchestra.run.smithi082.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-05-03T21:09:15.421 INFO:teuthology.orchestra.run.smithi082.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-05-03T21:09:15.422 INFO:teuthology.orchestra.run.smithi082.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-05-03T21:09:15.422 INFO:teuthology.orchestra.run.smithi082.stdout: python3-mako-1.1.4-6.el9.noarch 2024-05-03T21:09:15.422 INFO:teuthology.orchestra.run.smithi082.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-05-03T21:09:15.422 INFO:teuthology.orchestra.run.smithi082.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-05-03T21:09:15.422 INFO:teuthology.orchestra.run.smithi082.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-05-03T21:09:15.422 INFO:teuthology.orchestra.run.smithi082.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-05-03T21:09:15.422 INFO:teuthology.orchestra.run.smithi082.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-05-03T21:09:15.422 INFO:teuthology.orchestra.run.smithi082.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-05-03T21:09:15.422 INFO:teuthology.orchestra.run.smithi082.stdout: python3-packaging-20.9-5.el9.noarch 2024-05-03T21:09:15.422 INFO:teuthology.orchestra.run.smithi082.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-05-03T21:09:15.422 INFO:teuthology.orchestra.run.smithi082.stdout: python3-ply-3.11-14.el9.noarch 2024-05-03T21:09:15.422 INFO:teuthology.orchestra.run.smithi082.stdout: python3-portend-3.1.0-2.el9.noarch 2024-05-03T21:09:15.422 INFO:teuthology.orchestra.run.smithi082.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-05-03T21:09:15.422 INFO:teuthology.orchestra.run.smithi082.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-05-03T21:09:15.423 INFO:teuthology.orchestra.run.smithi082.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-05-03T21:09:15.423 INFO:teuthology.orchestra.run.smithi082.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-05-03T21:09:15.423 INFO:teuthology.orchestra.run.smithi082.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-05-03T21:09:15.423 INFO:teuthology.orchestra.run.smithi082.stdout: python3-pycparser-2.20-6.el9.noarch 2024-05-03T21:09:15.423 INFO:teuthology.orchestra.run.smithi082.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-05-03T21:09:15.423 INFO:teuthology.orchestra.run.smithi082.stdout: python3-pytz-2021.1-5.el9.noarch 2024-05-03T21:09:15.423 INFO:teuthology.orchestra.run.smithi082.stdout: python3-rados-2:19.0.0-3450.g9a047fe7.el9.x86_64 2024-05-03T21:09:15.423 INFO:teuthology.orchestra.run.smithi082.stdout: python3-rbd-2:19.0.0-3450.g9a047fe7.el9.x86_64 2024-05-03T21:09:15.423 INFO:teuthology.orchestra.run.smithi082.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-05-03T21:09:15.423 INFO:teuthology.orchestra.run.smithi082.stdout: python3-requests-2.25.1-8.el9.noarch 2024-05-03T21:09:15.423 INFO:teuthology.orchestra.run.smithi082.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-05-03T21:09:15.423 INFO:teuthology.orchestra.run.smithi082.stdout: python3-rgw-2:19.0.0-3450.g9a047fe7.el9.x86_64 2024-05-03T21:09:15.423 INFO:teuthology.orchestra.run.smithi082.stdout: python3-routes-2.5.1-5.el9.noarch 2024-05-03T21:09:15.423 INFO:teuthology.orchestra.run.smithi082.stdout: python3-rsa-4.9-2.el9.noarch 2024-05-03T21:09:15.423 INFO:teuthology.orchestra.run.smithi082.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-05-03T21:09:15.424 INFO:teuthology.orchestra.run.smithi082.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-05-03T21:09:15.424 INFO:teuthology.orchestra.run.smithi082.stdout: python3-toml-0.10.2-6.el9.noarch 2024-05-03T21:09:15.424 INFO:teuthology.orchestra.run.smithi082.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-05-03T21:09:15.424 INFO:teuthology.orchestra.run.smithi082.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-05-03T21:09:15.424 INFO:teuthology.orchestra.run.smithi082.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-05-03T21:09:15.424 INFO:teuthology.orchestra.run.smithi082.stdout: python3-webob-1.8.7-6.el9.noarch 2024-05-03T21:09:15.424 INFO:teuthology.orchestra.run.smithi082.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-05-03T21:09:15.424 INFO:teuthology.orchestra.run.smithi082.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-05-03T21:09:15.424 INFO:teuthology.orchestra.run.smithi082.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-05-03T21:09:15.424 INFO:teuthology.orchestra.run.smithi082.stdout: qatlib-24.02.0-1.el9.x86_64 2024-05-03T21:09:15.424 INFO:teuthology.orchestra.run.smithi082.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-05-03T21:09:15.424 INFO:teuthology.orchestra.run.smithi082.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-05-03T21:09:15.424 INFO:teuthology.orchestra.run.smithi082.stdout: rbd-fuse-2:19.0.0-3450.g9a047fe7.el9.x86_64 2024-05-03T21:09:15.424 INFO:teuthology.orchestra.run.smithi082.stdout: rbd-mirror-2:19.0.0-3450.g9a047fe7.el9.x86_64 2024-05-03T21:09:15.424 INFO:teuthology.orchestra.run.smithi082.stdout: rbd-nbd-2:19.0.0-3450.g9a047fe7.el9.x86_64 2024-05-03T21:09:15.425 INFO:teuthology.orchestra.run.smithi082.stdout: re2-1:20211101-3.el9.x86_64 2024-05-03T21:09:15.425 INFO:teuthology.orchestra.run.smithi082.stdout: socat-1.7.4.1-5.el9.x86_64 2024-05-03T21:09:15.425 INFO:teuthology.orchestra.run.smithi082.stdout: thrift-0.15.0-2.el9.x86_64 2024-05-03T21:09:15.425 INFO:teuthology.orchestra.run.smithi082.stdout: uuid-1.6.2-55.el9.x86_64 2024-05-03T21:09:15.425 INFO:teuthology.orchestra.run.smithi082.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-05-03T21:09:15.425 INFO:teuthology.orchestra.run.smithi082.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-05-03T21:09:15.425 INFO:teuthology.orchestra.run.smithi082.stdout: zlib-devel-1.2.11-41.el9.x86_64 2024-05-03T21:09:15.425 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-05-03T21:09:15.425 INFO:teuthology.orchestra.run.smithi082.stdout:Complete! 2024-05-03T21:09:15.530 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-3450.g9a047fe7.el 129/156 2024-05-03T21:09:15.555 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : ceph-mgr-2:19.0.0-3450.g9a047fe7.el9.x86_64 130/156 2024-05-03T21:09:15.579 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: ceph-mgr-2:19.0.0-3450.g9a047fe7.el9.x86_64 130/156 2024-05-03T21:09:15.579 INFO:teuthology.orchestra.run.smithi067.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-03T21:09:15.579 INFO:teuthology.orchestra.run.smithi067.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-05-03T21:09:15.579 INFO:teuthology.orchestra.run.smithi067.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-05-03T21:09:15.579 INFO:teuthology.orchestra.run.smithi067.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-05-03T21:09:15.580 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-05-03T21:09:15.631 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : ceph-mgr-rook-2:19.0.0-3450.g9a047fe7.el9.noarch 131/156 2024-05-03T21:09:15.642 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-3450.g9a047fe7.el9.noarch 131/156 2024-05-03T21:09:15.744 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : ceph-2:19.0.0-3450.g9a047fe7.el9.x86_64 132/156 2024-05-03T21:09:15.797 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : ceph-fuse-2:19.0.0-3450.g9a047fe7.el9.x86_64 133/156 2024-05-03T21:09:15.850 DEBUG:teuthology.parallel:result is None 2024-05-03T21:09:16.770 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : libtool-2.4.6-45.el9.x86_64 134/156 2024-05-03T21:09:16.800 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : ceph-radosgw-2:19.0.0-3450.g9a047fe7.el9.x86_64 135/156 2024-05-03T21:09:16.824 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-3450.g9a047fe7.el9.x86_64 135/156 2024-05-03T21:09:16.824 INFO:teuthology.orchestra.run.smithi067.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-03T21:09:16.824 INFO:teuthology.orchestra.run.smithi067.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-05-03T21:09:16.824 INFO:teuthology.orchestra.run.smithi067.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-05-03T21:09:16.824 INFO:teuthology.orchestra.run.smithi067.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-05-03T21:09:16.824 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-05-03T21:09:16.868 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-3450.g9a047 136/156 2024-05-03T21:09:16.890 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-3450.g9a047 136/156 2024-05-03T21:09:16.890 INFO:teuthology.orchestra.run.smithi067.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-03T21:09:16.890 INFO:teuthology.orchestra.run.smithi067.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-05-03T21:09:16.891 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-05-03T21:09:16.943 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : cephfs-mirror-2:19.0.0-3450.g9a047fe7.el9.x86_64 137/156 2024-05-03T21:09:16.966 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-3450.g9a047fe7.el9.x86_64 137/156 2024-05-03T21:09:16.966 INFO:teuthology.orchestra.run.smithi067.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-03T21:09:16.966 INFO:teuthology.orchestra.run.smithi067.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-05-03T21:09:16.966 INFO:teuthology.orchestra.run.smithi067.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-05-03T21:09:16.966 INFO:teuthology.orchestra.run.smithi067.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-05-03T21:09:16.966 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-05-03T21:09:17.285 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : rbd-mirror-2:19.0.0-3450.g9a047fe7.el9.x86_64 138/156 2024-05-03T21:09:17.309 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: rbd-mirror-2:19.0.0-3450.g9a047fe7.el9.x86_64 138/156 2024-05-03T21:09:17.309 INFO:teuthology.orchestra.run.smithi067.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-03T21:09:17.310 INFO:teuthology.orchestra.run.smithi067.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-05-03T21:09:17.310 INFO:teuthology.orchestra.run.smithi067.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-05-03T21:09:17.310 INFO:teuthology.orchestra.run.smithi067.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-05-03T21:09:17.310 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-05-03T21:09:18.814 INFO:teuthology.orchestra.run.smithi138.stdout: Running scriptlet: openssl-libs-1:3.0.7-27.el9.x86_64 156/156 2024-05-03T21:09:18.815 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : ceph-2:19.0.0-3450.g9a047fe7.el9.x86_64 1/156 2024-05-03T21:09:18.815 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : ceph-base-2:19.0.0-3450.g9a047fe7.el9.x86_64 2/156 2024-05-03T21:09:18.815 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : ceph-common-2:19.0.0-3450.g9a047fe7.el9.x86_64 3/156 2024-05-03T21:09:18.815 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : ceph-fuse-2:19.0.0-3450.g9a047fe7.el9.x86_64 4/156 2024-05-03T21:09:18.815 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-3450.g9a047 5/156 2024-05-03T21:09:18.815 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : ceph-mds-2:19.0.0-3450.g9a047fe7.el9.x86_64 6/156 2024-05-03T21:09:18.815 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : ceph-mgr-2:19.0.0-3450.g9a047fe7.el9.x86_64 7/156 2024-05-03T21:09:18.815 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : ceph-mon-2:19.0.0-3450.g9a047fe7.el9.x86_64 8/156 2024-05-03T21:09:18.815 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : ceph-osd-2:19.0.0-3450.g9a047fe7.el9.x86_64 9/156 2024-05-03T21:09:18.815 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : ceph-radosgw-2:19.0.0-3450.g9a047fe7.el9.x86_64 10/156 2024-05-03T21:09:18.815 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : ceph-selinux-2:19.0.0-3450.g9a047fe7.el9.x86_64 11/156 2024-05-03T21:09:18.815 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : ceph-test-2:19.0.0-3450.g9a047fe7.el9.x86_64 12/156 2024-05-03T21:09:18.816 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : cephfs-mirror-2:19.0.0-3450.g9a047fe7.el9.x86_64 13/156 2024-05-03T21:09:18.816 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : cephfs-shell-2:19.0.0-3450.g9a047fe7.el9.x86_64 14/156 2024-05-03T21:09:18.816 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : libcephfs-devel-2:19.0.0-3450.g9a047fe7.el9.x86_ 15/156 2024-05-03T21:09:18.816 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : libcephfs2-2:19.0.0-3450.g9a047fe7.el9.x86_64 16/156 2024-05-03T21:09:18.816 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : libcephsqlite-2:19.0.0-3450.g9a047fe7.el9.x86_64 17/156 2024-05-03T21:09:18.816 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : librados-devel-2:19.0.0-3450.g9a047fe7.el9.x86_6 18/156 2024-05-03T21:09:18.816 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : libradosstriper1-2:19.0.0-3450.g9a047fe7.el9.x86 19/156 2024-05-03T21:09:18.816 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : librgw2-2:19.0.0-3450.g9a047fe7.el9.x86_64 20/156 2024-05-03T21:09:18.816 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-ceph-argparse-2:19.0.0-3450.g9a047fe7.el 21/156 2024-05-03T21:09:18.816 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-ceph-common-2:19.0.0-3450.g9a047fe7.el9. 22/156 2024-05-03T21:09:18.816 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-cephfs-2:19.0.0-3450.g9a047fe7.el9.x86_6 23/156 2024-05-03T21:09:18.816 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-rados-2:19.0.0-3450.g9a047fe7.el9.x86_64 24/156 2024-05-03T21:09:18.816 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-rbd-2:19.0.0-3450.g9a047fe7.el9.x86_64 25/156 2024-05-03T21:09:18.816 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-rgw-2:19.0.0-3450.g9a047fe7.el9.x86_64 26/156 2024-05-03T21:09:18.816 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : rbd-fuse-2:19.0.0-3450.g9a047fe7.el9.x86_64 27/156 2024-05-03T21:09:18.816 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : rbd-mirror-2:19.0.0-3450.g9a047fe7.el9.x86_64 28/156 2024-05-03T21:09:18.828 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : rbd-nbd-2:19.0.0-3450.g9a047fe7.el9.x86_64 29/156 2024-05-03T21:09:18.829 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-3450.g9a047fe7. 30/156 2024-05-03T21:09:18.829 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-3450.g9a047fe7.el9.noa 31/156 2024-05-03T21:09:18.829 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-3450.g9a047fe7.el9.n 32/156 2024-05-03T21:09:18.829 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-3450.g9a0 33/156 2024-05-03T21:09:18.829 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-3450.g9a047fe7.el 34/156 2024-05-03T21:09:18.829 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : ceph-mgr-rook-2:19.0.0-3450.g9a047fe7.el9.noarch 35/156 2024-05-03T21:09:18.829 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-3450.g9a047fe7.e 36/156 2024-05-03T21:09:18.829 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : ceph-volume-2:19.0.0-3450.g9a047fe7.el9.noarch 37/156 2024-05-03T21:09:18.829 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : cephadm-2:19.0.0-3450.g9a047fe7.el9.noarch 38/156 2024-05-03T21:09:18.829 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : cephfs-top-2:19.0.0-3450.g9a047fe7.el9.noarch 39/156 2024-05-03T21:09:18.829 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 40/156 2024-05-03T21:09:18.829 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 41/156 2024-05-03T21:09:18.829 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 42/156 2024-05-03T21:09:18.830 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 43/156 2024-05-03T21:09:18.830 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 44/156 2024-05-03T21:09:18.830 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-ply-3.11-14.el9.noarch 45/156 2024-05-03T21:09:18.830 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 46/156 2024-05-03T21:09:18.830 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 47/156 2024-05-03T21:09:18.830 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 48/156 2024-05-03T21:09:18.830 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : automake-1.16.2-8.el9.noarch 49/156 2024-05-03T21:09:18.830 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : bison-3.7.4-5.el9.x86_64 50/156 2024-05-03T21:09:18.830 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 51/156 2024-05-03T21:09:18.830 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 52/156 2024-05-03T21:09:18.830 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : flex-2.6.4-9.el9.x86_64 53/156 2024-05-03T21:09:18.830 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 54/156 2024-05-03T21:09:18.830 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 55/156 2024-05-03T21:09:18.830 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 56/156 2024-05-03T21:09:18.831 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 57/156 2024-05-03T21:09:18.831 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 58/156 2024-05-03T21:09:18.831 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 59/156 2024-05-03T21:09:18.831 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 60/156 2024-05-03T21:09:18.831 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 61/156 2024-05-03T21:09:18.831 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 62/156 2024-05-03T21:09:18.831 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 63/156 2024-05-03T21:09:18.831 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 64/156 2024-05-03T21:09:18.831 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : lua-5.4.4-4.el9.x86_64 65/156 2024-05-03T21:09:18.831 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 66/156 2024-05-03T21:09:18.831 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 67/156 2024-05-03T21:09:18.831 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : openssl-devel-1:3.2.1-1.el9.x86_64 68/156 2024-05-03T21:09:18.831 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : postgresql-13.14-1.el9.x86_64 69/156 2024-05-03T21:09:18.831 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : postgresql-contrib-13.14-1.el9.x86_64 70/156 2024-05-03T21:09:18.831 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : postgresql-private-libs-13.14-1.el9.x86_64 71/156 2024-05-03T21:09:18.832 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : postgresql-server-13.14-1.el9.x86_64 72/156 2024-05-03T21:09:18.832 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 73/156 2024-05-03T21:09:18.832 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 74/156 2024-05-03T21:09:18.832 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 75/156 2024-05-03T21:09:18.832 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-devel-3.9.18-3.el9.x86_64 76/156 2024-05-03T21:09:18.832 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 77/156 2024-05-03T21:09:18.832 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 78/156 2024-05-03T21:09:18.832 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 79/156 2024-05-03T21:09:18.832 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 80/156 2024-05-03T21:09:18.832 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 81/156 2024-05-03T21:09:18.832 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 82/156 2024-05-03T21:09:18.832 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 83/156 2024-05-03T21:09:18.832 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 84/156 2024-05-03T21:09:18.832 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 85/156 2024-05-03T21:09:18.833 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 86/156 2024-05-03T21:09:18.833 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 87/156 2024-05-03T21:09:18.833 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 88/156 2024-05-03T21:09:18.833 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 89/156 2024-05-03T21:09:18.833 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 90/156 2024-05-03T21:09:18.833 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 91/156 2024-05-03T21:09:18.833 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 92/156 2024-05-03T21:09:18.833 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 93/156 2024-05-03T21:09:18.833 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 94/156 2024-05-03T21:09:18.833 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 95/156 2024-05-03T21:09:18.833 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 96/156 2024-05-03T21:09:18.833 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 97/156 2024-05-03T21:09:18.833 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 98/156 2024-05-03T21:09:18.833 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 99/156 2024-05-03T21:09:18.833 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 100/156 2024-05-03T21:09:18.834 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 101/156 2024-05-03T21:09:18.834 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : zlib-devel-1.2.11-41.el9.x86_64 102/156 2024-05-03T21:09:18.834 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 103/156 2024-05-03T21:09:18.834 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 104/156 2024-05-03T21:09:18.834 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 105/156 2024-05-03T21:09:18.834 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 106/156 2024-05-03T21:09:18.834 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 107/156 2024-05-03T21:09:18.834 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : libarrow-9.0.0-11.el9.x86_64 108/156 2024-05-03T21:09:18.834 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : libarrow-doc-9.0.0-11.el9.noarch 109/156 2024-05-03T21:09:18.834 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 110/156 2024-05-03T21:09:18.834 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 111/156 2024-05-03T21:09:18.834 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 112/156 2024-05-03T21:09:18.834 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : parquet-libs-9.0.0-11.el9.x86_64 113/156 2024-05-03T21:09:18.835 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 114/156 2024-05-03T21:09:18.835 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 115/156 2024-05-03T21:09:18.835 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 116/156 2024-05-03T21:09:18.835 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 117/156 2024-05-03T21:09:18.835 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 118/156 2024-05-03T21:09:18.835 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 119/156 2024-05-03T21:09:18.835 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 120/156 2024-05-03T21:09:18.835 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-colorama-0.4.4-12.el9.noarch 121/156 2024-05-03T21:09:18.835 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-google-auth-1:2.29.0-1.el9.noarch 122/156 2024-05-03T21:09:18.835 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 123/156 2024-05-03T21:09:18.835 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 124/156 2024-05-03T21:09:18.835 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 125/156 2024-05-03T21:09:18.835 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 126/156 2024-05-03T21:09:18.835 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 127/156 2024-05-03T21:09:18.836 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 128/156 2024-05-03T21:09:18.836 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 129/156 2024-05-03T21:09:18.836 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 130/156 2024-05-03T21:09:18.836 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 131/156 2024-05-03T21:09:18.836 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 132/156 2024-05-03T21:09:18.836 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 133/156 2024-05-03T21:09:18.836 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 134/156 2024-05-03T21:09:18.836 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 135/156 2024-05-03T21:09:18.836 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 136/156 2024-05-03T21:09:18.836 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 137/156 2024-05-03T21:09:18.836 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 138/156 2024-05-03T21:09:18.836 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 139/156 2024-05-03T21:09:18.836 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 140/156 2024-05-03T21:09:18.836 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 141/156 2024-05-03T21:09:18.836 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 142/156 2024-05-03T21:09:18.837 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 143/156 2024-05-03T21:09:18.837 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 144/156 2024-05-03T21:09:18.837 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 145/156 2024-05-03T21:09:18.837 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 146/156 2024-05-03T21:09:18.837 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : re2-1:20211101-3.el9.x86_64 147/156 2024-05-03T21:09:18.837 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 148/156 2024-05-03T21:09:18.837 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : librados2-2:19.0.0-3450.g9a047fe7.el9.x86_64 149/156 2024-05-03T21:09:18.837 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 150/156 2024-05-03T21:09:18.837 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : librbd1-2:19.0.0-3450.g9a047fe7.el9.x86_64 151/156 2024-05-03T21:09:18.837 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 152/156 2024-05-03T21:09:18.837 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : openssl-1:3.2.1-1.el9.x86_64 153/156 2024-05-03T21:09:18.837 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 154/156 2024-05-03T21:09:18.837 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : openssl-libs-1:3.2.1-1.el9.x86_64 155/156 2024-05-03T21:09:20.867 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 156/156 2024-05-03T21:09:20.868 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-05-03T21:09:20.868 INFO:teuthology.orchestra.run.smithi138.stdout:Upgraded: 2024-05-03T21:09:20.868 INFO:teuthology.orchestra.run.smithi138.stdout: librados2-2:19.0.0-3450.g9a047fe7.el9.x86_64 2024-05-03T21:09:20.868 INFO:teuthology.orchestra.run.smithi138.stdout: librbd1-2:19.0.0-3450.g9a047fe7.el9.x86_64 2024-05-03T21:09:20.868 INFO:teuthology.orchestra.run.smithi138.stdout: openssl-1:3.2.1-1.el9.x86_64 2024-05-03T21:09:20.868 INFO:teuthology.orchestra.run.smithi138.stdout: openssl-libs-1:3.2.1-1.el9.x86_64 2024-05-03T21:09:20.868 INFO:teuthology.orchestra.run.smithi138.stdout:Installed: 2024-05-03T21:09:20.868 INFO:teuthology.orchestra.run.smithi138.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-05-03T21:09:20.868 INFO:teuthology.orchestra.run.smithi138.stdout: automake-1.16.2-8.el9.noarch 2024-05-03T21:09:20.868 INFO:teuthology.orchestra.run.smithi138.stdout: bison-3.7.4-5.el9.x86_64 2024-05-03T21:09:20.868 INFO:teuthology.orchestra.run.smithi138.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-05-03T21:09:20.869 INFO:teuthology.orchestra.run.smithi138.stdout: ceph-2:19.0.0-3450.g9a047fe7.el9.x86_64 2024-05-03T21:09:20.869 INFO:teuthology.orchestra.run.smithi138.stdout: ceph-base-2:19.0.0-3450.g9a047fe7.el9.x86_64 2024-05-03T21:09:20.869 INFO:teuthology.orchestra.run.smithi138.stdout: ceph-common-2:19.0.0-3450.g9a047fe7.el9.x86_64 2024-05-03T21:09:20.869 INFO:teuthology.orchestra.run.smithi138.stdout: ceph-fuse-2:19.0.0-3450.g9a047fe7.el9.x86_64 2024-05-03T21:09:20.869 INFO:teuthology.orchestra.run.smithi138.stdout: ceph-grafana-dashboards-2:19.0.0-3450.g9a047fe7.el9.noarch 2024-05-03T21:09:20.869 INFO:teuthology.orchestra.run.smithi138.stdout: ceph-immutable-object-cache-2:19.0.0-3450.g9a047fe7.el9.x86_64 2024-05-03T21:09:20.869 INFO:teuthology.orchestra.run.smithi138.stdout: ceph-mds-2:19.0.0-3450.g9a047fe7.el9.x86_64 2024-05-03T21:09:20.869 INFO:teuthology.orchestra.run.smithi138.stdout: ceph-mgr-2:19.0.0-3450.g9a047fe7.el9.x86_64 2024-05-03T21:09:20.869 INFO:teuthology.orchestra.run.smithi138.stdout: ceph-mgr-cephadm-2:19.0.0-3450.g9a047fe7.el9.noarch 2024-05-03T21:09:20.869 INFO:teuthology.orchestra.run.smithi138.stdout: ceph-mgr-dashboard-2:19.0.0-3450.g9a047fe7.el9.noarch 2024-05-03T21:09:20.869 INFO:teuthology.orchestra.run.smithi138.stdout: ceph-mgr-diskprediction-local-2:19.0.0-3450.g9a047fe7.el9.noarch 2024-05-03T21:09:20.869 INFO:teuthology.orchestra.run.smithi138.stdout: ceph-mgr-modules-core-2:19.0.0-3450.g9a047fe7.el9.noarch 2024-05-03T21:09:20.869 INFO:teuthology.orchestra.run.smithi138.stdout: ceph-mgr-rook-2:19.0.0-3450.g9a047fe7.el9.noarch 2024-05-03T21:09:20.869 INFO:teuthology.orchestra.run.smithi138.stdout: ceph-mon-2:19.0.0-3450.g9a047fe7.el9.x86_64 2024-05-03T21:09:20.870 INFO:teuthology.orchestra.run.smithi138.stdout: ceph-osd-2:19.0.0-3450.g9a047fe7.el9.x86_64 2024-05-03T21:09:20.870 INFO:teuthology.orchestra.run.smithi138.stdout: ceph-prometheus-alerts-2:19.0.0-3450.g9a047fe7.el9.noarch 2024-05-03T21:09:20.870 INFO:teuthology.orchestra.run.smithi138.stdout: ceph-radosgw-2:19.0.0-3450.g9a047fe7.el9.x86_64 2024-05-03T21:09:20.870 INFO:teuthology.orchestra.run.smithi138.stdout: ceph-selinux-2:19.0.0-3450.g9a047fe7.el9.x86_64 2024-05-03T21:09:20.870 INFO:teuthology.orchestra.run.smithi138.stdout: ceph-test-2:19.0.0-3450.g9a047fe7.el9.x86_64 2024-05-03T21:09:20.870 INFO:teuthology.orchestra.run.smithi138.stdout: ceph-volume-2:19.0.0-3450.g9a047fe7.el9.noarch 2024-05-03T21:09:20.870 INFO:teuthology.orchestra.run.smithi138.stdout: cephadm-2:19.0.0-3450.g9a047fe7.el9.noarch 2024-05-03T21:09:20.870 INFO:teuthology.orchestra.run.smithi138.stdout: cephfs-mirror-2:19.0.0-3450.g9a047fe7.el9.x86_64 2024-05-03T21:09:20.870 INFO:teuthology.orchestra.run.smithi138.stdout: cephfs-shell-2:19.0.0-3450.g9a047fe7.el9.x86_64 2024-05-03T21:09:20.870 INFO:teuthology.orchestra.run.smithi138.stdout: cephfs-top-2:19.0.0-3450.g9a047fe7.el9.noarch 2024-05-03T21:09:20.870 INFO:teuthology.orchestra.run.smithi138.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-05-03T21:09:20.870 INFO:teuthology.orchestra.run.smithi138.stdout: flex-2.6.4-9.el9.x86_64 2024-05-03T21:09:20.870 INFO:teuthology.orchestra.run.smithi138.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-05-03T21:09:20.870 INFO:teuthology.orchestra.run.smithi138.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-05-03T21:09:20.871 INFO:teuthology.orchestra.run.smithi138.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-05-03T21:09:20.871 INFO:teuthology.orchestra.run.smithi138.stdout: fuse-2.9.9-15.el9.x86_64 2024-05-03T21:09:20.871 INFO:teuthology.orchestra.run.smithi138.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-05-03T21:09:20.871 INFO:teuthology.orchestra.run.smithi138.stdout: grpc-data-1.46.7-10.el9.noarch 2024-05-03T21:09:20.871 INFO:teuthology.orchestra.run.smithi138.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-05-03T21:09:20.871 INFO:teuthology.orchestra.run.smithi138.stdout: libarrow-9.0.0-11.el9.x86_64 2024-05-03T21:09:20.871 INFO:teuthology.orchestra.run.smithi138.stdout: libarrow-doc-9.0.0-11.el9.noarch 2024-05-03T21:09:20.871 INFO:teuthology.orchestra.run.smithi138.stdout: libcephfs-devel-2:19.0.0-3450.g9a047fe7.el9.x86_64 2024-05-03T21:09:20.871 INFO:teuthology.orchestra.run.smithi138.stdout: libcephfs2-2:19.0.0-3450.g9a047fe7.el9.x86_64 2024-05-03T21:09:20.871 INFO:teuthology.orchestra.run.smithi138.stdout: libcephsqlite-2:19.0.0-3450.g9a047fe7.el9.x86_64 2024-05-03T21:09:20.871 INFO:teuthology.orchestra.run.smithi138.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-05-03T21:09:20.871 INFO:teuthology.orchestra.run.smithi138.stdout: liboath-2.6.7-2.el9.x86_64 2024-05-03T21:09:20.871 INFO:teuthology.orchestra.run.smithi138.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-05-03T21:09:20.871 INFO:teuthology.orchestra.run.smithi138.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-05-03T21:09:20.872 INFO:teuthology.orchestra.run.smithi138.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-05-03T21:09:20.872 INFO:teuthology.orchestra.run.smithi138.stdout: librados-devel-2:19.0.0-3450.g9a047fe7.el9.x86_64 2024-05-03T21:09:20.872 INFO:teuthology.orchestra.run.smithi138.stdout: libradosstriper1-2:19.0.0-3450.g9a047fe7.el9.x86_64 2024-05-03T21:09:20.872 INFO:teuthology.orchestra.run.smithi138.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-05-03T21:09:20.872 INFO:teuthology.orchestra.run.smithi138.stdout: librgw2-2:19.0.0-3450.g9a047fe7.el9.x86_64 2024-05-03T21:09:20.872 INFO:teuthology.orchestra.run.smithi138.stdout: libtool-2.4.6-45.el9.x86_64 2024-05-03T21:09:20.872 INFO:teuthology.orchestra.run.smithi138.stdout: libunwind-1.6.2-1.el9.x86_64 2024-05-03T21:09:20.872 INFO:teuthology.orchestra.run.smithi138.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-05-03T21:09:20.872 INFO:teuthology.orchestra.run.smithi138.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-05-03T21:09:20.872 INFO:teuthology.orchestra.run.smithi138.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-05-03T21:09:20.872 INFO:teuthology.orchestra.run.smithi138.stdout: lua-5.4.4-4.el9.x86_64 2024-05-03T21:09:20.872 INFO:teuthology.orchestra.run.smithi138.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-05-03T21:09:20.872 INFO:teuthology.orchestra.run.smithi138.stdout: luarocks-3.9.2-1.el9.noarch 2024-05-03T21:09:20.872 INFO:teuthology.orchestra.run.smithi138.stdout: openblas-0.3.21-2.el9.x86_64 2024-05-03T21:09:20.873 INFO:teuthology.orchestra.run.smithi138.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-05-03T21:09:20.873 INFO:teuthology.orchestra.run.smithi138.stdout: openssl-devel-1:3.2.1-1.el9.x86_64 2024-05-03T21:09:20.873 INFO:teuthology.orchestra.run.smithi138.stdout: parquet-libs-9.0.0-11.el9.x86_64 2024-05-03T21:09:20.873 INFO:teuthology.orchestra.run.smithi138.stdout: postgresql-13.14-1.el9.x86_64 2024-05-03T21:09:20.873 INFO:teuthology.orchestra.run.smithi138.stdout: postgresql-contrib-13.14-1.el9.x86_64 2024-05-03T21:09:20.873 INFO:teuthology.orchestra.run.smithi138.stdout: postgresql-private-libs-13.14-1.el9.x86_64 2024-05-03T21:09:20.873 INFO:teuthology.orchestra.run.smithi138.stdout: postgresql-server-13.14-1.el9.x86_64 2024-05-03T21:09:20.873 INFO:teuthology.orchestra.run.smithi138.stdout: protobuf-3.14.0-13.el9.x86_64 2024-05-03T21:09:20.873 INFO:teuthology.orchestra.run.smithi138.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-05-03T21:09:20.873 INFO:teuthology.orchestra.run.smithi138.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-05-03T21:09:20.873 INFO:teuthology.orchestra.run.smithi138.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-05-03T21:09:20.873 INFO:teuthology.orchestra.run.smithi138.stdout: python3-babel-2.9.1-2.el9.noarch 2024-05-03T21:09:20.873 INFO:teuthology.orchestra.run.smithi138.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-05-03T21:09:20.873 INFO:teuthology.orchestra.run.smithi138.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-05-03T21:09:20.874 INFO:teuthology.orchestra.run.smithi138.stdout: python3-ceph-argparse-2:19.0.0-3450.g9a047fe7.el9.x86_64 2024-05-03T21:09:20.874 INFO:teuthology.orchestra.run.smithi138.stdout: python3-ceph-common-2:19.0.0-3450.g9a047fe7.el9.x86_64 2024-05-03T21:09:20.874 INFO:teuthology.orchestra.run.smithi138.stdout: python3-cephfs-2:19.0.0-3450.g9a047fe7.el9.x86_64 2024-05-03T21:09:20.874 INFO:teuthology.orchestra.run.smithi138.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-05-03T21:09:20.874 INFO:teuthology.orchestra.run.smithi138.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-05-03T21:09:20.874 INFO:teuthology.orchestra.run.smithi138.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-05-03T21:09:20.874 INFO:teuthology.orchestra.run.smithi138.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-05-03T21:09:20.874 INFO:teuthology.orchestra.run.smithi138.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-05-03T21:09:20.874 INFO:teuthology.orchestra.run.smithi138.stdout: python3-colorama-0.4.4-12.el9.noarch 2024-05-03T21:09:20.874 INFO:teuthology.orchestra.run.smithi138.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-05-03T21:09:20.875 INFO:teuthology.orchestra.run.smithi138.stdout: python3-devel-3.9.18-3.el9.x86_64 2024-05-03T21:09:20.875 INFO:teuthology.orchestra.run.smithi138.stdout: python3-google-auth-1:2.29.0-1.el9.noarch 2024-05-03T21:09:20.875 INFO:teuthology.orchestra.run.smithi138.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-05-03T21:09:20.875 INFO:teuthology.orchestra.run.smithi138.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-05-03T21:09:20.875 INFO:teuthology.orchestra.run.smithi138.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-05-03T21:09:20.875 INFO:teuthology.orchestra.run.smithi138.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-05-03T21:09:20.875 INFO:teuthology.orchestra.run.smithi138.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-05-03T21:09:20.875 INFO:teuthology.orchestra.run.smithi138.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-05-03T21:09:20.875 INFO:teuthology.orchestra.run.smithi138.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-05-03T21:09:20.875 INFO:teuthology.orchestra.run.smithi138.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-05-03T21:09:20.875 INFO:teuthology.orchestra.run.smithi138.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-05-03T21:09:20.875 INFO:teuthology.orchestra.run.smithi138.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-05-03T21:09:20.875 INFO:teuthology.orchestra.run.smithi138.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-05-03T21:09:20.876 INFO:teuthology.orchestra.run.smithi138.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-05-03T21:09:20.876 INFO:teuthology.orchestra.run.smithi138.stdout: python3-mako-1.1.4-6.el9.noarch 2024-05-03T21:09:20.876 INFO:teuthology.orchestra.run.smithi138.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-05-03T21:09:20.876 INFO:teuthology.orchestra.run.smithi138.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-05-03T21:09:20.876 INFO:teuthology.orchestra.run.smithi138.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-05-03T21:09:20.876 INFO:teuthology.orchestra.run.smithi138.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-05-03T21:09:20.876 INFO:teuthology.orchestra.run.smithi138.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-05-03T21:09:20.876 INFO:teuthology.orchestra.run.smithi138.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-05-03T21:09:20.876 INFO:teuthology.orchestra.run.smithi138.stdout: python3-packaging-20.9-5.el9.noarch 2024-05-03T21:09:20.876 INFO:teuthology.orchestra.run.smithi138.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-05-03T21:09:20.876 INFO:teuthology.orchestra.run.smithi138.stdout: python3-ply-3.11-14.el9.noarch 2024-05-03T21:09:20.876 INFO:teuthology.orchestra.run.smithi138.stdout: python3-portend-3.1.0-2.el9.noarch 2024-05-03T21:09:20.876 INFO:teuthology.orchestra.run.smithi138.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-05-03T21:09:20.876 INFO:teuthology.orchestra.run.smithi138.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-05-03T21:09:20.877 INFO:teuthology.orchestra.run.smithi138.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-05-03T21:09:20.877 INFO:teuthology.orchestra.run.smithi138.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-05-03T21:09:20.877 INFO:teuthology.orchestra.run.smithi138.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-05-03T21:09:20.877 INFO:teuthology.orchestra.run.smithi138.stdout: python3-pycparser-2.20-6.el9.noarch 2024-05-03T21:09:20.877 INFO:teuthology.orchestra.run.smithi138.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-05-03T21:09:20.877 INFO:teuthology.orchestra.run.smithi138.stdout: python3-pytz-2021.1-5.el9.noarch 2024-05-03T21:09:20.877 INFO:teuthology.orchestra.run.smithi138.stdout: python3-rados-2:19.0.0-3450.g9a047fe7.el9.x86_64 2024-05-03T21:09:20.877 INFO:teuthology.orchestra.run.smithi138.stdout: python3-rbd-2:19.0.0-3450.g9a047fe7.el9.x86_64 2024-05-03T21:09:20.877 INFO:teuthology.orchestra.run.smithi138.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-05-03T21:09:20.877 INFO:teuthology.orchestra.run.smithi138.stdout: python3-requests-2.25.1-8.el9.noarch 2024-05-03T21:09:20.877 INFO:teuthology.orchestra.run.smithi138.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-05-03T21:09:20.878 INFO:teuthology.orchestra.run.smithi138.stdout: python3-rgw-2:19.0.0-3450.g9a047fe7.el9.x86_64 2024-05-03T21:09:20.878 INFO:teuthology.orchestra.run.smithi138.stdout: python3-routes-2.5.1-5.el9.noarch 2024-05-03T21:09:20.878 INFO:teuthology.orchestra.run.smithi138.stdout: python3-rsa-4.9-2.el9.noarch 2024-05-03T21:09:20.878 INFO:teuthology.orchestra.run.smithi138.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-05-03T21:09:20.878 INFO:teuthology.orchestra.run.smithi138.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-05-03T21:09:20.878 INFO:teuthology.orchestra.run.smithi138.stdout: python3-toml-0.10.2-6.el9.noarch 2024-05-03T21:09:20.878 INFO:teuthology.orchestra.run.smithi138.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-05-03T21:09:20.878 INFO:teuthology.orchestra.run.smithi138.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-05-03T21:09:20.878 INFO:teuthology.orchestra.run.smithi138.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-05-03T21:09:20.878 INFO:teuthology.orchestra.run.smithi138.stdout: python3-webob-1.8.7-6.el9.noarch 2024-05-03T21:09:20.878 INFO:teuthology.orchestra.run.smithi138.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-05-03T21:09:20.878 INFO:teuthology.orchestra.run.smithi138.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-05-03T21:09:20.878 INFO:teuthology.orchestra.run.smithi138.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-05-03T21:09:20.878 INFO:teuthology.orchestra.run.smithi138.stdout: qatlib-24.02.0-1.el9.x86_64 2024-05-03T21:09:20.879 INFO:teuthology.orchestra.run.smithi138.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-05-03T21:09:20.879 INFO:teuthology.orchestra.run.smithi138.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-05-03T21:09:20.879 INFO:teuthology.orchestra.run.smithi138.stdout: rbd-fuse-2:19.0.0-3450.g9a047fe7.el9.x86_64 2024-05-03T21:09:20.879 INFO:teuthology.orchestra.run.smithi138.stdout: rbd-mirror-2:19.0.0-3450.g9a047fe7.el9.x86_64 2024-05-03T21:09:20.879 INFO:teuthology.orchestra.run.smithi138.stdout: rbd-nbd-2:19.0.0-3450.g9a047fe7.el9.x86_64 2024-05-03T21:09:20.879 INFO:teuthology.orchestra.run.smithi138.stdout: re2-1:20211101-3.el9.x86_64 2024-05-03T21:09:20.879 INFO:teuthology.orchestra.run.smithi138.stdout: socat-1.7.4.1-5.el9.x86_64 2024-05-03T21:09:20.879 INFO:teuthology.orchestra.run.smithi138.stdout: thrift-0.15.0-2.el9.x86_64 2024-05-03T21:09:20.879 INFO:teuthology.orchestra.run.smithi138.stdout: uuid-1.6.2-55.el9.x86_64 2024-05-03T21:09:20.879 INFO:teuthology.orchestra.run.smithi138.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-05-03T21:09:20.879 INFO:teuthology.orchestra.run.smithi138.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-05-03T21:09:20.880 INFO:teuthology.orchestra.run.smithi138.stdout: zlib-devel-1.2.11-41.el9.x86_64 2024-05-03T21:09:20.880 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-05-03T21:09:20.880 INFO:teuthology.orchestra.run.smithi138.stdout:Complete! 2024-05-03T21:09:21.238 DEBUG:teuthology.parallel:result is None 2024-05-03T21:09:22.156 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : ceph-test-2:19.0.0-3450.g9a047fe7.el9.x86_64 139/156 2024-05-03T21:09:22.197 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : rbd-fuse-2:19.0.0-3450.g9a047fe7.el9.x86_64 140/156 2024-05-03T21:09:22.239 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : rbd-nbd-2:19.0.0-3450.g9a047fe7.el9.x86_64 141/156 2024-05-03T21:09:22.270 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 142/156 2024-05-03T21:09:22.298 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 143/156 2024-05-03T21:09:22.493 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : cephfs-shell-2:19.0.0-3450.g9a047fe7.el9.x86_64 144/156 2024-05-03T21:09:22.544 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : postgresql-contrib-13.14-1.el9.x86_64 145/156 2024-05-03T21:09:22.565 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : libcephfs-devel-2:19.0.0-3450.g9a047fe7.el9.x86_ 146/156 2024-05-03T21:09:23.180 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 147/156 2024-05-03T21:09:23.229 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : postgresql-server-13.14-1.el9.x86_64 147/156 2024-05-03T21:09:23.249 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 147/156 2024-05-03T21:09:24.164 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : cephfs-top-2:19.0.0-3450.g9a047fe7.el9.noarch 148/156 2024-05-03T21:09:24.251 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : openssl-devel-1:3.2.1-1.el9.x86_64 149/156 2024-05-03T21:09:24.301 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 150/156 2024-05-03T21:09:24.385 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : flex-2.6.4-9.el9.x86_64 151/156 2024-05-03T21:09:24.430 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : bison-3.7.4-5.el9.x86_64 152/156 2024-05-03T21:09:24.430 INFO:teuthology.orchestra.run.smithi067.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 153/156 2024-05-03T21:09:24.514 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 153/156 2024-05-03T21:09:24.547 INFO:teuthology.orchestra.run.smithi067.stdout: Cleanup : openssl-1:3.0.7-27.el9.x86_64 154/156 2024-05-03T21:09:24.547 INFO:teuthology.orchestra.run.smithi067.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 155/156 2024-05-03T21:09:24.624 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 155/156 2024-05-03T21:09:24.698 INFO:teuthology.orchestra.run.smithi067.stdout: Cleanup : openssl-libs-1:3.0.7-27.el9.x86_64 156/156 2024-05-03T21:09:30.461 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: openssl-libs-1:3.0.7-27.el9.x86_64 156/156 2024-05-03T21:09:30.461 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : ceph-2:19.0.0-3450.g9a047fe7.el9.x86_64 1/156 2024-05-03T21:09:30.461 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : ceph-base-2:19.0.0-3450.g9a047fe7.el9.x86_64 2/156 2024-05-03T21:09:30.461 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : ceph-common-2:19.0.0-3450.g9a047fe7.el9.x86_64 3/156 2024-05-03T21:09:30.461 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : ceph-fuse-2:19.0.0-3450.g9a047fe7.el9.x86_64 4/156 2024-05-03T21:09:30.461 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-3450.g9a047 5/156 2024-05-03T21:09:30.461 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : ceph-mds-2:19.0.0-3450.g9a047fe7.el9.x86_64 6/156 2024-05-03T21:09:30.461 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : ceph-mgr-2:19.0.0-3450.g9a047fe7.el9.x86_64 7/156 2024-05-03T21:09:30.462 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : ceph-mon-2:19.0.0-3450.g9a047fe7.el9.x86_64 8/156 2024-05-03T21:09:30.462 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : ceph-osd-2:19.0.0-3450.g9a047fe7.el9.x86_64 9/156 2024-05-03T21:09:30.462 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : ceph-radosgw-2:19.0.0-3450.g9a047fe7.el9.x86_64 10/156 2024-05-03T21:09:30.462 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : ceph-selinux-2:19.0.0-3450.g9a047fe7.el9.x86_64 11/156 2024-05-03T21:09:30.462 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : ceph-test-2:19.0.0-3450.g9a047fe7.el9.x86_64 12/156 2024-05-03T21:09:30.462 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : cephfs-mirror-2:19.0.0-3450.g9a047fe7.el9.x86_64 13/156 2024-05-03T21:09:30.462 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : cephfs-shell-2:19.0.0-3450.g9a047fe7.el9.x86_64 14/156 2024-05-03T21:09:30.462 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : libcephfs-devel-2:19.0.0-3450.g9a047fe7.el9.x86_ 15/156 2024-05-03T21:09:30.462 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : libcephfs2-2:19.0.0-3450.g9a047fe7.el9.x86_64 16/156 2024-05-03T21:09:30.462 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : libcephsqlite-2:19.0.0-3450.g9a047fe7.el9.x86_64 17/156 2024-05-03T21:09:30.462 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : librados-devel-2:19.0.0-3450.g9a047fe7.el9.x86_6 18/156 2024-05-03T21:09:30.462 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : libradosstriper1-2:19.0.0-3450.g9a047fe7.el9.x86 19/156 2024-05-03T21:09:30.462 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : librgw2-2:19.0.0-3450.g9a047fe7.el9.x86_64 20/156 2024-05-03T21:09:30.463 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-ceph-argparse-2:19.0.0-3450.g9a047fe7.el 21/156 2024-05-03T21:09:30.463 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-ceph-common-2:19.0.0-3450.g9a047fe7.el9. 22/156 2024-05-03T21:09:30.463 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-cephfs-2:19.0.0-3450.g9a047fe7.el9.x86_6 23/156 2024-05-03T21:09:30.463 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-rados-2:19.0.0-3450.g9a047fe7.el9.x86_64 24/156 2024-05-03T21:09:30.463 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-rbd-2:19.0.0-3450.g9a047fe7.el9.x86_64 25/156 2024-05-03T21:09:30.463 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-rgw-2:19.0.0-3450.g9a047fe7.el9.x86_64 26/156 2024-05-03T21:09:30.463 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : rbd-fuse-2:19.0.0-3450.g9a047fe7.el9.x86_64 27/156 2024-05-03T21:09:30.463 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : rbd-mirror-2:19.0.0-3450.g9a047fe7.el9.x86_64 28/156 2024-05-03T21:09:30.463 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : rbd-nbd-2:19.0.0-3450.g9a047fe7.el9.x86_64 29/156 2024-05-03T21:09:30.463 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-3450.g9a047fe7. 30/156 2024-05-03T21:09:30.475 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-3450.g9a047fe7.el9.noa 31/156 2024-05-03T21:09:30.475 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-3450.g9a047fe7.el9.n 32/156 2024-05-03T21:09:30.475 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-3450.g9a0 33/156 2024-05-03T21:09:30.475 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-3450.g9a047fe7.el 34/156 2024-05-03T21:09:30.475 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : ceph-mgr-rook-2:19.0.0-3450.g9a047fe7.el9.noarch 35/156 2024-05-03T21:09:30.475 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-3450.g9a047fe7.e 36/156 2024-05-03T21:09:30.475 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : ceph-volume-2:19.0.0-3450.g9a047fe7.el9.noarch 37/156 2024-05-03T21:09:30.476 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : cephadm-2:19.0.0-3450.g9a047fe7.el9.noarch 38/156 2024-05-03T21:09:30.476 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : cephfs-top-2:19.0.0-3450.g9a047fe7.el9.noarch 39/156 2024-05-03T21:09:30.476 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 40/156 2024-05-03T21:09:30.476 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 41/156 2024-05-03T21:09:30.476 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 42/156 2024-05-03T21:09:30.476 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 43/156 2024-05-03T21:09:30.476 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 44/156 2024-05-03T21:09:30.476 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-ply-3.11-14.el9.noarch 45/156 2024-05-03T21:09:30.476 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 46/156 2024-05-03T21:09:30.476 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 47/156 2024-05-03T21:09:30.476 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 48/156 2024-05-03T21:09:30.476 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : automake-1.16.2-8.el9.noarch 49/156 2024-05-03T21:09:30.476 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : bison-3.7.4-5.el9.x86_64 50/156 2024-05-03T21:09:30.476 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 51/156 2024-05-03T21:09:30.477 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 52/156 2024-05-03T21:09:30.477 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : flex-2.6.4-9.el9.x86_64 53/156 2024-05-03T21:09:30.477 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 54/156 2024-05-03T21:09:30.477 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 55/156 2024-05-03T21:09:30.477 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 56/156 2024-05-03T21:09:30.477 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 57/156 2024-05-03T21:09:30.477 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 58/156 2024-05-03T21:09:30.477 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 59/156 2024-05-03T21:09:30.477 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 60/156 2024-05-03T21:09:30.477 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 61/156 2024-05-03T21:09:30.477 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 62/156 2024-05-03T21:09:30.477 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 63/156 2024-05-03T21:09:30.477 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 64/156 2024-05-03T21:09:30.478 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : lua-5.4.4-4.el9.x86_64 65/156 2024-05-03T21:09:30.478 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 66/156 2024-05-03T21:09:30.478 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 67/156 2024-05-03T21:09:30.478 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : openssl-devel-1:3.2.1-1.el9.x86_64 68/156 2024-05-03T21:09:30.478 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : postgresql-13.14-1.el9.x86_64 69/156 2024-05-03T21:09:30.478 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : postgresql-contrib-13.14-1.el9.x86_64 70/156 2024-05-03T21:09:30.478 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : postgresql-private-libs-13.14-1.el9.x86_64 71/156 2024-05-03T21:09:30.478 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : postgresql-server-13.14-1.el9.x86_64 72/156 2024-05-03T21:09:30.478 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 73/156 2024-05-03T21:09:30.478 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 74/156 2024-05-03T21:09:30.478 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 75/156 2024-05-03T21:09:30.478 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-devel-3.9.18-3.el9.x86_64 76/156 2024-05-03T21:09:30.479 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 77/156 2024-05-03T21:09:30.479 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 78/156 2024-05-03T21:09:30.479 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 79/156 2024-05-03T21:09:30.479 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 80/156 2024-05-03T21:09:30.479 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 81/156 2024-05-03T21:09:30.479 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 82/156 2024-05-03T21:09:30.479 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 83/156 2024-05-03T21:09:30.479 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 84/156 2024-05-03T21:09:30.479 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 85/156 2024-05-03T21:09:30.479 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 86/156 2024-05-03T21:09:30.479 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 87/156 2024-05-03T21:09:30.479 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 88/156 2024-05-03T21:09:30.479 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 89/156 2024-05-03T21:09:30.479 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 90/156 2024-05-03T21:09:30.480 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 91/156 2024-05-03T21:09:30.480 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 92/156 2024-05-03T21:09:30.481 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 93/156 2024-05-03T21:09:30.481 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 94/156 2024-05-03T21:09:30.481 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 95/156 2024-05-03T21:09:30.481 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 96/156 2024-05-03T21:09:30.481 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 97/156 2024-05-03T21:09:30.481 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 98/156 2024-05-03T21:09:30.481 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 99/156 2024-05-03T21:09:30.481 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 100/156 2024-05-03T21:09:30.482 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 101/156 2024-05-03T21:09:30.482 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : zlib-devel-1.2.11-41.el9.x86_64 102/156 2024-05-03T21:09:30.482 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 103/156 2024-05-03T21:09:30.482 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 104/156 2024-05-03T21:09:30.482 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 105/156 2024-05-03T21:09:30.482 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 106/156 2024-05-03T21:09:30.482 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 107/156 2024-05-03T21:09:30.482 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : libarrow-9.0.0-11.el9.x86_64 108/156 2024-05-03T21:09:30.482 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : libarrow-doc-9.0.0-11.el9.noarch 109/156 2024-05-03T21:09:30.482 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 110/156 2024-05-03T21:09:30.482 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 111/156 2024-05-03T21:09:30.482 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 112/156 2024-05-03T21:09:30.482 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : parquet-libs-9.0.0-11.el9.x86_64 113/156 2024-05-03T21:09:30.483 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 114/156 2024-05-03T21:09:30.483 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 115/156 2024-05-03T21:09:30.483 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 116/156 2024-05-03T21:09:30.483 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 117/156 2024-05-03T21:09:30.483 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 118/156 2024-05-03T21:09:30.483 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 119/156 2024-05-03T21:09:30.483 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 120/156 2024-05-03T21:09:30.483 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-colorama-0.4.4-12.el9.noarch 121/156 2024-05-03T21:09:30.483 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-google-auth-1:2.29.0-1.el9.noarch 122/156 2024-05-03T21:09:30.483 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 123/156 2024-05-03T21:09:30.483 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 124/156 2024-05-03T21:09:30.483 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 125/156 2024-05-03T21:09:30.483 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 126/156 2024-05-03T21:09:30.483 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 127/156 2024-05-03T21:09:30.484 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 128/156 2024-05-03T21:09:30.484 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 129/156 2024-05-03T21:09:30.484 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 130/156 2024-05-03T21:09:30.484 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 131/156 2024-05-03T21:09:30.484 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 132/156 2024-05-03T21:09:30.484 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 133/156 2024-05-03T21:09:30.484 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 134/156 2024-05-03T21:09:30.484 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 135/156 2024-05-03T21:09:30.484 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 136/156 2024-05-03T21:09:30.484 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 137/156 2024-05-03T21:09:30.484 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 138/156 2024-05-03T21:09:30.484 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 139/156 2024-05-03T21:09:30.484 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 140/156 2024-05-03T21:09:30.484 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 141/156 2024-05-03T21:09:30.484 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 142/156 2024-05-03T21:09:30.485 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 143/156 2024-05-03T21:09:30.485 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 144/156 2024-05-03T21:09:30.485 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 145/156 2024-05-03T21:09:30.485 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 146/156 2024-05-03T21:09:30.485 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : re2-1:20211101-3.el9.x86_64 147/156 2024-05-03T21:09:30.485 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 148/156 2024-05-03T21:09:30.485 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : librados2-2:19.0.0-3450.g9a047fe7.el9.x86_64 149/156 2024-05-03T21:09:30.485 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 150/156 2024-05-03T21:09:30.485 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : librbd1-2:19.0.0-3450.g9a047fe7.el9.x86_64 151/156 2024-05-03T21:09:30.485 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 152/156 2024-05-03T21:09:30.485 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : openssl-1:3.2.1-1.el9.x86_64 153/156 2024-05-03T21:09:30.485 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 154/156 2024-05-03T21:09:30.485 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : openssl-libs-1:3.2.1-1.el9.x86_64 155/156 2024-05-03T21:09:32.525 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 156/156 2024-05-03T21:09:32.525 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-05-03T21:09:32.525 INFO:teuthology.orchestra.run.smithi067.stdout:Upgraded: 2024-05-03T21:09:32.525 INFO:teuthology.orchestra.run.smithi067.stdout: librados2-2:19.0.0-3450.g9a047fe7.el9.x86_64 2024-05-03T21:09:32.526 INFO:teuthology.orchestra.run.smithi067.stdout: librbd1-2:19.0.0-3450.g9a047fe7.el9.x86_64 2024-05-03T21:09:32.526 INFO:teuthology.orchestra.run.smithi067.stdout: openssl-1:3.2.1-1.el9.x86_64 2024-05-03T21:09:32.526 INFO:teuthology.orchestra.run.smithi067.stdout: openssl-libs-1:3.2.1-1.el9.x86_64 2024-05-03T21:09:32.526 INFO:teuthology.orchestra.run.smithi067.stdout:Installed: 2024-05-03T21:09:32.526 INFO:teuthology.orchestra.run.smithi067.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-05-03T21:09:32.526 INFO:teuthology.orchestra.run.smithi067.stdout: automake-1.16.2-8.el9.noarch 2024-05-03T21:09:32.526 INFO:teuthology.orchestra.run.smithi067.stdout: bison-3.7.4-5.el9.x86_64 2024-05-03T21:09:32.526 INFO:teuthology.orchestra.run.smithi067.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-05-03T21:09:32.526 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-2:19.0.0-3450.g9a047fe7.el9.x86_64 2024-05-03T21:09:32.526 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-base-2:19.0.0-3450.g9a047fe7.el9.x86_64 2024-05-03T21:09:32.526 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-common-2:19.0.0-3450.g9a047fe7.el9.x86_64 2024-05-03T21:09:32.526 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-fuse-2:19.0.0-3450.g9a047fe7.el9.x86_64 2024-05-03T21:09:32.527 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-grafana-dashboards-2:19.0.0-3450.g9a047fe7.el9.noarch 2024-05-03T21:09:32.527 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-immutable-object-cache-2:19.0.0-3450.g9a047fe7.el9.x86_64 2024-05-03T21:09:32.527 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-mds-2:19.0.0-3450.g9a047fe7.el9.x86_64 2024-05-03T21:09:32.527 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-mgr-2:19.0.0-3450.g9a047fe7.el9.x86_64 2024-05-03T21:09:32.527 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-mgr-cephadm-2:19.0.0-3450.g9a047fe7.el9.noarch 2024-05-03T21:09:32.527 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-mgr-dashboard-2:19.0.0-3450.g9a047fe7.el9.noarch 2024-05-03T21:09:32.527 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-mgr-diskprediction-local-2:19.0.0-3450.g9a047fe7.el9.noarch 2024-05-03T21:09:32.527 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-mgr-modules-core-2:19.0.0-3450.g9a047fe7.el9.noarch 2024-05-03T21:09:32.527 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-mgr-rook-2:19.0.0-3450.g9a047fe7.el9.noarch 2024-05-03T21:09:32.527 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-mon-2:19.0.0-3450.g9a047fe7.el9.x86_64 2024-05-03T21:09:32.527 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-osd-2:19.0.0-3450.g9a047fe7.el9.x86_64 2024-05-03T21:09:32.527 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-prometheus-alerts-2:19.0.0-3450.g9a047fe7.el9.noarch 2024-05-03T21:09:32.527 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-radosgw-2:19.0.0-3450.g9a047fe7.el9.x86_64 2024-05-03T21:09:32.528 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-selinux-2:19.0.0-3450.g9a047fe7.el9.x86_64 2024-05-03T21:09:32.528 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-test-2:19.0.0-3450.g9a047fe7.el9.x86_64 2024-05-03T21:09:32.528 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-volume-2:19.0.0-3450.g9a047fe7.el9.noarch 2024-05-03T21:09:32.528 INFO:teuthology.orchestra.run.smithi067.stdout: cephadm-2:19.0.0-3450.g9a047fe7.el9.noarch 2024-05-03T21:09:32.528 INFO:teuthology.orchestra.run.smithi067.stdout: cephfs-mirror-2:19.0.0-3450.g9a047fe7.el9.x86_64 2024-05-03T21:09:32.528 INFO:teuthology.orchestra.run.smithi067.stdout: cephfs-shell-2:19.0.0-3450.g9a047fe7.el9.x86_64 2024-05-03T21:09:32.528 INFO:teuthology.orchestra.run.smithi067.stdout: cephfs-top-2:19.0.0-3450.g9a047fe7.el9.noarch 2024-05-03T21:09:32.528 INFO:teuthology.orchestra.run.smithi067.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-05-03T21:09:32.528 INFO:teuthology.orchestra.run.smithi067.stdout: flex-2.6.4-9.el9.x86_64 2024-05-03T21:09:32.528 INFO:teuthology.orchestra.run.smithi067.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-05-03T21:09:32.528 INFO:teuthology.orchestra.run.smithi067.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-05-03T21:09:32.528 INFO:teuthology.orchestra.run.smithi067.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-05-03T21:09:32.528 INFO:teuthology.orchestra.run.smithi067.stdout: fuse-2.9.9-15.el9.x86_64 2024-05-03T21:09:32.529 INFO:teuthology.orchestra.run.smithi067.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-05-03T21:09:32.529 INFO:teuthology.orchestra.run.smithi067.stdout: grpc-data-1.46.7-10.el9.noarch 2024-05-03T21:09:32.529 INFO:teuthology.orchestra.run.smithi067.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-05-03T21:09:32.529 INFO:teuthology.orchestra.run.smithi067.stdout: libarrow-9.0.0-11.el9.x86_64 2024-05-03T21:09:32.529 INFO:teuthology.orchestra.run.smithi067.stdout: libarrow-doc-9.0.0-11.el9.noarch 2024-05-03T21:09:32.529 INFO:teuthology.orchestra.run.smithi067.stdout: libcephfs-devel-2:19.0.0-3450.g9a047fe7.el9.x86_64 2024-05-03T21:09:32.529 INFO:teuthology.orchestra.run.smithi067.stdout: libcephfs2-2:19.0.0-3450.g9a047fe7.el9.x86_64 2024-05-03T21:09:32.529 INFO:teuthology.orchestra.run.smithi067.stdout: libcephsqlite-2:19.0.0-3450.g9a047fe7.el9.x86_64 2024-05-03T21:09:32.529 INFO:teuthology.orchestra.run.smithi067.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-05-03T21:09:32.530 INFO:teuthology.orchestra.run.smithi067.stdout: liboath-2.6.7-2.el9.x86_64 2024-05-03T21:09:32.530 INFO:teuthology.orchestra.run.smithi067.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-05-03T21:09:32.530 INFO:teuthology.orchestra.run.smithi067.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-05-03T21:09:32.530 INFO:teuthology.orchestra.run.smithi067.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-05-03T21:09:32.530 INFO:teuthology.orchestra.run.smithi067.stdout: librados-devel-2:19.0.0-3450.g9a047fe7.el9.x86_64 2024-05-03T21:09:32.530 INFO:teuthology.orchestra.run.smithi067.stdout: libradosstriper1-2:19.0.0-3450.g9a047fe7.el9.x86_64 2024-05-03T21:09:32.530 INFO:teuthology.orchestra.run.smithi067.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-05-03T21:09:32.530 INFO:teuthology.orchestra.run.smithi067.stdout: librgw2-2:19.0.0-3450.g9a047fe7.el9.x86_64 2024-05-03T21:09:32.530 INFO:teuthology.orchestra.run.smithi067.stdout: libtool-2.4.6-45.el9.x86_64 2024-05-03T21:09:32.531 INFO:teuthology.orchestra.run.smithi067.stdout: libunwind-1.6.2-1.el9.x86_64 2024-05-03T21:09:32.531 INFO:teuthology.orchestra.run.smithi067.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-05-03T21:09:32.531 INFO:teuthology.orchestra.run.smithi067.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-05-03T21:09:32.531 INFO:teuthology.orchestra.run.smithi067.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-05-03T21:09:32.531 INFO:teuthology.orchestra.run.smithi067.stdout: lua-5.4.4-4.el9.x86_64 2024-05-03T21:09:32.531 INFO:teuthology.orchestra.run.smithi067.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-05-03T21:09:32.531 INFO:teuthology.orchestra.run.smithi067.stdout: luarocks-3.9.2-1.el9.noarch 2024-05-03T21:09:32.531 INFO:teuthology.orchestra.run.smithi067.stdout: openblas-0.3.21-2.el9.x86_64 2024-05-03T21:09:32.531 INFO:teuthology.orchestra.run.smithi067.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-05-03T21:09:32.531 INFO:teuthology.orchestra.run.smithi067.stdout: openssl-devel-1:3.2.1-1.el9.x86_64 2024-05-03T21:09:32.531 INFO:teuthology.orchestra.run.smithi067.stdout: parquet-libs-9.0.0-11.el9.x86_64 2024-05-03T21:09:32.531 INFO:teuthology.orchestra.run.smithi067.stdout: postgresql-13.14-1.el9.x86_64 2024-05-03T21:09:32.531 INFO:teuthology.orchestra.run.smithi067.stdout: postgresql-contrib-13.14-1.el9.x86_64 2024-05-03T21:09:32.531 INFO:teuthology.orchestra.run.smithi067.stdout: postgresql-private-libs-13.14-1.el9.x86_64 2024-05-03T21:09:32.532 INFO:teuthology.orchestra.run.smithi067.stdout: postgresql-server-13.14-1.el9.x86_64 2024-05-03T21:09:32.532 INFO:teuthology.orchestra.run.smithi067.stdout: protobuf-3.14.0-13.el9.x86_64 2024-05-03T21:09:32.532 INFO:teuthology.orchestra.run.smithi067.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-05-03T21:09:32.532 INFO:teuthology.orchestra.run.smithi067.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-05-03T21:09:32.532 INFO:teuthology.orchestra.run.smithi067.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-05-03T21:09:32.532 INFO:teuthology.orchestra.run.smithi067.stdout: python3-babel-2.9.1-2.el9.noarch 2024-05-03T21:09:32.532 INFO:teuthology.orchestra.run.smithi067.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-05-03T21:09:32.532 INFO:teuthology.orchestra.run.smithi067.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-05-03T21:09:32.532 INFO:teuthology.orchestra.run.smithi067.stdout: python3-ceph-argparse-2:19.0.0-3450.g9a047fe7.el9.x86_64 2024-05-03T21:09:32.532 INFO:teuthology.orchestra.run.smithi067.stdout: python3-ceph-common-2:19.0.0-3450.g9a047fe7.el9.x86_64 2024-05-03T21:09:32.532 INFO:teuthology.orchestra.run.smithi067.stdout: python3-cephfs-2:19.0.0-3450.g9a047fe7.el9.x86_64 2024-05-03T21:09:32.532 INFO:teuthology.orchestra.run.smithi067.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-05-03T21:09:32.532 INFO:teuthology.orchestra.run.smithi067.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-05-03T21:09:32.532 INFO:teuthology.orchestra.run.smithi067.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-05-03T21:09:32.533 INFO:teuthology.orchestra.run.smithi067.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-05-03T21:09:32.533 INFO:teuthology.orchestra.run.smithi067.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-05-03T21:09:32.533 INFO:teuthology.orchestra.run.smithi067.stdout: python3-colorama-0.4.4-12.el9.noarch 2024-05-03T21:09:32.533 INFO:teuthology.orchestra.run.smithi067.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-05-03T21:09:32.533 INFO:teuthology.orchestra.run.smithi067.stdout: python3-devel-3.9.18-3.el9.x86_64 2024-05-03T21:09:32.533 INFO:teuthology.orchestra.run.smithi067.stdout: python3-google-auth-1:2.29.0-1.el9.noarch 2024-05-03T21:09:32.533 INFO:teuthology.orchestra.run.smithi067.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-05-03T21:09:32.533 INFO:teuthology.orchestra.run.smithi067.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-05-03T21:09:32.533 INFO:teuthology.orchestra.run.smithi067.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-05-03T21:09:32.533 INFO:teuthology.orchestra.run.smithi067.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-05-03T21:09:32.533 INFO:teuthology.orchestra.run.smithi067.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-05-03T21:09:32.533 INFO:teuthology.orchestra.run.smithi067.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-05-03T21:09:32.533 INFO:teuthology.orchestra.run.smithi067.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-05-03T21:09:32.533 INFO:teuthology.orchestra.run.smithi067.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-05-03T21:09:32.533 INFO:teuthology.orchestra.run.smithi067.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-05-03T21:09:32.534 INFO:teuthology.orchestra.run.smithi067.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-05-03T21:09:32.534 INFO:teuthology.orchestra.run.smithi067.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-05-03T21:09:32.534 INFO:teuthology.orchestra.run.smithi067.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-05-03T21:09:32.534 INFO:teuthology.orchestra.run.smithi067.stdout: python3-mako-1.1.4-6.el9.noarch 2024-05-03T21:09:32.534 INFO:teuthology.orchestra.run.smithi067.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-05-03T21:09:32.534 INFO:teuthology.orchestra.run.smithi067.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-05-03T21:09:32.534 INFO:teuthology.orchestra.run.smithi067.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-05-03T21:09:32.534 INFO:teuthology.orchestra.run.smithi067.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-05-03T21:09:32.534 INFO:teuthology.orchestra.run.smithi067.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-05-03T21:09:32.534 INFO:teuthology.orchestra.run.smithi067.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-05-03T21:09:32.534 INFO:teuthology.orchestra.run.smithi067.stdout: python3-packaging-20.9-5.el9.noarch 2024-05-03T21:09:32.534 INFO:teuthology.orchestra.run.smithi067.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-05-03T21:09:32.534 INFO:teuthology.orchestra.run.smithi067.stdout: python3-ply-3.11-14.el9.noarch 2024-05-03T21:09:32.535 INFO:teuthology.orchestra.run.smithi067.stdout: python3-portend-3.1.0-2.el9.noarch 2024-05-03T21:09:32.535 INFO:teuthology.orchestra.run.smithi067.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-05-03T21:09:32.535 INFO:teuthology.orchestra.run.smithi067.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-05-03T21:09:32.535 INFO:teuthology.orchestra.run.smithi067.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-05-03T21:09:32.535 INFO:teuthology.orchestra.run.smithi067.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-05-03T21:09:32.535 INFO:teuthology.orchestra.run.smithi067.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-05-03T21:09:32.535 INFO:teuthology.orchestra.run.smithi067.stdout: python3-pycparser-2.20-6.el9.noarch 2024-05-03T21:09:32.535 INFO:teuthology.orchestra.run.smithi067.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-05-03T21:09:32.535 INFO:teuthology.orchestra.run.smithi067.stdout: python3-pytz-2021.1-5.el9.noarch 2024-05-03T21:09:32.535 INFO:teuthology.orchestra.run.smithi067.stdout: python3-rados-2:19.0.0-3450.g9a047fe7.el9.x86_64 2024-05-03T21:09:32.535 INFO:teuthology.orchestra.run.smithi067.stdout: python3-rbd-2:19.0.0-3450.g9a047fe7.el9.x86_64 2024-05-03T21:09:32.535 INFO:teuthology.orchestra.run.smithi067.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-05-03T21:09:32.535 INFO:teuthology.orchestra.run.smithi067.stdout: python3-requests-2.25.1-8.el9.noarch 2024-05-03T21:09:32.535 INFO:teuthology.orchestra.run.smithi067.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-05-03T21:09:32.535 INFO:teuthology.orchestra.run.smithi067.stdout: python3-rgw-2:19.0.0-3450.g9a047fe7.el9.x86_64 2024-05-03T21:09:32.536 INFO:teuthology.orchestra.run.smithi067.stdout: python3-routes-2.5.1-5.el9.noarch 2024-05-03T21:09:32.536 INFO:teuthology.orchestra.run.smithi067.stdout: python3-rsa-4.9-2.el9.noarch 2024-05-03T21:09:32.536 INFO:teuthology.orchestra.run.smithi067.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-05-03T21:09:32.536 INFO:teuthology.orchestra.run.smithi067.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-05-03T21:09:32.536 INFO:teuthology.orchestra.run.smithi067.stdout: python3-toml-0.10.2-6.el9.noarch 2024-05-03T21:09:32.536 INFO:teuthology.orchestra.run.smithi067.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-05-03T21:09:32.536 INFO:teuthology.orchestra.run.smithi067.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-05-03T21:09:32.536 INFO:teuthology.orchestra.run.smithi067.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-05-03T21:09:32.536 INFO:teuthology.orchestra.run.smithi067.stdout: python3-webob-1.8.7-6.el9.noarch 2024-05-03T21:09:32.536 INFO:teuthology.orchestra.run.smithi067.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-05-03T21:09:32.536 INFO:teuthology.orchestra.run.smithi067.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-05-03T21:09:32.536 INFO:teuthology.orchestra.run.smithi067.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-05-03T21:09:32.537 INFO:teuthology.orchestra.run.smithi067.stdout: qatlib-24.02.0-1.el9.x86_64 2024-05-03T21:09:32.537 INFO:teuthology.orchestra.run.smithi067.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-05-03T21:09:32.537 INFO:teuthology.orchestra.run.smithi067.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-05-03T21:09:32.537 INFO:teuthology.orchestra.run.smithi067.stdout: rbd-fuse-2:19.0.0-3450.g9a047fe7.el9.x86_64 2024-05-03T21:09:32.537 INFO:teuthology.orchestra.run.smithi067.stdout: rbd-mirror-2:19.0.0-3450.g9a047fe7.el9.x86_64 2024-05-03T21:09:32.537 INFO:teuthology.orchestra.run.smithi067.stdout: rbd-nbd-2:19.0.0-3450.g9a047fe7.el9.x86_64 2024-05-03T21:09:32.537 INFO:teuthology.orchestra.run.smithi067.stdout: re2-1:20211101-3.el9.x86_64 2024-05-03T21:09:32.537 INFO:teuthology.orchestra.run.smithi067.stdout: socat-1.7.4.1-5.el9.x86_64 2024-05-03T21:09:32.537 INFO:teuthology.orchestra.run.smithi067.stdout: thrift-0.15.0-2.el9.x86_64 2024-05-03T21:09:32.537 INFO:teuthology.orchestra.run.smithi067.stdout: uuid-1.6.2-55.el9.x86_64 2024-05-03T21:09:32.537 INFO:teuthology.orchestra.run.smithi067.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-05-03T21:09:32.537 INFO:teuthology.orchestra.run.smithi067.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-05-03T21:09:32.537 INFO:teuthology.orchestra.run.smithi067.stdout: zlib-devel-1.2.11-41.el9.x86_64 2024-05-03T21:09:32.537 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-05-03T21:09:32.537 INFO:teuthology.orchestra.run.smithi067.stdout:Complete! 2024-05-03T21:09:32.878 DEBUG:teuthology.parallel:result is None 2024-05-03T21:09:32.878 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=9a047fe7599a549042d06a3fb5b6813a203eabea 2024-05-03T21:09:33.094 DEBUG:teuthology.orchestra.run.smithi067:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-05-03T21:09:33.116 INFO:teuthology.orchestra.run.smithi067.stdout:19.0.0-3450.g9a047fe7.el9 2024-05-03T21:09:33.116 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-3450.g9a047fe7.el9 2024-05-03T21:09:33.116 INFO:teuthology.task.install:The correct ceph version 19.0.0-3450.g9a047fe7 is installed. 2024-05-03T21:09:33.118 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=9a047fe7599a549042d06a3fb5b6813a203eabea 2024-05-03T21:09:33.332 DEBUG:teuthology.orchestra.run.smithi082:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-05-03T21:09:33.358 INFO:teuthology.orchestra.run.smithi082.stdout:19.0.0-3450.g9a047fe7.el9 2024-05-03T21:09:33.359 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-3450.g9a047fe7.el9 2024-05-03T21:09:33.359 INFO:teuthology.task.install:The correct ceph version 19.0.0-3450.g9a047fe7 is installed. 2024-05-03T21:09:33.361 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=9a047fe7599a549042d06a3fb5b6813a203eabea 2024-05-03T21:09:33.574 DEBUG:teuthology.orchestra.run.smithi138:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-05-03T21:09:33.601 INFO:teuthology.orchestra.run.smithi138.stdout:19.0.0-3450.g9a047fe7.el9 2024-05-03T21:09:33.601 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-3450.g9a047fe7.el9 2024-05-03T21:09:33.601 INFO:teuthology.task.install:The correct ceph version 19.0.0-3450.g9a047fe7 is installed. 2024-05-03T21:09:33.603 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-05-03T21:09:33.603 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-05-03T21:09:33.604 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-05-03T21:09:33.630 DEBUG:teuthology.orchestra.run.smithi082:> set -ex 2024-05-03T21:09:33.630 DEBUG:teuthology.orchestra.run.smithi082:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-05-03T21:09:33.662 DEBUG:teuthology.orchestra.run.smithi138:> set -ex 2024-05-03T21:09:33.662 DEBUG:teuthology.orchestra.run.smithi138:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-05-03T21:09:33.692 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-05-03T21:09:33.693 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-05-03T21:09:33.693 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd of=/usr/bin/daemon-helper 2024-05-03T21:09:33.717 DEBUG:teuthology.orchestra.run.smithi067:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-05-03T21:09:33.783 DEBUG:teuthology.orchestra.run.smithi082:> set -ex 2024-05-03T21:09:33.783 DEBUG:teuthology.orchestra.run.smithi082:> sudo dd of=/usr/bin/daemon-helper 2024-05-03T21:09:33.811 DEBUG:teuthology.orchestra.run.smithi082:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-05-03T21:09:33.878 DEBUG:teuthology.orchestra.run.smithi138:> set -ex 2024-05-03T21:09:33.879 DEBUG:teuthology.orchestra.run.smithi138:> sudo dd of=/usr/bin/daemon-helper 2024-05-03T21:09:33.908 DEBUG:teuthology.orchestra.run.smithi138:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-05-03T21:09:33.976 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-05-03T21:09:33.976 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-05-03T21:09:33.976 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd of=/usr/bin/adjust-ulimits 2024-05-03T21:09:34.001 DEBUG:teuthology.orchestra.run.smithi067:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-05-03T21:09:34.068 DEBUG:teuthology.orchestra.run.smithi082:> set -ex 2024-05-03T21:09:34.068 DEBUG:teuthology.orchestra.run.smithi082:> sudo dd of=/usr/bin/adjust-ulimits 2024-05-03T21:09:34.095 DEBUG:teuthology.orchestra.run.smithi082:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-05-03T21:09:34.163 DEBUG:teuthology.orchestra.run.smithi138:> set -ex 2024-05-03T21:09:34.163 DEBUG:teuthology.orchestra.run.smithi138:> sudo dd of=/usr/bin/adjust-ulimits 2024-05-03T21:09:34.191 DEBUG:teuthology.orchestra.run.smithi138:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-05-03T21:09:34.260 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-05-03T21:09:34.260 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-05-03T21:09:34.260 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd of=/usr/bin/stdin-killer 2024-05-03T21:09:34.285 DEBUG:teuthology.orchestra.run.smithi067:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-05-03T21:09:34.350 DEBUG:teuthology.orchestra.run.smithi082:> set -ex 2024-05-03T21:09:34.351 DEBUG:teuthology.orchestra.run.smithi082:> sudo dd of=/usr/bin/stdin-killer 2024-05-03T21:09:34.378 DEBUG:teuthology.orchestra.run.smithi082:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-05-03T21:09:34.444 DEBUG:teuthology.orchestra.run.smithi138:> set -ex 2024-05-03T21:09:34.444 DEBUG:teuthology.orchestra.run.smithi138:> sudo dd of=/usr/bin/stdin-killer 2024-05-03T21:09:34.468 DEBUG:teuthology.orchestra.run.smithi138:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-05-03T21:09:34.532 INFO:teuthology.run_tasks:Running task cephadm... 2024-05-03T21:09:34.641 INFO:tasks.cephadm:Config: {'roleless': False, 'cephfs': {'max_mds': 3, 'session_timeout': 300, 'standby_replay': True}, 'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'fuse default permissions': False, 'fuse set user groups': True, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'ms die on skipped message': False}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds bal fragment size max': 10000, 'mds bal merge size': 5, 'mds bal split bits': 3, 'mds bal split size': 100, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'mds_bal_replicate_threshold': 1, 'mds_export_ephemeral_random_max': 0.1, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'cephsqlite lock renewal timeout': 900000, 'debug client': 20, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon down mkfs grace': 300, 'mon op complaint time': 120}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180, 'osd shutdown pgref assert': True, 'osd_max_omap_entries_per_request': 10}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'MDS_UP_LESS_THAN_MAX', 'filesystem is online with fewer MDS than max_mds', 'POOL_APP_NOT_ENABLED', 'do not have an application enabled', 'overall HEALTH_', 'Replacing daemon', 'deprecated feature inline_data', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'PG_AVAILABILITY', 'PG_DEGRADED', 'Reduced data availability'], 'sha1': '9a047fe7599a549042d06a3fb5b6813a203eabea', 'subvols': {'create': 2}} 2024-05-03T21:09:34.641 INFO:tasks.cephadm:Cluster image is quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9a047fe7599a549042d06a3fb5b6813a203eabea 2024-05-03T21:09:34.642 INFO:tasks.cephadm:Cluster fsid is 713bc57a-0991-11ef-bc97-c7b262605968 2024-05-03T21:09:34.642 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2024-05-03T21:09:34.643 INFO:tasks.cephadm:Monitor IPs: {'mon.a': '172.21.15.67', 'mon.b': '172.21.15.82', 'mon.c': '172.21.15.138'} 2024-05-03T21:09:34.643 INFO:tasks.cephadm:First mon is mon.a on smithi067 2024-05-03T21:09:34.643 INFO:tasks.cephadm:First mgr is x 2024-05-03T21:09:34.643 INFO:tasks.cephadm:Normalizing hostnames... 2024-05-03T21:09:34.643 DEBUG:teuthology.orchestra.run.smithi067:> sudo hostname $(hostname -s) 2024-05-03T21:09:34.668 DEBUG:teuthology.orchestra.run.smithi082:> sudo hostname $(hostname -s) 2024-05-03T21:09:34.696 DEBUG:teuthology.orchestra.run.smithi138:> sudo hostname $(hostname -s) 2024-05-03T21:09:34.720 INFO:tasks.cephadm:Downloading "compiled" cephadm from cachra 2024-05-03T21:09:34.720 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=9a047fe7599a549042d06a3fb5b6813a203eabea 2024-05-03T21:09:34.860 INFO:tasks.cephadm:builder_project result: [{'status': 'ready', 'sha1': '9a047fe7599a549042d06a3fb5b6813a203eabea', 'extra': {'build_url': 'https://jenkins.ceph.com/job/ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic/79034/', 'root_build_cause': 'SCMTRIGGER', 'version': '19.0.0-3450-g9a047fe7', 'node_name': '172.21.5.34+adami04', 'job_name': 'ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic', 'package_manager_version': '19.0.0-3450.g9a047fe7'}, 'url': 'https://4.chacra.ceph.com/r/ceph/wip-rishabh-testing-20240501.193033/9a047fe7599a549042d06a3fb5b6813a203eabea/centos/9/flavors/default/', 'distro_codename': None, 'modified': '2024-05-02 00:36:37.664792', 'distro_version': '9', 'project': 'ceph', 'flavor': 'default', 'ref': 'wip-rishabh-testing-20240501.193033', 'chacra_url': 'https://4.chacra.ceph.com/repos/ceph/wip-rishabh-testing-20240501.193033/9a047fe7599a549042d06a3fb5b6813a203eabea/centos/9/flavors/default/', 'archs': ['source', 'x86_64'], 'distro': 'centos'}] 2024-05-03T21:09:34.999 INFO:tasks.util.chacra:got chacra host 4.chacra.ceph.com, ref wip-rishabh-testing-20240501.193033, sha1 9a047fe7599a549042d06a3fb5b6813a203eabea from https://shaman.ceph.com/api/search/?project=ceph&distros=centos%2F9%2Fx86_64&flavor=default&sha1=9a047fe7599a549042d06a3fb5b6813a203eabea 2024-05-03T21:09:35.002 INFO:tasks.cephadm:Discovered cachra url: https://4.chacra.ceph.com/binaries/ceph/wip-rishabh-testing-20240501.193033/9a047fe7599a549042d06a3fb5b6813a203eabea/centos/9/x86_64/flavors/default/cephadm 2024-05-03T21:09:35.003 DEBUG:teuthology.orchestra.run.smithi067:> curl --silent -L https://4.chacra.ceph.com/binaries/ceph/wip-rishabh-testing-20240501.193033/9a047fe7599a549042d06a3fb5b6813a203eabea/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-05-03T21:09:35.431 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r--. 1 ubuntu ubuntu 782467 May 3 21:09 /home/ubuntu/cephtest/cephadm 2024-05-03T21:09:35.431 DEBUG:teuthology.orchestra.run.smithi082:> curl --silent -L https://4.chacra.ceph.com/binaries/ceph/wip-rishabh-testing-20240501.193033/9a047fe7599a549042d06a3fb5b6813a203eabea/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-05-03T21:09:35.869 INFO:teuthology.orchestra.run.smithi082.stdout:-rw-r--r--. 1 ubuntu ubuntu 782467 May 3 21:09 /home/ubuntu/cephtest/cephadm 2024-05-03T21:09:35.869 DEBUG:teuthology.orchestra.run.smithi138:> curl --silent -L https://4.chacra.ceph.com/binaries/ceph/wip-rishabh-testing-20240501.193033/9a047fe7599a549042d06a3fb5b6813a203eabea/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-05-03T21:09:36.474 INFO:teuthology.orchestra.run.smithi138.stdout:-rw-r--r--. 1 ubuntu ubuntu 782467 May 3 21:09 /home/ubuntu/cephtest/cephadm 2024-05-03T21:09:36.475 DEBUG:teuthology.orchestra.run.smithi067:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-05-03T21:09:36.495 DEBUG:teuthology.orchestra.run.smithi082:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-05-03T21:09:36.514 DEBUG:teuthology.orchestra.run.smithi138:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-05-03T21:09:36.547 INFO:tasks.cephadm:Pulling image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9a047fe7599a549042d06a3fb5b6813a203eabea on all hosts... 2024-05-03T21:09:36.547 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9a047fe7599a549042d06a3fb5b6813a203eabea pull 2024-05-03T21:09:36.550 DEBUG:teuthology.orchestra.run.smithi082:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9a047fe7599a549042d06a3fb5b6813a203eabea pull 2024-05-03T21:09:36.557 DEBUG:teuthology.orchestra.run.smithi138:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9a047fe7599a549042d06a3fb5b6813a203eabea pull 2024-05-03T21:09:38.055 INFO:teuthology.orchestra.run.smithi082.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9a047fe7599a549042d06a3fb5b6813a203eabea... 2024-05-03T21:09:38.088 INFO:teuthology.orchestra.run.smithi138.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9a047fe7599a549042d06a3fb5b6813a203eabea... 2024-05-03T21:09:38.166 INFO:teuthology.orchestra.run.smithi067.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9a047fe7599a549042d06a3fb5b6813a203eabea... 2024-05-03T21:10:15.948 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2024-05-03T21:10:15.949 INFO:teuthology.orchestra.run.smithi138.stdout: "ceph_version": "ceph version 19.0.0-3450-g9a047fe7 (9a047fe7599a549042d06a3fb5b6813a203eabea) squid (dev)", 2024-05-03T21:10:15.949 INFO:teuthology.orchestra.run.smithi138.stdout: "image_id": "ee4d846202496ba6096593e8e8becc0700a5eb7ebd95536031187f5f337d4012", 2024-05-03T21:10:15.949 INFO:teuthology.orchestra.run.smithi138.stdout: "repo_digests": [ 2024-05-03T21:10:15.949 INFO:teuthology.orchestra.run.smithi138.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:68e15ae152ef5c805c733e76da8fc1d3dd4ee669d5ba025f457e05725baa24dd" 2024-05-03T21:10:15.949 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-05-03T21:10:15.949 INFO:teuthology.orchestra.run.smithi138.stdout:} 2024-05-03T21:10:18.420 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2024-05-03T21:10:18.420 INFO:teuthology.orchestra.run.smithi082.stdout: "ceph_version": "ceph version 19.0.0-3450-g9a047fe7 (9a047fe7599a549042d06a3fb5b6813a203eabea) squid (dev)", 2024-05-03T21:10:18.420 INFO:teuthology.orchestra.run.smithi082.stdout: "image_id": "ee4d846202496ba6096593e8e8becc0700a5eb7ebd95536031187f5f337d4012", 2024-05-03T21:10:18.420 INFO:teuthology.orchestra.run.smithi082.stdout: "repo_digests": [ 2024-05-03T21:10:18.420 INFO:teuthology.orchestra.run.smithi082.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:68e15ae152ef5c805c733e76da8fc1d3dd4ee669d5ba025f457e05725baa24dd" 2024-05-03T21:10:18.420 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-05-03T21:10:18.420 INFO:teuthology.orchestra.run.smithi082.stdout:} 2024-05-03T21:10:25.155 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-05-03T21:10:25.155 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph_version": "ceph version 19.0.0-3450-g9a047fe7 (9a047fe7599a549042d06a3fb5b6813a203eabea) squid (dev)", 2024-05-03T21:10:25.155 INFO:teuthology.orchestra.run.smithi067.stdout: "image_id": "ee4d846202496ba6096593e8e8becc0700a5eb7ebd95536031187f5f337d4012", 2024-05-03T21:10:25.155 INFO:teuthology.orchestra.run.smithi067.stdout: "repo_digests": [ 2024-05-03T21:10:25.155 INFO:teuthology.orchestra.run.smithi067.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:68e15ae152ef5c805c733e76da8fc1d3dd4ee669d5ba025f457e05725baa24dd" 2024-05-03T21:10:25.156 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-05-03T21:10:25.156 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-05-03T21:10:25.181 DEBUG:teuthology.orchestra.run.smithi067:> sudo mkdir -p /etc/ceph 2024-05-03T21:10:25.219 DEBUG:teuthology.orchestra.run.smithi082:> sudo mkdir -p /etc/ceph 2024-05-03T21:10:25.252 DEBUG:teuthology.orchestra.run.smithi138:> sudo mkdir -p /etc/ceph 2024-05-03T21:10:25.284 DEBUG:teuthology.orchestra.run.smithi067:> sudo chmod 777 /etc/ceph 2024-05-03T21:10:25.309 DEBUG:teuthology.orchestra.run.smithi082:> sudo chmod 777 /etc/ceph 2024-05-03T21:10:25.336 DEBUG:teuthology.orchestra.run.smithi138:> sudo chmod 777 /etc/ceph 2024-05-03T21:10:25.364 INFO:tasks.cephadm:Writing seed config... 2024-05-03T21:10:25.365 INFO:tasks.cephadm: override: [client] client mount timeout = 600 2024-05-03T21:10:25.365 INFO:tasks.cephadm: override: [client] debug client = 20 2024-05-03T21:10:25.365 INFO:tasks.cephadm: override: [client] debug ms = 1 2024-05-03T21:10:25.365 INFO:tasks.cephadm: override: [client] fuse default permissions = False 2024-05-03T21:10:25.365 INFO:tasks.cephadm: override: [client] fuse set user groups = True 2024-05-03T21:10:25.366 INFO:tasks.cephadm: override: [client] rados mon op timeout = 900 2024-05-03T21:10:25.366 INFO:tasks.cephadm: override: [client] rados osd op timeout = 900 2024-05-03T21:10:25.366 INFO:tasks.cephadm: override: [global] ms die on skipped message = False 2024-05-03T21:10:25.366 INFO:tasks.cephadm: override: [mds] debug mds = 20 2024-05-03T21:10:25.366 INFO:tasks.cephadm: override: [mds] debug mds balancer = 20 2024-05-03T21:10:25.366 INFO:tasks.cephadm: override: [mds] debug ms = 1 2024-05-03T21:10:25.366 INFO:tasks.cephadm: override: [mds] mds bal fragment size max = 10000 2024-05-03T21:10:25.366 INFO:tasks.cephadm: override: [mds] mds bal merge size = 5 2024-05-03T21:10:25.366 INFO:tasks.cephadm: override: [mds] mds bal split bits = 3 2024-05-03T21:10:25.366 INFO:tasks.cephadm: override: [mds] mds bal split size = 100 2024-05-03T21:10:25.366 INFO:tasks.cephadm: override: [mds] mds debug frag = True 2024-05-03T21:10:25.366 INFO:tasks.cephadm: override: [mds] mds debug scatterstat = True 2024-05-03T21:10:25.366 INFO:tasks.cephadm: override: [mds] mds op complaint time = 180 2024-05-03T21:10:25.367 INFO:tasks.cephadm: override: [mds] mds verify scatter = True 2024-05-03T21:10:25.367 INFO:tasks.cephadm: override: [mds] mds_bal_replicate_threshold = 1 2024-05-03T21:10:25.367 INFO:tasks.cephadm: override: [mds] mds_export_ephemeral_random_max = 0.1 2024-05-03T21:10:25.367 INFO:tasks.cephadm: override: [mds] osd op complaint time = 180 2024-05-03T21:10:25.367 INFO:tasks.cephadm: override: [mds] rados mon op timeout = 900 2024-05-03T21:10:25.367 INFO:tasks.cephadm: override: [mds] rados osd op timeout = 900 2024-05-03T21:10:25.367 INFO:tasks.cephadm: override: [mgr] cephsqlite lock renewal timeout = 900000 2024-05-03T21:10:25.367 INFO:tasks.cephadm: override: [mgr] debug client = 20 2024-05-03T21:10:25.367 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2024-05-03T21:10:25.367 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2024-05-03T21:10:25.367 INFO:tasks.cephadm: override: [mon] debug mon = 20 2024-05-03T21:10:25.367 INFO:tasks.cephadm: override: [mon] debug ms = 1 2024-05-03T21:10:25.367 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2024-05-03T21:10:25.367 INFO:tasks.cephadm: override: [mon] mon down mkfs grace = 300 2024-05-03T21:10:25.367 INFO:tasks.cephadm: override: [mon] mon op complaint time = 120 2024-05-03T21:10:25.368 INFO:tasks.cephadm: override: [osd] bdev async discard = True 2024-05-03T21:10:25.368 INFO:tasks.cephadm: override: [osd] bdev enable discard = True 2024-05-03T21:10:25.368 INFO:tasks.cephadm: override: [osd] bluestore allocator = bitmap 2024-05-03T21:10:25.368 INFO:tasks.cephadm: override: [osd] bluestore block size = 96636764160 2024-05-03T21:10:25.368 INFO:tasks.cephadm: override: [osd] bluestore fsck on mount = True 2024-05-03T21:10:25.368 INFO:tasks.cephadm: override: [osd] debug bluefs = 1/20 2024-05-03T21:10:25.368 INFO:tasks.cephadm: override: [osd] debug bluestore = 1/20 2024-05-03T21:10:25.368 INFO:tasks.cephadm: override: [osd] debug ms = 1 2024-05-03T21:10:25.368 INFO:tasks.cephadm: override: [osd] debug osd = 20 2024-05-03T21:10:25.368 INFO:tasks.cephadm: override: [osd] debug rocksdb = 4/10 2024-05-03T21:10:25.368 INFO:tasks.cephadm: override: [osd] mon osd backfillfull_ratio = 0.85 2024-05-03T21:10:25.368 INFO:tasks.cephadm: override: [osd] mon osd full ratio = 0.9 2024-05-03T21:10:25.368 INFO:tasks.cephadm: override: [osd] mon osd nearfull ratio = 0.8 2024-05-03T21:10:25.369 INFO:tasks.cephadm: override: [osd] osd failsafe full ratio = 0.95 2024-05-03T21:10:25.369 INFO:tasks.cephadm: override: [osd] osd objectstore = bluestore 2024-05-03T21:10:25.369 INFO:tasks.cephadm: override: [osd] osd op complaint time = 180 2024-05-03T21:10:25.369 INFO:tasks.cephadm: override: [osd] osd shutdown pgref assert = True 2024-05-03T21:10:25.369 INFO:tasks.cephadm: override: [osd] osd_max_omap_entries_per_request = 10 2024-05-03T21:10:25.370 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-05-03T21:10:25.370 DEBUG:teuthology.orchestra.run.smithi067:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2024-05-03T21:10:25.385 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log to file = true mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = 713bc57a-0991-11ef-bc97-c7b262605968 ms die on skipped message = False [osd] osd scrub load threshold = 5.0 osd scrub max interval = 600 osd mclock profile = high_recovery_ops osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = True bdev debug aio = true osd sloppy crc = true bdev async discard = True bdev enable discard = True bluestore allocator = bitmap bluestore block size = 96636764160 bluestore fsck on mount = True debug bluefs = 1/20 debug bluestore = 1/20 debug ms = 1 debug osd = 20 debug rocksdb = 4/10 mon osd backfillfull_ratio = 0.85 mon osd full ratio = 0.9 mon osd nearfull ratio = 0.8 osd failsafe full ratio = 0.95 osd objectstore = bluestore osd op complaint time = 180 osd_max_omap_entries_per_request = 10 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false cephsqlite lock renewal timeout = 900000 debug client = 20 debug mgr = 20 debug ms = 1 [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 mon down mkfs grace = 300 mon op complaint time = 120 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true [client] client mount timeout = 600 debug client = 20 debug ms = 1 fuse default permissions = False fuse set user groups = True rados mon op timeout = 900 rados osd op timeout = 900 [mds] debug mds = 20 debug mds balancer = 20 debug ms = 1 mds bal fragment size max = 10000 mds bal merge size = 5 mds bal split bits = 3 mds bal split size = 100 mds debug frag = True mds debug scatterstat = True mds op complaint time = 180 mds verify scatter = True mds_bal_replicate_threshold = 1 mds_export_ephemeral_random_max = 0.1 osd op complaint time = 180 rados mon op timeout = 900 rados osd op timeout = 900 2024-05-03T21:10:25.386 DEBUG:teuthology.orchestra.run.smithi067:mon.a> sudo journalctl -f -n 0 -u ceph-713bc57a-0991-11ef-bc97-c7b262605968@mon.a.service 2024-05-03T21:10:25.428 DEBUG:teuthology.orchestra.run.smithi067:mgr.x> sudo journalctl -f -n 0 -u ceph-713bc57a-0991-11ef-bc97-c7b262605968@mgr.x.service 2024-05-03T21:10:25.471 INFO:tasks.cephadm:Bootstrapping... 2024-05-03T21:10:25.471 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9a047fe7599a549042d06a3fb5b6813a203eabea -v bootstrap --fsid 713bc57a-0991-11ef-bc97-c7b262605968 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-id a --mgr-id x --orphan-initial-daemons --skip-monitoring-stack --mon-ip 172.21.15.67 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2024-05-03T21:10:25.667 INFO:teuthology.orchestra.run.smithi067.stdout:-------------------------------------------------------------------------------- 2024-05-03T21:10:25.668 INFO:teuthology.orchestra.run.smithi067.stdout:cephadm ['--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9a047fe7599a549042d06a3fb5b6813a203eabea', '-v', 'bootstrap', '--fsid', '713bc57a-0991-11ef-bc97-c7b262605968', '--config', '/home/ubuntu/cephtest/seed.ceph.conf', '--output-config', '/etc/ceph/ceph.conf', '--output-keyring', '/etc/ceph/ceph.client.admin.keyring', '--output-pub-ssh-key', '/home/ubuntu/cephtest/ceph.pub', '--mon-id', 'a', '--mgr-id', 'x', '--orphan-initial-daemons', '--skip-monitoring-stack', '--mon-ip', '172.21.15.67', '--skip-admin-label'] 2024-05-03T21:10:25.668 INFO:teuthology.orchestra.run.smithi067.stderr:Specifying an fsid for your cluster offers no advantages and may increase the likelihood of fsid conflicts. 2024-05-03T21:10:25.668 INFO:teuthology.orchestra.run.smithi067.stdout:Verifying podman|docker is present... 2024-05-03T21:10:25.708 INFO:teuthology.orchestra.run.smithi067.stdout:/bin/podman: stdout 4.9.4-dev 2024-05-03T21:10:25.708 INFO:teuthology.orchestra.run.smithi067.stdout:Verifying lvm2 is present... 2024-05-03T21:10:25.708 INFO:teuthology.orchestra.run.smithi067.stdout:Verifying time synchronization is in place... 2024-05-03T21:10:25.716 INFO:teuthology.orchestra.run.smithi067.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-05-03T21:10:25.716 INFO:teuthology.orchestra.run.smithi067.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-05-03T21:10:25.722 INFO:teuthology.orchestra.run.smithi067.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-05-03T21:10:25.722 INFO:teuthology.orchestra.run.smithi067.stdout:systemctl: stdout inactive 2024-05-03T21:10:25.728 INFO:teuthology.orchestra.run.smithi067.stdout:systemctl: stdout enabled 2024-05-03T21:10:25.735 INFO:teuthology.orchestra.run.smithi067.stdout:systemctl: stdout active 2024-05-03T21:10:25.735 INFO:teuthology.orchestra.run.smithi067.stdout:Unit chronyd.service is enabled and running 2024-05-03T21:10:25.735 INFO:teuthology.orchestra.run.smithi067.stdout:Repeating the final host check... 2024-05-03T21:10:25.777 INFO:teuthology.orchestra.run.smithi067.stdout:/bin/podman: stdout 4.9.4-dev 2024-05-03T21:10:25.777 INFO:teuthology.orchestra.run.smithi067.stdout:podman (/bin/podman) version 4.9.4 is present 2024-05-03T21:10:25.777 INFO:teuthology.orchestra.run.smithi067.stdout:systemctl is present 2024-05-03T21:10:25.778 INFO:teuthology.orchestra.run.smithi067.stdout:lvcreate is present 2024-05-03T21:10:25.784 INFO:teuthology.orchestra.run.smithi067.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-05-03T21:10:25.784 INFO:teuthology.orchestra.run.smithi067.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-05-03T21:10:25.790 INFO:teuthology.orchestra.run.smithi067.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-05-03T21:10:25.790 INFO:teuthology.orchestra.run.smithi067.stdout:systemctl: stdout inactive 2024-05-03T21:10:25.796 INFO:teuthology.orchestra.run.smithi067.stdout:systemctl: stdout enabled 2024-05-03T21:10:25.803 INFO:teuthology.orchestra.run.smithi067.stdout:systemctl: stdout active 2024-05-03T21:10:25.803 INFO:teuthology.orchestra.run.smithi067.stdout:Unit chronyd.service is enabled and running 2024-05-03T21:10:25.803 INFO:teuthology.orchestra.run.smithi067.stdout:Host looks OK 2024-05-03T21:10:25.803 INFO:teuthology.orchestra.run.smithi067.stdout:Cluster fsid: 713bc57a-0991-11ef-bc97-c7b262605968 2024-05-03T21:10:25.803 INFO:teuthology.orchestra.run.smithi067.stdout:Acquiring lock 140288507686192 on /run/cephadm/713bc57a-0991-11ef-bc97-c7b262605968.lock 2024-05-03T21:10:25.804 INFO:teuthology.orchestra.run.smithi067.stdout:Lock 140288507686192 acquired on /run/cephadm/713bc57a-0991-11ef-bc97-c7b262605968.lock 2024-05-03T21:10:25.804 INFO:teuthology.orchestra.run.smithi067.stdout:Verifying IP 172.21.15.67 port 3300 ... 2024-05-03T21:10:25.804 INFO:teuthology.orchestra.run.smithi067.stdout:Verifying IP 172.21.15.67 port 6789 ... 2024-05-03T21:10:25.804 INFO:teuthology.orchestra.run.smithi067.stdout:Base mon IP(s) is [172.21.15.67:3300, 172.21.15.67:6789], mon addrv is [v2:172.21.15.67:3300,v1:172.21.15.67:6789] 2024-05-03T21:10:25.807 INFO:teuthology.orchestra.run.smithi067.stdout:/sbin/ip: stdout default via 172.21.15.254 dev enp3s0f1 proto dhcp src 172.21.15.67 metric 100 2024-05-03T21:10:25.807 INFO:teuthology.orchestra.run.smithi067.stdout:/sbin/ip: stdout 172.21.0.0/20 dev enp3s0f1 proto kernel scope link src 172.21.15.67 metric 100 2024-05-03T21:10:25.810 INFO:teuthology.orchestra.run.smithi067.stdout:/sbin/ip: stdout ::1 dev lo proto kernel metric 256 pref medium 2024-05-03T21:10:25.810 INFO:teuthology.orchestra.run.smithi067.stdout:/sbin/ip: stdout fe80::/64 dev enp3s0f1 proto kernel metric 1024 pref medium 2024-05-03T21:10:25.810 INFO:teuthology.orchestra.run.smithi067.stdout:/sbin/ip: stdout default via fe80::327c:5e00:6487:71e0 dev enp3s0f1 proto ra metric 100 pref medium 2024-05-03T21:10:25.812 INFO:teuthology.orchestra.run.smithi067.stdout:/sbin/ip: stdout 1: lo: mtu 65536 state UNKNOWN qlen 1000 2024-05-03T21:10:25.812 INFO:teuthology.orchestra.run.smithi067.stdout:/sbin/ip: stdout inet6 ::1/128 scope host 2024-05-03T21:10:25.812 INFO:teuthology.orchestra.run.smithi067.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-05-03T21:10:25.812 INFO:teuthology.orchestra.run.smithi067.stdout:/sbin/ip: stdout 5: enp3s0f1: mtu 1500 state UP qlen 1000 2024-05-03T21:10:25.812 INFO:teuthology.orchestra.run.smithi067.stdout:/sbin/ip: stdout inet6 fe80::b1d0:1ed6:6e4a:bb3d/64 scope link noprefixroute 2024-05-03T21:10:25.812 INFO:teuthology.orchestra.run.smithi067.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-05-03T21:10:25.813 INFO:teuthology.orchestra.run.smithi067.stdout:Mon IP `172.21.15.67` is in CIDR network `172.21.0.0/20` 2024-05-03T21:10:25.813 INFO:teuthology.orchestra.run.smithi067.stdout:Mon IP `172.21.15.67` is in CIDR network `172.21.0.0/20` 2024-05-03T21:10:25.813 INFO:teuthology.orchestra.run.smithi067.stdout:Inferred mon public CIDR from local network configuration ['172.21.0.0/20', '172.21.0.0/20'] 2024-05-03T21:10:25.814 INFO:teuthology.orchestra.run.smithi067.stdout:Internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2024-05-03T21:10:25.814 INFO:teuthology.orchestra.run.smithi067.stdout:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9a047fe7599a549042d06a3fb5b6813a203eabea... 2024-05-03T21:10:26.283 INFO:teuthology.orchestra.run.smithi067.stdout:/bin/podman: stdout ee4d846202496ba6096593e8e8becc0700a5eb7ebd95536031187f5f337d4012 2024-05-03T21:10:26.283 INFO:teuthology.orchestra.run.smithi067.stdout:/bin/podman: stderr Trying to pull quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9a047fe7599a549042d06a3fb5b6813a203eabea... 2024-05-03T21:10:26.283 INFO:teuthology.orchestra.run.smithi067.stdout:/bin/podman: stderr Getting image source signatures 2024-05-03T21:10:26.284 INFO:teuthology.orchestra.run.smithi067.stdout:/bin/podman: stderr Copying blob sha256:e2f2fd2adca92139da4b4b907fa90a6775d3eb4eddaa977c42ef37c66eea70bd 2024-05-03T21:10:26.284 INFO:teuthology.orchestra.run.smithi067.stdout:/bin/podman: stderr Copying blob sha256:8173fd91c96404ae0424bb08037c57749051dcbdfaeee9354a65e201dad7b0e2 2024-05-03T21:10:26.284 INFO:teuthology.orchestra.run.smithi067.stdout:/bin/podman: stderr Copying config sha256:ee4d846202496ba6096593e8e8becc0700a5eb7ebd95536031187f5f337d4012 2024-05-03T21:10:26.284 INFO:teuthology.orchestra.run.smithi067.stdout:/bin/podman: stderr Writing manifest to image destination 2024-05-03T21:10:27.221 INFO:teuthology.orchestra.run.smithi067.stdout:ceph: stdout ceph version 19.0.0-3450-g9a047fe7 (9a047fe7599a549042d06a3fb5b6813a203eabea) squid (dev) 2024-05-03T21:10:27.221 INFO:teuthology.orchestra.run.smithi067.stdout:Ceph version: ceph version 19.0.0-3450-g9a047fe7 (9a047fe7599a549042d06a3fb5b6813a203eabea) squid (dev) 2024-05-03T21:10:27.221 INFO:teuthology.orchestra.run.smithi067.stdout:Extracting ceph user uid/gid from container image... 2024-05-03T21:10:28.196 INFO:teuthology.orchestra.run.smithi067.stdout:stat: stdout 167 167 2024-05-03T21:10:28.197 INFO:teuthology.orchestra.run.smithi067.stdout:Creating initial keys... 2024-05-03T21:10:29.010 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph-authtool: stdout AQDEUjVm0H0IHBAAuJADOE6+fP1UotDkdBXSyQ== 2024-05-03T21:10:29.997 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph-authtool: stdout AQDFUjVmxpOUFRAA0QUkwc8y8wBqzCf6ZUNU1g== 2024-05-03T21:10:30.903 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph-authtool: stdout AQDGUjVm8ALRExAAxCZS7a1NjIX3pkT3RhJoSA== 2024-05-03T21:10:30.904 INFO:teuthology.orchestra.run.smithi067.stdout:Creating initial monmap... 2024-05-03T21:10:31.759 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: monmap file /tmp/monmap 2024-05-03T21:10:31.759 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/monmaptool: stdout setting min_mon_release = quincy 2024-05-03T21:10:31.759 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: set fsid to 713bc57a-0991-11ef-bc97-c7b262605968 2024-05-03T21:10:31.759 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-05-03T21:10:31.759 INFO:teuthology.orchestra.run.smithi067.stdout:monmaptool for a [v2:172.21.15.67:3300,v1:172.21.15.67:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2024-05-03T21:10:31.760 INFO:teuthology.orchestra.run.smithi067.stdout:setting min_mon_release = quincy 2024-05-03T21:10:31.760 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/monmaptool: set fsid to 713bc57a-0991-11ef-bc97-c7b262605968 2024-05-03T21:10:31.760 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-05-03T21:10:31.760 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-05-03T21:10:31.760 INFO:teuthology.orchestra.run.smithi067.stdout:Creating mon... 2024-05-03T21:10:32.622 INFO:teuthology.orchestra.run.smithi067.stdout:create mon.a on 2024-05-03T21:10:32.831 INFO:teuthology.orchestra.run.smithi067.stdout:systemctl: stderr Removed "/etc/systemd/system/multi-user.target.wants/ceph.target". 2024-05-03T21:10:32.981 INFO:teuthology.orchestra.run.smithi067.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2024-05-03T21:10:33.151 INFO:teuthology.orchestra.run.smithi067.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph-713bc57a-0991-11ef-bc97-c7b262605968.target → /etc/systemd/system/ceph-713bc57a-0991-11ef-bc97-c7b262605968.target. 2024-05-03T21:10:33.151 INFO:teuthology.orchestra.run.smithi067.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph.target.wants/ceph-713bc57a-0991-11ef-bc97-c7b262605968.target → /etc/systemd/system/ceph-713bc57a-0991-11ef-bc97-c7b262605968.target. 2024-05-03T21:10:33.406 INFO:teuthology.orchestra.run.smithi067.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-713bc57a-0991-11ef-bc97-c7b262605968@mon.a 2024-05-03T21:10:33.406 INFO:teuthology.orchestra.run.smithi067.stdout:systemctl: stderr Failed to reset failed state of unit ceph-713bc57a-0991-11ef-bc97-c7b262605968@mon.a.service: Unit ceph-713bc57a-0991-11ef-bc97-c7b262605968@mon.a.service not loaded. 2024-05-03T21:10:33.586 INFO:teuthology.orchestra.run.smithi067.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-713bc57a-0991-11ef-bc97-c7b262605968.target.wants/ceph-713bc57a-0991-11ef-bc97-c7b262605968@mon.a.service → /etc/systemd/system/ceph-713bc57a-0991-11ef-bc97-c7b262605968@.service. 2024-05-03T21:10:34.155 INFO:teuthology.orchestra.run.smithi067.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-05-03T21:10:34.156 INFO:teuthology.orchestra.run.smithi067.stdout:systemctl: stdout disabled 2024-05-03T21:10:34.162 INFO:teuthology.orchestra.run.smithi067.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-05-03T21:10:34.163 INFO:teuthology.orchestra.run.smithi067.stdout:systemctl: stdout inactive 2024-05-03T21:10:34.163 INFO:teuthology.orchestra.run.smithi067.stdout:firewalld.service is not enabled 2024-05-03T21:10:34.163 INFO:teuthology.orchestra.run.smithi067.stdout:Not possible to enable service . firewalld.service is not available 2024-05-03T21:10:34.163 INFO:teuthology.orchestra.run.smithi067.stdout:Waiting for mon to start... 2024-05-03T21:10:34.163 INFO:teuthology.orchestra.run.smithi067.stdout:Waiting for mon... 2024-05-03T21:10:35.156 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout cluster: 2024-05-03T21:10:35.157 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout id: 713bc57a-0991-11ef-bc97-c7b262605968 2024-05-03T21:10:35.157 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout health: HEALTH_OK 2024-05-03T21:10:35.157 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout 2024-05-03T21:10:35.157 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout services: 2024-05-03T21:10:35.157 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout mon: 1 daemons, quorum a (age 0.606199s) 2024-05-03T21:10:35.157 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout mgr: no daemons active 2024-05-03T21:10:35.157 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout osd: 0 osds: 0 up, 0 in 2024-05-03T21:10:35.157 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout 2024-05-03T21:10:35.157 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout data: 2024-05-03T21:10:35.157 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout pools: 0 pools, 0 pgs 2024-05-03T21:10:35.157 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout objects: 0 objects, 0 B 2024-05-03T21:10:35.157 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout usage: 0 B used, 0 B / 0 B avail 2024-05-03T21:10:35.157 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout pgs: 2024-05-03T21:10:35.158 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout 2024-05-03T21:10:35.158 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:10:34.594+0000 7f061d8a8700 1 Processor -- start 2024-05-03T21:10:35.158 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:10:34.594+0000 7f061d8a8700 1 -- start start 2024-05-03T21:10:35.158 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:10:34.594+0000 7f061d8a8700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0618109cc0 0x7f061810c0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:10:35.158 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:10:34.594+0000 7f061d8a8700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f061807ced0 con 0x7f0618109cc0 2024-05-03T21:10:35.158 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:10:34.595+0000 7f0616ffd700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0618109cc0 0x7f061810c0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:10:35.158 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:10:34.595+0000 7f0616ffd700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0618109cc0 0x7f061810c0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:36794/0 (socket says 172.21.15.67:36794) 2024-05-03T21:10:35.158 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:10:34.595+0000 7f0616ffd700 1 -- 172.21.15.67:0/626079511 learned_addr learned my addr 172.21.15.67:0/626079511 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-03T21:10:35.158 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:10:34.595+0000 7f0616ffd700 1 -- 172.21.15.67:0/626079511 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f061810c5f0 con 0x7f0618109cc0 2024-05-03T21:10:35.158 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:10:34.595+0000 7f0616ffd700 1 --2- 172.21.15.67:0/626079511 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0618109cc0 0x7f061810c0b0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f0610009e80 tx=0x7f061000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6e47be6e17310f29 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T21:10:35.158 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:10:34.595+0000 7f0615ffb700 1 -- 172.21.15.67:0/626079511 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0610017070 con 0x7f0618109cc0 2024-05-03T21:10:35.158 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:10:34.596+0000 7f0615ffb700 1 -- 172.21.15.67:0/626079511 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0610003a20 con 0x7f0618109cc0 2024-05-03T21:10:35.158 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:10:34.596+0000 7f0615ffb700 1 -- 172.21.15.67:0/626079511 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0610003d20 con 0x7f0618109cc0 2024-05-03T21:10:35.159 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:10:34.596+0000 7f061d8a8700 1 -- 172.21.15.67:0/626079511 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0618109cc0 msgr2=0x7f061810c0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:10:35.159 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:10:34.596+0000 7f061d8a8700 1 --2- 172.21.15.67:0/626079511 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0618109cc0 0x7f061810c0b0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f0610009e80 tx=0x7f061000b330 comp rx=0 tx=0).stop 2024-05-03T21:10:35.159 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:10:34.596+0000 7f061d8a8700 1 -- 172.21.15.67:0/626079511 shutdown_connections 2024-05-03T21:10:35.159 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:10:34.596+0000 7f061d8a8700 1 --2- 172.21.15.67:0/626079511 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0618109cc0 0x7f061810c0b0 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:10:35.159 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:10:34.596+0000 7f061d8a8700 1 -- 172.21.15.67:0/626079511 >> 172.21.15.67:0/626079511 conn(0x7f0618103750 msgr2=0x7f0618105b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T21:10:35.159 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:10:34.596+0000 7f061d8a8700 1 -- 172.21.15.67:0/626079511 shutdown_connections 2024-05-03T21:10:35.159 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:10:34.596+0000 7f061d8a8700 1 -- 172.21.15.67:0/626079511 wait complete. 2024-05-03T21:10:35.159 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:10:34.597+0000 7f061d8a8700 1 Processor -- start 2024-05-03T21:10:35.159 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:10:34.597+0000 7f061d8a8700 1 -- start start 2024-05-03T21:10:35.159 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:10:34.597+0000 7f061d8a8700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0618109cc0 0x7f0618109540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:10:35.159 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:10:34.597+0000 7f061d8a8700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f061810cac0 con 0x7f0618109cc0 2024-05-03T21:10:35.159 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:10:34.597+0000 7f0616ffd700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0618109cc0 0x7f0618109540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:10:35.159 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:10:34.597+0000 7f0616ffd700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0618109cc0 0x7f0618109540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:36802/0 (socket says 172.21.15.67:36802) 2024-05-03T21:10:35.159 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:10:34.597+0000 7f0616ffd700 1 -- 172.21.15.67:0/204508989 learned_addr learned my addr 172.21.15.67:0/204508989 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-03T21:10:35.160 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:10:34.597+0000 7f0616ffd700 1 -- 172.21.15.67:0/204508989 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0618109a80 con 0x7f0618109cc0 2024-05-03T21:10:35.160 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:10:34.598+0000 7f0616ffd700 1 --2- 172.21.15.67:0/204508989 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0618109cc0 0x7f0618109540 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f061807c310 tx=0x7f0610004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T21:10:35.160 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:10:34.598+0000 7f061c8a6700 1 -- 172.21.15.67:0/204508989 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0610022020 con 0x7f0618109cc0 2024-05-03T21:10:35.160 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:10:34.598+0000 7f061c8a6700 1 -- 172.21.15.67:0/204508989 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0610018040 con 0x7f0618109cc0 2024-05-03T21:10:35.160 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:10:34.598+0000 7f061c8a6700 1 -- 172.21.15.67:0/204508989 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0610017070 con 0x7f0618109cc0 2024-05-03T21:10:35.160 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:10:34.598+0000 7f061d8a8700 1 -- 172.21.15.67:0/204508989 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0618107c70 con 0x7f0618109cc0 2024-05-03T21:10:35.160 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:10:34.598+0000 7f061d8a8700 1 -- 172.21.15.67:0/204508989 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f06181081b0 con 0x7f0618109cc0 2024-05-03T21:10:35.160 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:10:34.598+0000 7f061c8a6700 1 -- 172.21.15.67:0/204508989 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f0610025020 con 0x7f0618109cc0 2024-05-03T21:10:35.160 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:10:34.598+0000 7f061c8a6700 1 -- 172.21.15.67:0/204508989 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f061000f050 con 0x7f0618109cc0 2024-05-03T21:10:35.160 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:10:34.599+0000 7f061d8a8700 1 -- 172.21.15.67:0/204508989 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0618064b00 con 0x7f0618109cc0 2024-05-03T21:10:35.160 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:10:34.601+0000 7f061c8a6700 1 -- 172.21.15.67:0/204508989 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7f061002c030 con 0x7f0618109cc0 2024-05-03T21:10:35.160 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:10:34.674+0000 7f061d8a8700 1 -- 172.21.15.67:0/204508989 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "status"} v 0) -- 0x7f0618196310 con 0x7f0618109cc0 2024-05-03T21:10:35.160 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:10:34.674+0000 7f061c8a6700 1 -- 172.21.15.67:0/204508989 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "status"}]=0 v0) ==== 54+0+317 (secure 0 0 0) 0x7f061001e480 con 0x7f0618109cc0 2024-05-03T21:10:35.160 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:10:34.675+0000 7f061d8a8700 1 -- 172.21.15.67:0/204508989 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0618109cc0 msgr2=0x7f0618109540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:10:35.160 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:10:34.675+0000 7f061d8a8700 1 --2- 172.21.15.67:0/204508989 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0618109cc0 0x7f0618109540 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f061807c310 tx=0x7f0610004ad0 comp rx=0 tx=0).stop 2024-05-03T21:10:35.161 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:10:34.675+0000 7f061d8a8700 1 -- 172.21.15.67:0/204508989 shutdown_connections 2024-05-03T21:10:35.161 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:10:34.675+0000 7f061d8a8700 1 --2- 172.21.15.67:0/204508989 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0618109cc0 0x7f0618109540 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:10:35.161 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:10:34.675+0000 7f061d8a8700 1 -- 172.21.15.67:0/204508989 >> 172.21.15.67:0/204508989 conn(0x7f0618103750 msgr2=0x7f0618103b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T21:10:35.161 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:10:34.675+0000 7f061d8a8700 1 -- 172.21.15.67:0/204508989 shutdown_connections 2024-05-03T21:10:35.161 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:10:34.675+0000 7f061d8a8700 1 -- 172.21.15.67:0/204508989 wait complete. 2024-05-03T21:10:35.161 INFO:teuthology.orchestra.run.smithi067.stdout:mon is available 2024-05-03T21:10:35.161 INFO:teuthology.orchestra.run.smithi067.stdout:Assimilating anything we can from ceph.conf... 2024-05-03T21:10:36.188 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout 2024-05-03T21:10:36.188 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout [global] 2024-05-03T21:10:36.188 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout fsid = 713bc57a-0991-11ef-bc97-c7b262605968 2024-05-03T21:10:36.188 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout mon_cluster_log_file_level = debug 2024-05-03T21:10:36.188 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout mon_host = [v2:172.21.15.67:3300,v1:172.21.15.67:6789] 2024-05-03T21:10:36.188 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-05-03T21:10:36.188 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-05-03T21:10:36.189 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-05-03T21:10:36.189 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-05-03T21:10:36.189 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout 2024-05-03T21:10:36.189 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout [mgr] 2024-05-03T21:10:36.189 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-05-03T21:10:36.189 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout 2024-05-03T21:10:36.189 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout [osd] 2024-05-03T21:10:36.189 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout bdev_async_discard = True 2024-05-03T21:10:36.189 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2024-05-03T21:10:36.189 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2024-05-03T21:10:36.189 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2024-05-03T21:10:36.189 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-05-03T21:10:36.189 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-05-03T21:10:36.189 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:10:35.532+0000 7fd58d409700 1 Processor -- start 2024-05-03T21:10:36.189 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:10:35.532+0000 7fd58d409700 1 -- start start 2024-05-03T21:10:36.190 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:10:35.532+0000 7fd58d409700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd58807edf0 0x7fd58807d360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:10:36.190 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:10:35.532+0000 7fd58d409700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fd58807d8a0 con 0x7fd58807edf0 2024-05-03T21:10:36.190 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:10:35.532+0000 7fd586ffd700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd58807edf0 0x7fd58807d360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:10:36.190 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:10:35.532+0000 7fd586ffd700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd58807edf0 0x7fd58807d360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:36818/0 (socket says 172.21.15.67:36818) 2024-05-03T21:10:36.190 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:10:35.532+0000 7fd586ffd700 1 -- 172.21.15.67:0/3796711051 learned_addr learned my addr 172.21.15.67:0/3796711051 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-03T21:10:36.190 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:10:35.533+0000 7fd586ffd700 1 -- 172.21.15.67:0/3796711051 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd58807da20 con 0x7fd58807edf0 2024-05-03T21:10:36.190 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:10:35.533+0000 7fd586ffd700 1 --2- 172.21.15.67:0/3796711051 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd58807edf0 0x7fd58807d360 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7fd580009e80 tx=0x7fd58000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=49dd6885a67dd974 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T21:10:36.190 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:10:35.533+0000 7fd585ffb700 1 -- 172.21.15.67:0/3796711051 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd580017070 con 0x7fd58807edf0 2024-05-03T21:10:36.190 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:10:35.533+0000 7fd585ffb700 1 -- 172.21.15.67:0/3796711051 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd580003a20 con 0x7fd58807edf0 2024-05-03T21:10:36.190 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:10:35.533+0000 7fd585ffb700 1 -- 172.21.15.67:0/3796711051 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd580003d20 con 0x7fd58807edf0 2024-05-03T21:10:36.190 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:10:35.534+0000 7fd58d409700 1 -- 172.21.15.67:0/3796711051 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd58807edf0 msgr2=0x7fd58807d360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:10:36.190 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:10:35.534+0000 7fd58d409700 1 --2- 172.21.15.67:0/3796711051 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd58807edf0 0x7fd58807d360 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7fd580009e80 tx=0x7fd58000b330 comp rx=0 tx=0).stop 2024-05-03T21:10:36.190 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:10:35.534+0000 7fd58d409700 1 -- 172.21.15.67:0/3796711051 shutdown_connections 2024-05-03T21:10:36.190 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:10:35.534+0000 7fd58d409700 1 --2- 172.21.15.67:0/3796711051 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd58807edf0 0x7fd58807d360 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:10:36.191 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:10:35.534+0000 7fd58d409700 1 -- 172.21.15.67:0/3796711051 >> 172.21.15.67:0/3796711051 conn(0x7fd588104950 msgr2=0x7fd588106d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T21:10:36.191 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:10:35.534+0000 7fd58d409700 1 -- 172.21.15.67:0/3796711051 shutdown_connections 2024-05-03T21:10:36.191 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:10:35.534+0000 7fd58d409700 1 -- 172.21.15.67:0/3796711051 wait complete. 2024-05-03T21:10:36.191 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:10:35.534+0000 7fd58d409700 1 Processor -- start 2024-05-03T21:10:36.191 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:10:35.535+0000 7fd58d409700 1 -- start start 2024-05-03T21:10:36.191 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:10:35.535+0000 7fd58d409700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd58807edf0 0x7fd58819eb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:10:36.191 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:10:35.535+0000 7fd58d409700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fd58807def0 con 0x7fd58807edf0 2024-05-03T21:10:36.191 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:10:35.535+0000 7fd586ffd700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd58807edf0 0x7fd58819eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:10:36.191 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:10:35.535+0000 7fd586ffd700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd58807edf0 0x7fd58819eb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:36830/0 (socket says 172.21.15.67:36830) 2024-05-03T21:10:36.191 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:10:35.535+0000 7fd586ffd700 1 -- 172.21.15.67:0/4091030148 learned_addr learned my addr 172.21.15.67:0/4091030148 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-03T21:10:36.191 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:10:35.535+0000 7fd586ffd700 1 -- 172.21.15.67:0/4091030148 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd58819f090 con 0x7fd58807edf0 2024-05-03T21:10:36.191 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:10:35.535+0000 7fd586ffd700 1 --2- 172.21.15.67:0/4091030148 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd58807edf0 0x7fd58819eb50 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7fd58807e430 tx=0x7fd580004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T21:10:36.191 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:10:35.536+0000 7fd57ffff700 1 -- 172.21.15.67:0/4091030148 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd580022020 con 0x7fd58807edf0 2024-05-03T21:10:36.191 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:10:35.536+0000 7fd57ffff700 1 -- 172.21.15.67:0/4091030148 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd580018040 con 0x7fd58807edf0 2024-05-03T21:10:36.192 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:10:35.536+0000 7fd57ffff700 1 -- 172.21.15.67:0/4091030148 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd580017070 con 0x7fd58807edf0 2024-05-03T21:10:36.192 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:10:35.536+0000 7fd58d409700 1 -- 172.21.15.67:0/4091030148 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd58819f320 con 0x7fd58807edf0 2024-05-03T21:10:36.192 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:10:35.536+0000 7fd58d409700 1 -- 172.21.15.67:0/4091030148 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd5881a2010 con 0x7fd58807edf0 2024-05-03T21:10:36.192 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:10:35.536+0000 7fd57ffff700 1 -- 172.21.15.67:0/4091030148 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fd580025020 con 0x7fd58807edf0 2024-05-03T21:10:36.192 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:10:35.536+0000 7fd57ffff700 1 -- 172.21.15.67:0/4091030148 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7fd58000f050 con 0x7fd58807edf0 2024-05-03T21:10:36.192 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:10:35.537+0000 7fd58d409700 1 -- 172.21.15.67:0/4091030148 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd588064b00 con 0x7fd58807edf0 2024-05-03T21:10:36.192 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:10:35.539+0000 7fd57ffff700 1 -- 172.21.15.67:0/4091030148 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7fd58002c030 con 0x7fd58807edf0 2024-05-03T21:10:36.192 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:10:35.609+0000 7fd58d409700 1 -- 172.21.15.67:0/4091030148 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) -- 0x7fd58802e6d0 con 0x7fd58807edf0 2024-05-03T21:10:36.192 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:10:35.612+0000 7fd57ffff700 1 -- 172.21.15.67:0/4091030148 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v2) ==== 70+0+492 (secure 0 0 0) 0x7fd58001e480 con 0x7fd58807edf0 2024-05-03T21:10:36.192 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:10:35.612+0000 7fd57ffff700 1 -- 172.21.15.67:0/4091030148 <== mon.0 v2:172.21.15.67:3300/0 8 ==== config(28 keys) ==== 1139+0+0 (secure 0 0 0) 0x7fd580003a20 con 0x7fd58807edf0 2024-05-03T21:10:36.192 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:10:35.613+0000 7fd58d409700 1 -- 172.21.15.67:0/4091030148 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd58807edf0 msgr2=0x7fd58819eb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:10:36.192 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:10:35.613+0000 7fd58d409700 1 --2- 172.21.15.67:0/4091030148 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd58807edf0 0x7fd58819eb50 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7fd58807e430 tx=0x7fd580004ad0 comp rx=0 tx=0).stop 2024-05-03T21:10:36.192 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:10:35.613+0000 7fd58d409700 1 -- 172.21.15.67:0/4091030148 shutdown_connections 2024-05-03T21:10:36.192 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:10:35.613+0000 7fd58d409700 1 --2- 172.21.15.67:0/4091030148 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd58807edf0 0x7fd58819eb50 unknown :-1 s=CLOSED pgs=4 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:10:36.192 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:10:35.613+0000 7fd58d409700 1 -- 172.21.15.67:0/4091030148 >> 172.21.15.67:0/4091030148 conn(0x7fd588104950 msgr2=0x7fd588106c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T21:10:36.193 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:10:35.613+0000 7fd58d409700 1 -- 172.21.15.67:0/4091030148 shutdown_connections 2024-05-03T21:10:36.193 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:10:35.613+0000 7fd58d409700 1 -- 172.21.15.67:0/4091030148 wait complete. 2024-05-03T21:10:36.193 INFO:teuthology.orchestra.run.smithi067.stdout:Generating new minimal ceph.conf... 2024-05-03T21:10:37.260 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:10:37 smithi067 ceph-mon[29353]: from='client.? 172.21.15.67:0/1516669148' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T21:10:37.267 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:10:36.614+0000 7f46218e2700 1 Processor -- start 2024-05-03T21:10:37.267 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:10:36.614+0000 7f46218e2700 1 -- start start 2024-05-03T21:10:37.267 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:10:36.614+0000 7f46218e2700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f461c108110 0x7f461c108530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:10:37.267 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:10:36.614+0000 7f46218e2700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f461c108b00 con 0x7f461c108110 2024-05-03T21:10:37.267 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:10:36.614+0000 7f461affd700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f461c108110 0x7f461c108530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:10:37.267 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:10:36.615+0000 7f461affd700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f461c108110 0x7f461c108530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:59606/0 (socket says 172.21.15.67:59606) 2024-05-03T21:10:37.267 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:10:36.615+0000 7f461affd700 1 -- 172.21.15.67:0/3809054913 learned_addr learned my addr 172.21.15.67:0/3809054913 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-03T21:10:37.267 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:10:36.615+0000 7f461affd700 1 -- 172.21.15.67:0/3809054913 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f461c109310 con 0x7f461c108110 2024-05-03T21:10:37.267 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:10:36.615+0000 7f461affd700 1 --2- 172.21.15.67:0/3809054913 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f461c108110 0x7f461c108530 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f4614009e80 tx=0x7f461400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=40e9638731662334 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T21:10:37.267 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:10:36.615+0000 7f4619ffb700 1 -- 172.21.15.67:0/3809054913 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4614017070 con 0x7f461c108110 2024-05-03T21:10:37.268 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:10:36.615+0000 7f4619ffb700 1 -- 172.21.15.67:0/3809054913 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(28 keys) ==== 1139+0+0 (secure 0 0 0) 0x7f461400b8c0 con 0x7f461c108110 2024-05-03T21:10:37.268 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:10:36.615+0000 7f4619ffb700 1 -- 172.21.15.67:0/3809054913 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4614004620 con 0x7f461c108110 2024-05-03T21:10:37.268 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:10:36.616+0000 7f46218e2700 1 -- 172.21.15.67:0/3809054913 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f461c108110 msgr2=0x7f461c108530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:10:37.268 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:10:36.616+0000 7f46218e2700 1 --2- 172.21.15.67:0/3809054913 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f461c108110 0x7f461c108530 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f4614009e80 tx=0x7f461400b330 comp rx=0 tx=0).stop 2024-05-03T21:10:37.268 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:10:36.616+0000 7f46218e2700 1 -- 172.21.15.67:0/3809054913 shutdown_connections 2024-05-03T21:10:37.268 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:10:36.616+0000 7f46218e2700 1 --2- 172.21.15.67:0/3809054913 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f461c108110 0x7f461c108530 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:10:37.268 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:10:36.616+0000 7f46218e2700 1 -- 172.21.15.67:0/3809054913 >> 172.21.15.67:0/3809054913 conn(0x7f461c1037b0 msgr2=0x7f461c105bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T21:10:37.268 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:10:36.616+0000 7f46218e2700 1 -- 172.21.15.67:0/3809054913 shutdown_connections 2024-05-03T21:10:37.268 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:10:36.616+0000 7f46218e2700 1 -- 172.21.15.67:0/3809054913 wait complete. 2024-05-03T21:10:37.268 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:10:36.616+0000 7f46218e2700 1 Processor -- start 2024-05-03T21:10:37.268 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:10:36.617+0000 7f46218e2700 1 -- start start 2024-05-03T21:10:37.268 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:10:36.617+0000 7f46218e2700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f461c108110 0x7f461c19d240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:10:37.268 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:10:36.617+0000 7f46218e2700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f461c10c6c0 con 0x7f461c108110 2024-05-03T21:10:37.268 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:10:36.617+0000 7f461affd700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f461c108110 0x7f461c19d240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:10:37.268 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:10:36.617+0000 7f461affd700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f461c108110 0x7f461c19d240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:59620/0 (socket says 172.21.15.67:59620) 2024-05-03T21:10:37.269 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:10:36.617+0000 7f461affd700 1 -- 172.21.15.67:0/1516669148 learned_addr learned my addr 172.21.15.67:0/1516669148 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-03T21:10:37.269 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:10:36.617+0000 7f461affd700 1 -- 172.21.15.67:0/1516669148 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f461c19d780 con 0x7f461c108110 2024-05-03T21:10:37.269 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:10:36.617+0000 7f461affd700 1 --2- 172.21.15.67:0/1516669148 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f461c108110 0x7f461c19d240 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f461400d010 tx=0x7f4614003d00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T21:10:37.269 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:10:36.618+0000 7f46208e0700 1 -- 172.21.15.67:0/1516669148 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4614021020 con 0x7f461c108110 2024-05-03T21:10:37.269 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:10:36.618+0000 7f46218e2700 1 -- 172.21.15.67:0/1516669148 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f461c19da10 con 0x7f461c108110 2024-05-03T21:10:37.269 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:10:36.618+0000 7f46208e0700 1 -- 172.21.15.67:0/1516669148 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(28 keys) ==== 1139+0+0 (secure 0 0 0) 0x7f46140094d0 con 0x7f461c108110 2024-05-03T21:10:37.269 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:10:36.618+0000 7f46208e0700 1 -- 172.21.15.67:0/1516669148 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4614017040 con 0x7f461c108110 2024-05-03T21:10:37.269 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:10:36.618+0000 7f46218e2700 1 -- 172.21.15.67:0/1516669148 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f461c1a0700 con 0x7f461c108110 2024-05-03T21:10:37.269 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:10:36.618+0000 7f46208e0700 1 -- 172.21.15.67:0/1516669148 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f4614024020 con 0x7f461c108110 2024-05-03T21:10:37.269 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:10:36.618+0000 7f46208e0700 1 -- 172.21.15.67:0/1516669148 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f461400f050 con 0x7f461c108110 2024-05-03T21:10:37.269 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:10:36.619+0000 7f46218e2700 1 -- 172.21.15.67:0/1516669148 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f461c10c840 con 0x7f461c108110 2024-05-03T21:10:37.269 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:10:36.621+0000 7f46208e0700 1 -- 172.21.15.67:0/1516669148 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7f461402b020 con 0x7f461c108110 2024-05-03T21:10:37.269 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:10:36.691+0000 7f46218e2700 1 -- 172.21.15.67:0/1516669148 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) -- 0x7f461c02e630 con 0x7f461c108110 2024-05-03T21:10:37.269 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:10:36.691+0000 7f46208e0700 1 -- 172.21.15.67:0/1516669148 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v2) ==== 76+0+175 (secure 0 0 0) 0x7f461400b860 con 0x7f461c108110 2024-05-03T21:10:37.270 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:10:36.692+0000 7f46218e2700 1 -- 172.21.15.67:0/1516669148 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f461c108110 msgr2=0x7f461c19d240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:10:37.270 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:10:36.692+0000 7f46218e2700 1 --2- 172.21.15.67:0/1516669148 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f461c108110 0x7f461c19d240 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f461400d010 tx=0x7f4614003d00 comp rx=0 tx=0).stop 2024-05-03T21:10:37.270 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:10:36.692+0000 7f46218e2700 1 -- 172.21.15.67:0/1516669148 shutdown_connections 2024-05-03T21:10:37.270 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:10:36.692+0000 7f46218e2700 1 --2- 172.21.15.67:0/1516669148 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f461c108110 0x7f461c19d240 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:10:37.270 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:10:36.692+0000 7f46218e2700 1 -- 172.21.15.67:0/1516669148 >> 172.21.15.67:0/1516669148 conn(0x7f461c1037b0 msgr2=0x7f461c104670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T21:10:37.270 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:10:36.692+0000 7f46218e2700 1 -- 172.21.15.67:0/1516669148 shutdown_connections 2024-05-03T21:10:37.270 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:10:36.692+0000 7f46218e2700 1 -- 172.21.15.67:0/1516669148 wait complete. 2024-05-03T21:10:37.270 INFO:teuthology.orchestra.run.smithi067.stdout:Restarting the monitor... 2024-05-03T21:10:37.539 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:10:37 smithi067 systemd[1]: Stopping Ceph mon.a for 713bc57a-0991-11ef-bc97-c7b262605968... 2024-05-03T21:10:37.963 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:10:37 smithi067 ceph-713bc57a-0991-11ef-bc97-c7b262605968-mon-a[29349]: 2024-05-03T21:10:37.536+0000 7f1c3ecfc700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-05-03T21:10:37.964 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:10:37 smithi067 ceph-713bc57a-0991-11ef-bc97-c7b262605968-mon-a[29349]: 2024-05-03T21:10:37.536+0000 7f1c3ecfc700 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2024-05-03T21:10:37.964 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:10:37 smithi067 podman[29600]: 2024-05-03 21:10:37.76210047 +0000 UTC m=+0.325846668 container died ca633300b696329ffa31aa80ae329686f23d76f7f3f808904362bdd185933d53 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9a047fe7599a549042d06a3fb5b6813a203eabea, name=ceph-713bc57a-0991-11ef-bc97-c7b262605968-mon-a, org.label-schema.build-date=20240430, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, ceph=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, RELEASE=wip-rishabh-testing-20240501.193033-9a047fe, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0) 2024-05-03T21:10:38.221 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:10:38 smithi067 podman[29600]: 2024-05-03 21:10:38.070145952 +0000 UTC m=+0.633892140 container cleanup ca633300b696329ffa31aa80ae329686f23d76f7f3f808904362bdd185933d53 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9a047fe7599a549042d06a3fb5b6813a203eabea, name=ceph-713bc57a-0991-11ef-bc97-c7b262605968-mon-a, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-rishabh-testing-20240501.193033-9a047fe, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240430) 2024-05-03T21:10:38.221 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:10:38 smithi067 bash[29600]: ceph-713bc57a-0991-11ef-bc97-c7b262605968-mon-a 2024-05-03T21:10:38.721 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:10:38 smithi067 podman[29614]: 2024-05-03 21:10:38.287817724 +0000 UTC m=+0.525445467 container remove ca633300b696329ffa31aa80ae329686f23d76f7f3f808904362bdd185933d53 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9a047fe7599a549042d06a3fb5b6813a203eabea, name=ceph-713bc57a-0991-11ef-bc97-c7b262605968-mon-a, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_CLEAN=True, RELEASE=wip-rishabh-testing-20240501.193033-9a047fe, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240430, CEPH_POINT_RELEASE=) 2024-05-03T21:10:38.721 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:10:38 smithi067 systemd[1]: ceph-713bc57a-0991-11ef-bc97-c7b262605968@mon.a.service: Deactivated successfully. 2024-05-03T21:10:38.721 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:10:38 smithi067 systemd[1]: Stopped Ceph mon.a for 713bc57a-0991-11ef-bc97-c7b262605968. 2024-05-03T21:10:38.721 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:10:38 smithi067 systemd[1]: Starting Ceph mon.a for 713bc57a-0991-11ef-bc97-c7b262605968... 2024-05-03T21:10:39.003 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:10:38 smithi067 podman[29711]: 2024-05-03 21:10:38.64446767 +0000 UTC m=+0.029439235 image pull ee4d846202496ba6096593e8e8becc0700a5eb7ebd95536031187f5f337d4012 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9a047fe7599a549042d06a3fb5b6813a203eabea 2024-05-03T21:10:39.003 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:10:38 smithi067 podman[29711]: 2024-05-03 21:10:38.763339731 +0000 UTC m=+0.148311269 container create a7aeb235f8119c2c2a3a62abe67c53ebef7f232f2064adc35acb1147ad41d33f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9a047fe7599a549042d06a3fb5b6813a203eabea, name=ceph-713bc57a-0991-11ef-bc97-c7b262605968-mon-a, org.label-schema.license=GPLv2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240430, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, RELEASE=wip-rishabh-testing-20240501.193033-9a047fe, ceph=True) 2024-05-03T21:10:39.003 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:10:38 smithi067 podman[29711]: 2024-05-03 21:10:38.974452889 +0000 UTC m=+0.359424434 container init a7aeb235f8119c2c2a3a62abe67c53ebef7f232f2064adc35acb1147ad41d33f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9a047fe7599a549042d06a3fb5b6813a203eabea, name=ceph-713bc57a-0991-11ef-bc97-c7b262605968-mon-a, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, RELEASE=wip-rishabh-testing-20240501.193033-9a047fe, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.build-date=20240430, ceph=True, org.label-schema.schema-version=1.0) 2024-05-03T21:10:39.003 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:10:38 smithi067 podman[29711]: 2024-05-03 21:10:38.977659248 +0000 UTC m=+0.362630787 container start a7aeb235f8119c2c2a3a62abe67c53ebef7f232f2064adc35acb1147ad41d33f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9a047fe7599a549042d06a3fb5b6813a203eabea, name=ceph-713bc57a-0991-11ef-bc97-c7b262605968-mon-a, org.label-schema.build-date=20240430, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_CLEAN=True, org.label-schema.vendor=CentOS, ceph=True, maintainer=Guillaume Abrioux , RELEASE=wip-rishabh-testing-20240501.193033-9a047fe, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2) 2024-05-03T21:10:39.003 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:10:39 smithi067 ceph-mon[29730]: set uid:gid to 167:167 (ceph:ceph) 2024-05-03T21:10:39.003 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:10:39 smithi067 ceph-mon[29730]: ceph version 19.0.0-3450-g9a047fe7 (9a047fe7599a549042d06a3fb5b6813a203eabea) squid (dev), process ceph-mon, pid 2 2024-05-03T21:10:39.003 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:10:39 smithi067 ceph-mon[29730]: pidfile_write: ignore empty --pid-file 2024-05-03T21:10:39.062 INFO:teuthology.orchestra.run.smithi067.stdout:Setting public_network to 172.21.0.0/20 in mon config section 2024-05-03T21:10:39.279 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:10:39 smithi067 ceph-mon[29730]: load: jerasure load: lrc 2024-05-03T21:10:39.279 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:10:39 smithi067 ceph-mon[29730]: rocksdb: RocksDB version: 7.9.2 2024-05-03T21:10:39.279 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:10:39 smithi067 ceph-mon[29730]: rocksdb: Git sha 0 2024-05-03T21:10:39.280 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:10:39 smithi067 ceph-mon[29730]: rocksdb: Compile date 2024-05-01 21:12:03 2024-05-03T21:10:39.280 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:10:39 smithi067 ceph-mon[29730]: rocksdb: DB SUMMARY 2024-05-03T21:10:39.280 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:10:39 smithi067 ceph-mon[29730]: rocksdb: DB Session ID: FTSBO1GETWOEHTG6YIJH 2024-05-03T21:10:39.280 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:10:39 smithi067 ceph-mon[29730]: rocksdb: CURRENT file: CURRENT 2024-05-03T21:10:39.280 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:10:39 smithi067 ceph-mon[29730]: rocksdb: IDENTITY file: IDENTITY 2024-05-03T21:10:39.280 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:10:39 smithi067 ceph-mon[29730]: rocksdb: MANIFEST file: MANIFEST-000010 size: 179 Bytes 2024-05-03T21:10:39.280 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:10:39 smithi067 ceph-mon[29730]: rocksdb: SST files in /var/lib/ceph/mon/ceph-a/store.db dir, Total Num: 1, files: 000008.sst 2024-05-03T21:10:39.280 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:10:39 smithi067 ceph-mon[29730]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-a/store.db: 000009.log size: 106586 ; 2024-05-03T21:10:39.280 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:10:39 smithi067 ceph-mon[29730]: rocksdb: Options.error_if_exists: 0 2024-05-03T21:10:39.280 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:10:39 smithi067 ceph-mon[29730]: rocksdb: Options.create_if_missing: 0 2024-05-03T21:10:39.280 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:10:39 smithi067 ceph-mon[29730]: rocksdb: Options.paranoid_checks: 1 2024-05-03T21:10:39.281 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:10:39 smithi067 ceph-mon[29730]: rocksdb: Options.flush_verify_memtable_count: 1 2024-05-03T21:10:39.281 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:10:39 smithi067 ceph-mon[29730]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-05-03T21:10:39.281 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:10:39 smithi067 ceph-mon[29730]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-05-03T21:10:39.281 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:10:39 smithi067 ceph-mon[29730]: rocksdb: Options.env: 0x56288b9cfcc0 2024-05-03T21:10:39.281 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:10:39 smithi067 ceph-mon[29730]: rocksdb: Options.fs: PosixFileSystem 2024-05-03T21:10:39.281 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:10:39 smithi067 ceph-mon[29730]: rocksdb: Options.info_log: 0x56288c31bda0 2024-05-03T21:10:39.281 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:10:39 smithi067 ceph-mon[29730]: rocksdb: Options.max_file_opening_threads: 16 2024-05-03T21:10:39.281 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:10:39 smithi067 ceph-mon[29730]: rocksdb: Options.statistics: (nil) 2024-05-03T21:10:39.281 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:10:39 smithi067 ceph-mon[29730]: rocksdb: Options.use_fsync: 0 2024-05-03T21:10:39.281 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:10:39 smithi067 ceph-mon[29730]: rocksdb: Options.max_log_file_size: 0 2024-05-03T21:10:39.281 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:10:39 smithi067 ceph-mon[29730]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-05-03T21:10:39.282 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:10:39 smithi067 ceph-mon[29730]: rocksdb: Options.log_file_time_to_roll: 0 2024-05-03T21:10:39.282 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:10:39 smithi067 ceph-mon[29730]: rocksdb: Options.keep_log_file_num: 1000 2024-05-03T21:10:39.282 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:10:39 smithi067 ceph-mon[29730]: rocksdb: Options.recycle_log_file_num: 0 2024-05-03T21:10:39.282 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:10:39 smithi067 ceph-mon[29730]: rocksdb: Options.allow_fallocate: 1 2024-05-03T21:10:39.282 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:10:39 smithi067 ceph-mon[29730]: rocksdb: Options.allow_mmap_reads: 0 2024-05-03T21:10:39.282 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:10:39 smithi067 ceph-mon[29730]: rocksdb: Options.allow_mmap_writes: 0 2024-05-03T21:10:39.282 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:10:39 smithi067 ceph-mon[29730]: rocksdb: Options.use_direct_reads: 0 2024-05-03T21:10:39.282 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:10:39 smithi067 ceph-mon[29730]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-05-03T21:10:39.282 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:10:39 smithi067 ceph-mon[29730]: rocksdb: Options.create_missing_column_families: 0 2024-05-03T21:10:39.282 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:10:39 smithi067 ceph-mon[29730]: rocksdb: Options.db_log_dir: 2024-05-03T21:10:39.282 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:10:39 smithi067 ceph-mon[29730]: rocksdb: Options.wal_dir: 2024-05-03T21:10:39.282 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:10:39 smithi067 ceph-mon[29730]: rocksdb: Options.table_cache_numshardbits: 6 2024-05-03T21:10:39.283 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:10:39 smithi067 ceph-mon[29730]: rocksdb: Options.WAL_ttl_seconds: 0 2024-05-03T21:10:39.283 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:10:39 smithi067 ceph-mon[29730]: rocksdb: Options.WAL_size_limit_MB: 0 2024-05-03T21:10:39.283 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:10:39 smithi067 ceph-mon[29730]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-05-03T21:10:39.283 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:10:39 smithi067 ceph-mon[29730]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-05-03T21:10:39.283 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:10:39 smithi067 ceph-mon[29730]: rocksdb: Options.is_fd_close_on_exec: 1 2024-05-03T21:10:39.283 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:10:39 smithi067 ceph-mon[29730]: rocksdb: Options.advise_random_on_open: 1 2024-05-03T21:10:39.283 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:10:39 smithi067 ceph-mon[29730]: rocksdb: Options.db_write_buffer_size: 0 2024-05-03T21:10:39.283 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:10:39 smithi067 ceph-mon[29730]: rocksdb: Options.write_buffer_manager: 0x56288c2ed0e0 2024-05-03T21:10:39.283 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:10:39 smithi067 ceph-mon[29730]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-05-03T21:10:39.283 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:10:39 smithi067 ceph-mon[29730]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-05-03T21:10:39.283 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:10:39 smithi067 ceph-mon[29730]: rocksdb: Options.use_adaptive_mutex: 0 2024-05-03T21:10:39.283 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:10:39 smithi067 ceph-mon[29730]: rocksdb: Options.rate_limiter: (nil) 2024-05-03T21:10:39.283 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:10:39 smithi067 ceph-mon[29730]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-05-03T21:10:39.284 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:10:39 smithi067 ceph-mon[29730]: rocksdb: Options.wal_recovery_mode: 2 2024-05-03T21:10:39.284 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:10:39 smithi067 ceph-mon[29730]: rocksdb: Options.enable_thread_tracking: 0 2024-05-03T21:10:39.284 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:10:39 smithi067 ceph-mon[29730]: rocksdb: Options.enable_pipelined_write: 0 2024-05-03T21:10:39.284 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:10:39 smithi067 ceph-mon[29730]: rocksdb: Options.unordered_write: 0 2024-05-03T21:10:39.284 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:10:39 smithi067 ceph-mon[29730]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-05-03T21:10:39.284 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:10:39 smithi067 ceph-mon[29730]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-05-03T21:10:39.284 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:10:39 smithi067 ceph-mon[29730]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-05-03T21:10:39.284 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:10:39 smithi067 ceph-mon[29730]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-05-03T21:10:39.284 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:10:39 smithi067 ceph-mon[29730]: rocksdb: Options.row_cache: None 2024-05-03T21:10:39.284 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:10:39 smithi067 ceph-mon[29730]: rocksdb: Options.wal_filter: None 2024-05-03T21:10:39.284 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:10:39 smithi067 ceph-mon[29730]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-05-03T21:10:39.284 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:10:39 smithi067 ceph-mon[29730]: rocksdb: Options.allow_ingest_behind: 0 2024-05-03T21:10:39.284 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:10:39 smithi067 ceph-mon[29730]: rocksdb: Options.two_write_queues: 0 2024-05-03T21:10:39.285 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:10:39 smithi067 ceph-mon[29730]: rocksdb: Options.manual_wal_flush: 0 2024-05-03T21:10:39.285 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:10:39 smithi067 ceph-mon[29730]: rocksdb: Options.wal_compression: 0 2024-05-03T21:10:39.285 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:10:39 smithi067 ceph-mon[29730]: rocksdb: Options.atomic_flush: 0 2024-05-03T21:10:39.285 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:10:39 smithi067 ceph-mon[29730]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-05-03T21:10:39.285 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:10:39 smithi067 ceph-mon[29730]: rocksdb: Options.persist_stats_to_disk: 0 2024-05-03T21:10:39.285 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:10:39 smithi067 ceph-mon[29730]: rocksdb: Options.write_dbid_to_manifest: 0 2024-05-03T21:10:39.285 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:10:39 smithi067 ceph-mon[29730]: rocksdb: Options.log_readahead_size: 0 2024-05-03T21:10:39.285 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:10:39 smithi067 ceph-mon[29730]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-05-03T21:10:39.285 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:10:39 smithi067 ceph-mon[29730]: rocksdb: Options.best_efforts_recovery: 0 2024-05-03T21:10:39.285 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:10:39 smithi067 ceph-mon[29730]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-05-03T21:10:39.286 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:10:39 smithi067 ceph-mon[29730]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-05-03T21:10:39.286 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:10:39 smithi067 ceph-mon[29730]: rocksdb: Options.allow_data_in_errors: 0 2024-05-03T21:10:39.286 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:10:39 smithi067 ceph-mon[29730]: rocksdb: Options.db_host_id: __hostname__ 2024-05-03T21:10:39.286 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:10:39 smithi067 ceph-mon[29730]: rocksdb: Options.enforce_single_del_contracts: true 2024-05-03T21:10:39.286 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:10:39 smithi067 ceph-mon[29730]: rocksdb: Options.max_background_jobs: 2 2024-05-03T21:10:39.286 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:10:39 smithi067 ceph-mon[29730]: rocksdb: Options.max_background_compactions: -1 2024-05-03T21:10:39.286 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:10:39 smithi067 ceph-mon[29730]: rocksdb: Options.max_subcompactions: 1 2024-05-03T21:10:39.287 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:10:39 smithi067 ceph-mon[29730]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-05-03T21:10:39.288 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:10:39 smithi067 ceph-mon[29730]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-05-03T21:10:39.288 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:10:39 smithi067 ceph-mon[29730]: rocksdb: Options.delayed_write_rate : 16777216 2024-05-03T21:10:39.288 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:10:39 smithi067 ceph-mon[29730]: rocksdb: Options.max_total_wal_size: 0 2024-05-03T21:10:39.288 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:10:39 smithi067 ceph-mon[29730]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-05-03T21:10:39.288 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:10:39 smithi067 ceph-mon[29730]: rocksdb: Options.stats_dump_period_sec: 600 2024-05-03T21:10:39.288 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:10:39 smithi067 ceph-mon[29730]: rocksdb: Options.stats_persist_period_sec: 600 2024-05-03T21:10:39.288 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:10:39 smithi067 ceph-mon[29730]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-05-03T21:10:39.288 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:10:39 smithi067 ceph-mon[29730]: rocksdb: Options.max_open_files: -1 2024-05-03T21:10:39.288 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:10:39 smithi067 ceph-mon[29730]: rocksdb: Options.bytes_per_sync: 0 2024-05-03T21:10:39.288 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:10:39 smithi067 ceph-mon[29730]: rocksdb: Options.wal_bytes_per_sync: 0 2024-05-03T21:10:39.288 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:10:39 smithi067 ceph-mon[29730]: rocksdb: Options.strict_bytes_per_sync: 0 2024-05-03T21:10:39.288 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:10:39 smithi067 ceph-mon[29730]: rocksdb: Options.compaction_readahead_size: 0 2024-05-03T21:10:39.289 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:10:39 smithi067 ceph-mon[29730]: rocksdb: Options.max_background_flushes: -1 2024-05-03T21:10:39.289 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:10:39 smithi067 ceph-mon[29730]: rocksdb: Compression algorithms supported: 2024-05-03T21:10:39.289 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:10:39 smithi067 ceph-mon[29730]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-05-03T21:10:39.289 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:10:39 smithi067 ceph-mon[29730]: rocksdb: kZSTD supported: 0 2024-05-03T21:10:39.289 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:10:39 smithi067 ceph-mon[29730]: rocksdb: kXpressCompression supported: 0 2024-05-03T21:10:39.289 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:10:39 smithi067 ceph-mon[29730]: rocksdb: kLZ4HCCompression supported: 1 2024-05-03T21:10:39.289 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:10:39 smithi067 ceph-mon[29730]: rocksdb: kZlibCompression supported: 1 2024-05-03T21:10:39.289 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:10:39 smithi067 ceph-mon[29730]: rocksdb: kSnappyCompression supported: 1 2024-05-03T21:10:39.289 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:10:39 smithi067 ceph-mon[29730]: rocksdb: kLZ4Compression supported: 1 2024-05-03T21:10:39.289 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:10:39 smithi067 ceph-mon[29730]: rocksdb: kBZip2Compression supported: 0 2024-05-03T21:10:39.289 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:10:39 smithi067 ceph-mon[29730]: rocksdb: Fast CRC32 supported: Supported on x86 2024-05-03T21:10:39.289 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:10:39 smithi067 ceph-mon[29730]: rocksdb: DMutex implementation: pthread_mutex_t 2024-05-03T21:10:39.289 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:10:39 smithi067 ceph-mon[29730]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 2024-05-03T21:10:39.290 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:10:39 smithi067 ceph-mon[29730]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-05-03T21:10:39.290 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:10:39 smithi067 ceph-mon[29730]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-05-03T21:10:39.290 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:10:39 smithi067 ceph-mon[29730]: rocksdb: Options.merge_operator: 2024-05-03T21:10:39.290 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:10:39 smithi067 ceph-mon[29730]: rocksdb: Options.compaction_filter: None 2024-05-03T21:10:39.290 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:10:39 smithi067 ceph-mon[29730]: rocksdb: Options.compaction_filter_factory: None 2024-05-03T21:10:39.290 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:10:39 smithi067 ceph-mon[29730]: rocksdb: Options.sst_partitioner_factory: None 2024-05-03T21:10:39.290 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:10:39 smithi067 ceph-mon[29730]: rocksdb: Options.memtable_factory: SkipListFactory 2024-05-03T21:10:39.290 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:10:39 smithi067 ceph-mon[29730]: rocksdb: Options.table_factory: BlockBasedTable 2024-05-03T21:10:39.290 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:10:39 smithi067 ceph-mon[29730]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x56288c31bf60) 2024-05-03T21:10:39.290 INFO:journalctl@ceph.mon.a.smithi067.stdout: cache_index_and_filter_blocks: 1 2024-05-03T21:10:39.290 INFO:journalctl@ceph.mon.a.smithi067.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-05-03T21:10:39.290 INFO:journalctl@ceph.mon.a.smithi067.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-05-03T21:10:39.291 INFO:journalctl@ceph.mon.a.smithi067.stdout: pin_top_level_index_and_filter: 1 2024-05-03T21:10:39.291 INFO:journalctl@ceph.mon.a.smithi067.stdout: index_type: 0 2024-05-03T21:10:39.291 INFO:journalctl@ceph.mon.a.smithi067.stdout: data_block_index_type: 0 2024-05-03T21:10:39.291 INFO:journalctl@ceph.mon.a.smithi067.stdout: index_shortening: 1 2024-05-03T21:10:39.291 INFO:journalctl@ceph.mon.a.smithi067.stdout: data_block_hash_table_util_ratio: 0.750000 2024-05-03T21:10:39.291 INFO:journalctl@ceph.mon.a.smithi067.stdout: checksum: 4 2024-05-03T21:10:39.291 INFO:journalctl@ceph.mon.a.smithi067.stdout: no_block_cache: 0 2024-05-03T21:10:39.291 INFO:journalctl@ceph.mon.a.smithi067.stdout: block_cache: 0x56288c31f1f0 2024-05-03T21:10:39.291 INFO:journalctl@ceph.mon.a.smithi067.stdout: block_cache_name: BinnedLRUCache 2024-05-03T21:10:39.291 INFO:journalctl@ceph.mon.a.smithi067.stdout: block_cache_options: 2024-05-03T21:10:39.291 INFO:journalctl@ceph.mon.a.smithi067.stdout: capacity : 536870912 2024-05-03T21:10:39.291 INFO:journalctl@ceph.mon.a.smithi067.stdout: num_shard_bits : 4 2024-05-03T21:10:39.291 INFO:journalctl@ceph.mon.a.smithi067.stdout: strict_capacity_limit : 0 2024-05-03T21:10:39.292 INFO:journalctl@ceph.mon.a.smithi067.stdout: high_pri_pool_ratio: 0.000 2024-05-03T21:10:39.292 INFO:journalctl@ceph.mon.a.smithi067.stdout: block_cache_compressed: (nil) 2024-05-03T21:10:39.292 INFO:journalctl@ceph.mon.a.smithi067.stdout: persistent_cache: (nil) 2024-05-03T21:10:39.292 INFO:journalctl@ceph.mon.a.smithi067.stdout: block_size: 4096 2024-05-03T21:10:39.292 INFO:journalctl@ceph.mon.a.smithi067.stdout: block_size_deviation: 10 2024-05-03T21:10:39.292 INFO:journalctl@ceph.mon.a.smithi067.stdout: block_restart_interval: 16 2024-05-03T21:10:39.292 INFO:journalctl@ceph.mon.a.smithi067.stdout: index_block_restart_interval: 1 2024-05-03T21:10:39.292 INFO:journalctl@ceph.mon.a.smithi067.stdout: metadata_block_size: 4096 2024-05-03T21:10:39.292 INFO:journalctl@ceph.mon.a.smithi067.stdout: partition_filters: 0 2024-05-03T21:10:39.292 INFO:journalctl@ceph.mon.a.smithi067.stdout: use_delta_encoding: 1 2024-05-03T21:10:39.292 INFO:journalctl@ceph.mon.a.smithi067.stdout: filter_policy: bloomfilter 2024-05-03T21:10:39.292 INFO:journalctl@ceph.mon.a.smithi067.stdout: whole_key_filtering: 1 2024-05-03T21:10:39.293 INFO:journalctl@ceph.mon.a.smithi067.stdout: verify_compression: 0 2024-05-03T21:10:39.293 INFO:journalctl@ceph.mon.a.smithi067.stdout: read_amp_bytes_per_bit: 0 2024-05-03T21:10:39.293 INFO:journalctl@ceph.mon.a.smithi067.stdout: format_version: 5 2024-05-03T21:10:39.293 INFO:journalctl@ceph.mon.a.smithi067.stdout: enable_index_compression: 1 2024-05-03T21:10:39.293 INFO:journalctl@ceph.mon.a.smithi067.stdout: block_align: 0 2024-05-03T21:10:39.293 INFO:journalctl@ceph.mon.a.smithi067.stdout: max_auto_readahead_size: 262144 2024-05-03T21:10:39.293 INFO:journalctl@ceph.mon.a.smithi067.stdout: prepopulate_block_cache: 0 2024-05-03T21:10:39.293 INFO:journalctl@ceph.mon.a.smithi067.stdout: initial_auto_readahead_size: 8192 2024-05-03T21:10:39.293 INFO:journalctl@ceph.mon.a.smithi067.stdout: num_file_reads_for_auto_readahead: 2 2024-05-03T21:10:39.293 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:10:39 smithi067 ceph-mon[29730]: rocksdb: Options.write_buffer_size: 33554432 2024-05-03T21:10:39.293 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:10:39 smithi067 ceph-mon[29730]: rocksdb: Options.max_write_buffer_number: 2 2024-05-03T21:10:39.293 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:10:39 smithi067 ceph-mon[29730]: rocksdb: Options.compression: NoCompression 2024-05-03T21:10:39.293 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:10:39 smithi067 ceph-mon[29730]: rocksdb: Options.bottommost_compression: Disabled 2024-05-03T21:10:39.294 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:10:39 smithi067 ceph-mon[29730]: rocksdb: Options.prefix_extractor: nullptr 2024-05-03T21:10:39.294 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:10:39 smithi067 ceph-mon[29730]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-05-03T21:10:39.294 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:10:39 smithi067 ceph-mon[29730]: rocksdb: Options.num_levels: 7 2024-05-03T21:10:39.295 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:10:39 smithi067 ceph-mon[29730]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-05-03T21:10:39.295 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:10:39 smithi067 ceph-mon[29730]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-05-03T21:10:39.295 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:10:39 smithi067 ceph-mon[29730]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-05-03T21:10:39.295 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:10:39 smithi067 ceph-mon[29730]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-05-03T21:10:39.296 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:10:39 smithi067 ceph-mon[29730]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-05-03T21:10:39.296 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:10:39 smithi067 ceph-mon[29730]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-05-03T21:10:39.296 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:10:39 smithi067 ceph-mon[29730]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-05-03T21:10:39.296 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:10:39 smithi067 ceph-mon[29730]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-05-03T21:10:39.296 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:10:39 smithi067 ceph-mon[29730]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-05-03T21:10:39.296 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:10:39 smithi067 ceph-mon[29730]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-05-03T21:10:39.296 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:10:39 smithi067 ceph-mon[29730]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-05-03T21:10:39.296 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:10:39 smithi067 ceph-mon[29730]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-05-03T21:10:39.296 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:10:39 smithi067 ceph-mon[29730]: rocksdb: Options.compression_opts.window_bits: -14 2024-05-03T21:10:39.296 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:10:39 smithi067 ceph-mon[29730]: rocksdb: Options.compression_opts.level: 32767 2024-05-03T21:10:39.296 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:10:39 smithi067 ceph-mon[29730]: rocksdb: Options.compression_opts.strategy: 0 2024-05-03T21:10:39.296 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:10:39 smithi067 ceph-mon[29730]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-05-03T21:10:39.297 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:10:39 smithi067 ceph-mon[29730]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-05-03T21:10:39.297 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:10:39 smithi067 ceph-mon[29730]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-05-03T21:10:39.297 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:10:39 smithi067 ceph-mon[29730]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-05-03T21:10:39.297 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:10:39 smithi067 ceph-mon[29730]: rocksdb: Options.compression_opts.enabled: false 2024-05-03T21:10:39.297 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:10:39 smithi067 ceph-mon[29730]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-05-03T21:10:39.297 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:10:39 smithi067 ceph-mon[29730]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-05-03T21:10:39.297 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:10:39 smithi067 ceph-mon[29730]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-05-03T21:10:39.297 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:10:39 smithi067 ceph-mon[29730]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-05-03T21:10:39.297 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:10:39 smithi067 ceph-mon[29730]: rocksdb: Options.target_file_size_base: 67108864 2024-05-03T21:10:39.297 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:10:39 smithi067 ceph-mon[29730]: rocksdb: Options.target_file_size_multiplier: 1 2024-05-03T21:10:39.297 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:10:39 smithi067 ceph-mon[29730]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-05-03T21:10:39.297 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:10:39 smithi067 ceph-mon[29730]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-05-03T21:10:39.297 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:10:39 smithi067 ceph-mon[29730]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-05-03T21:10:39.298 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:10:39 smithi067 ceph-mon[29730]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-05-03T21:10:39.298 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:10:39 smithi067 ceph-mon[29730]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-05-03T21:10:39.298 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:10:39 smithi067 ceph-mon[29730]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-05-03T21:10:39.298 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:10:39 smithi067 ceph-mon[29730]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-05-03T21:10:39.298 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:10:39 smithi067 ceph-mon[29730]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-05-03T21:10:39.298 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:10:39 smithi067 ceph-mon[29730]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-05-03T21:10:39.298 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:10:39 smithi067 ceph-mon[29730]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-05-03T21:10:39.298 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:10:39 smithi067 ceph-mon[29730]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-05-03T21:10:39.298 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:10:39 smithi067 ceph-mon[29730]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-05-03T21:10:39.298 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:10:39 smithi067 ceph-mon[29730]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-05-03T21:10:39.298 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:10:39 smithi067 ceph-mon[29730]: rocksdb: Options.arena_block_size: 1048576 2024-05-03T21:10:39.298 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:10:39 smithi067 ceph-mon[29730]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-05-03T21:10:39.299 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:10:39 smithi067 ceph-mon[29730]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-05-03T21:10:39.299 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:10:39 smithi067 ceph-mon[29730]: rocksdb: Options.disable_auto_compactions: 0 2024-05-03T21:10:39.299 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:10:39 smithi067 ceph-mon[29730]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-05-03T21:10:39.299 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:10:39 smithi067 ceph-mon[29730]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-05-03T21:10:39.299 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:10:39 smithi067 ceph-mon[29730]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-05-03T21:10:39.299 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:10:39 smithi067 ceph-mon[29730]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-05-03T21:10:39.299 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:10:39 smithi067 ceph-mon[29730]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-05-03T21:10:39.299 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:10:39 smithi067 ceph-mon[29730]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-05-03T21:10:39.299 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:10:39 smithi067 ceph-mon[29730]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-05-03T21:10:39.299 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:10:39 smithi067 ceph-mon[29730]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-05-03T21:10:39.299 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:10:39 smithi067 ceph-mon[29730]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-05-03T21:10:39.299 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:10:39 smithi067 ceph-mon[29730]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-05-03T21:10:39.299 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:10:39 smithi067 ceph-mon[29730]: rocksdb: Options.table_properties_collectors: CompactOnDeletionCollector (Sliding window size = 32768 Deletion trigger = 16384 Deletion ratio = 0); 2024-05-03T21:10:39.300 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:10:39 smithi067 ceph-mon[29730]: rocksdb: Options.inplace_update_support: 0 2024-05-03T21:10:39.300 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:10:39 smithi067 ceph-mon[29730]: rocksdb: Options.inplace_update_num_locks: 10000 2024-05-03T21:10:39.300 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:10:39 smithi067 ceph-mon[29730]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-05-03T21:10:39.300 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:10:39 smithi067 ceph-mon[29730]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-05-03T21:10:39.300 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:10:39 smithi067 ceph-mon[29730]: rocksdb: Options.memtable_huge_page_size: 0 2024-05-03T21:10:39.300 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:10:39 smithi067 ceph-mon[29730]: rocksdb: Options.bloom_locality: 0 2024-05-03T21:10:39.300 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:10:39 smithi067 ceph-mon[29730]: rocksdb: Options.max_successive_merges: 0 2024-05-03T21:10:39.300 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:10:39 smithi067 ceph-mon[29730]: rocksdb: Options.optimize_filters_for_hits: 0 2024-05-03T21:10:39.300 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:10:39 smithi067 ceph-mon[29730]: rocksdb: Options.paranoid_file_checks: 0 2024-05-03T21:10:39.300 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:10:39 smithi067 ceph-mon[29730]: rocksdb: Options.force_consistency_checks: 1 2024-05-03T21:10:39.300 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:10:39 smithi067 ceph-mon[29730]: rocksdb: Options.report_bg_io_stats: 0 2024-05-03T21:10:39.300 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:10:39 smithi067 ceph-mon[29730]: rocksdb: Options.ttl: 2592000 2024-05-03T21:10:39.300 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:10:39 smithi067 ceph-mon[29730]: rocksdb: Options.periodic_compaction_seconds: 0 2024-05-03T21:10:39.300 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:10:39 smithi067 ceph-mon[29730]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-05-03T21:10:39.301 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:10:39 smithi067 ceph-mon[29730]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-05-03T21:10:39.301 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:10:39 smithi067 ceph-mon[29730]: rocksdb: Options.enable_blob_files: false 2024-05-03T21:10:39.301 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:10:39 smithi067 ceph-mon[29730]: rocksdb: Options.min_blob_size: 0 2024-05-03T21:10:39.301 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:10:39 smithi067 ceph-mon[29730]: rocksdb: Options.blob_file_size: 268435456 2024-05-03T21:10:39.301 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:10:39 smithi067 ceph-mon[29730]: rocksdb: Options.blob_compression_type: NoCompression 2024-05-03T21:10:39.301 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:10:39 smithi067 ceph-mon[29730]: rocksdb: Options.enable_blob_garbage_collection: false 2024-05-03T21:10:39.301 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:10:39 smithi067 ceph-mon[29730]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-05-03T21:10:39.301 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:10:39 smithi067 ceph-mon[29730]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-05-03T21:10:39.301 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:10:39 smithi067 ceph-mon[29730]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-05-03T21:10:39.301 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:10:39 smithi067 ceph-mon[29730]: rocksdb: Options.blob_file_starting_level: 0 2024-05-03T21:10:39.301 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:10:39 smithi067 ceph-mon[29730]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-05-03T21:10:39.301 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:10:39 smithi067 ceph-mon[29730]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 succeeded,manifest_file_number is 10, next_file_number is 12, last_sequence is 5, log_number is 5,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 5 2024-05-03T21:10:39.301 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:10:39 smithi067 ceph-mon[29730]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 5 2024-05-03T21:10:39.302 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:10:39 smithi067 ceph-mon[29730]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: a8fb6e15-65ce-4075-93d1-64d7bed5bfe9 2024-05-03T21:10:39.302 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:10:39 smithi067 ceph-mon[29730]: rocksdb: EVENT_LOG_v1 {"time_micros": 1714770639007481, "job": 1, "event": "recovery_started", "wal_files": [9]} 2024-05-03T21:10:39.302 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:10:39 smithi067 ceph-mon[29730]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #9 mode 2 2024-05-03T21:10:39.302 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:10:39 smithi067 ceph-mon[29730]: rocksdb: EVENT_LOG_v1 {"time_micros": 1714770639008500, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 13, "file_size": 101567, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 8, "largest_seqno": 332, "table_properties": {"data_size": 99463, "index_size": 296, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 773, "raw_key_size": 14552, "raw_average_key_size": 50, "raw_value_size": 91569, "raw_average_value_size": 314, "num_data_blocks": 12, "num_entries": 291, "num_filter_entries": 291, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[CompactOnDeletionCollector]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1714770639, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "a8fb6e15-65ce-4075-93d1-64d7bed5bfe9", "db_session_id": "FTSBO1GETWOEHTG6YIJH", "orig_file_number": 13, "seqno_to_time_mapping": "N/A"}} 2024-05-03T21:10:39.302 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:10:39 smithi067 ceph-mon[29730]: rocksdb: EVENT_LOG_v1 {"time_micros": 1714770639008584, "job": 1, "event": "recovery_finished"} 2024-05-03T21:10:39.302 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:10:39 smithi067 ceph-mon[29730]: rocksdb: [db/version_set.cc:5047] Creating manifest 15 2024-05-03T21:10:39.302 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:10:39 smithi067 ceph-mon[29730]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-a/store.db/000009.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-05-03T21:10:39.302 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:10:39 smithi067 ceph-mon[29730]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x56288c43c000 2024-05-03T21:10:39.302 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:10:39 smithi067 ceph-mon[29730]: rocksdb: DB pointer 0x56288c428000 2024-05-03T21:10:39.302 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:10:39 smithi067 ceph-mon[29730]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-05-03T21:10:39.302 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:10:39 smithi067 ceph-mon[29730]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-05-03T21:10:39.302 INFO:journalctl@ceph.mon.a.smithi067.stdout: ** DB Stats ** 2024-05-03T21:10:39.303 INFO:journalctl@ceph.mon.a.smithi067.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-05-03T21:10:39.303 INFO:journalctl@ceph.mon.a.smithi067.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-05-03T21:10:39.303 INFO:journalctl@ceph.mon.a.smithi067.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-05-03T21:10:39.303 INFO:journalctl@ceph.mon.a.smithi067.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-05-03T21:10:39.303 INFO:journalctl@ceph.mon.a.smithi067.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-05-03T21:10:39.303 INFO:journalctl@ceph.mon.a.smithi067.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-05-03T21:10:39.303 INFO:journalctl@ceph.mon.a.smithi067.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-05-03T21:10:39.303 INFO:journalctl@ceph.mon.a.smithi067.stdout: 2024-05-03T21:10:39.303 INFO:journalctl@ceph.mon.a.smithi067.stdout: ** Compaction Stats [default] ** 2024-05-03T21:10:39.303 INFO:journalctl@ceph.mon.a.smithi067.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-05-03T21:10:39.303 INFO:journalctl@ceph.mon.a.smithi067.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-05-03T21:10:39.303 INFO:journalctl@ceph.mon.a.smithi067.stdout: L0 2/0 101.05 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 161.4 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-05-03T21:10:39.303 INFO:journalctl@ceph.mon.a.smithi067.stdout: Sum 2/0 101.05 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 161.4 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-05-03T21:10:39.304 INFO:journalctl@ceph.mon.a.smithi067.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 161.4 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-05-03T21:10:39.304 INFO:journalctl@ceph.mon.a.smithi067.stdout: 2024-05-03T21:10:39.304 INFO:journalctl@ceph.mon.a.smithi067.stdout: ** Compaction Stats [default] ** 2024-05-03T21:10:39.304 INFO:journalctl@ceph.mon.a.smithi067.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-05-03T21:10:39.304 INFO:journalctl@ceph.mon.a.smithi067.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-05-03T21:10:39.304 INFO:journalctl@ceph.mon.a.smithi067.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 161.4 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-05-03T21:10:39.304 INFO:journalctl@ceph.mon.a.smithi067.stdout: 2024-05-03T21:10:39.304 INFO:journalctl@ceph.mon.a.smithi067.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-05-03T21:10:39.304 INFO:journalctl@ceph.mon.a.smithi067.stdout: 2024-05-03T21:10:39.304 INFO:journalctl@ceph.mon.a.smithi067.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-05-03T21:10:39.304 INFO:journalctl@ceph.mon.a.smithi067.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-05-03T21:10:39.304 INFO:journalctl@ceph.mon.a.smithi067.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-05-03T21:10:39.304 INFO:journalctl@ceph.mon.a.smithi067.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-05-03T21:10:39.305 INFO:journalctl@ceph.mon.a.smithi067.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-05-03T21:10:39.305 INFO:journalctl@ceph.mon.a.smithi067.stdout: AddFile(Keys): cumulative 0, interval 0 2024-05-03T21:10:39.305 INFO:journalctl@ceph.mon.a.smithi067.stdout: Cumulative compaction: 0.00 GB write, 32.03 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-05-03T21:10:39.305 INFO:journalctl@ceph.mon.a.smithi067.stdout: Interval compaction: 0.00 GB write, 32.03 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-05-03T21:10:39.305 INFO:journalctl@ceph.mon.a.smithi067.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-05-03T21:10:39.305 INFO:journalctl@ceph.mon.a.smithi067.stdout: Block cache BinnedLRUCache@0x56288c31f1f0#2 capacity: 512.00 MB usage: 1.52 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 1e-05 secs_since: 0 2024-05-03T21:10:39.305 INFO:journalctl@ceph.mon.a.smithi067.stdout: Block cache entry stats(count,size,portion): FilterBlock(2,1.02 KB,0.000193715%) IndexBlock(2,0.50 KB,9.53674e-05%) Misc(1,0.00 KB,0%) 2024-05-03T21:10:39.305 INFO:journalctl@ceph.mon.a.smithi067.stdout: 2024-05-03T21:10:39.305 INFO:journalctl@ceph.mon.a.smithi067.stdout: ** File Read Latency Histogram By Level [default] ** 2024-05-03T21:10:39.305 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:10:39 smithi067 ceph-mon[29730]: starting mon.a rank 0 at public addrs [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] at bind addrs [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon_data /var/lib/ceph/mon/ceph-a fsid 713bc57a-0991-11ef-bc97-c7b262605968 2024-05-03T21:10:39.305 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:10:39 smithi067 ceph-mon[29730]: mon.a@-1(???) e1 preinit fsid 713bc57a-0991-11ef-bc97-c7b262605968 2024-05-03T21:10:39.305 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:10:39 smithi067 ceph-mon[29730]: mon.a@-1(???).mds e1 new map 2024-05-03T21:10:39.305 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:10:39 smithi067 ceph-mon[29730]: mon.a@-1(???).mds e1 print_map 2024-05-03T21:10:39.306 INFO:journalctl@ceph.mon.a.smithi067.stdout: e1 2024-05-03T21:10:39.306 INFO:journalctl@ceph.mon.a.smithi067.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-05-03T21:10:39.306 INFO:journalctl@ceph.mon.a.smithi067.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-05-03T21:10:39.306 INFO:journalctl@ceph.mon.a.smithi067.stdout: legacy client fscid: -1 2024-05-03T21:10:39.306 INFO:journalctl@ceph.mon.a.smithi067.stdout: 2024-05-03T21:10:39.306 INFO:journalctl@ceph.mon.a.smithi067.stdout: No filesystems configured 2024-05-03T21:10:39.306 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:10:39 smithi067 ceph-mon[29730]: mon.a@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2024-05-03T21:10:39.306 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:10:39 smithi067 ceph-mon[29730]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-05-03T21:10:39.306 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:10:39 smithi067 ceph-mon[29730]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-05-03T21:10:39.306 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:10:39 smithi067 ceph-mon[29730]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-05-03T21:10:39.306 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:10:39 smithi067 ceph-mon[29730]: mon.a@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2024-05-03T21:10:39.306 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:10:39 smithi067 bash[29711]: a7aeb235f8119c2c2a3a62abe67c53ebef7f232f2064adc35acb1147ad41d33f 2024-05-03T21:10:39.307 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:10:39 smithi067 systemd[1]: Started Ceph mon.a for 713bc57a-0991-11ef-bc97-c7b262605968. 2024-05-03T21:10:40.299 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:10:40 smithi067 ceph-mon[29730]: mon.a is new leader, mons a in quorum (ranks 0) 2024-05-03T21:10:40.299 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:10:40 smithi067 ceph-mon[29730]: monmap epoch 1 2024-05-03T21:10:40.299 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:10:40 smithi067 ceph-mon[29730]: fsid 713bc57a-0991-11ef-bc97-c7b262605968 2024-05-03T21:10:40.299 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:10:40 smithi067 ceph-mon[29730]: last_changed 2024-05-03T21:10:31.243292+0000 2024-05-03T21:10:40.299 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:10:40 smithi067 ceph-mon[29730]: created 2024-05-03T21:10:31.243292+0000 2024-05-03T21:10:40.299 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:10:40 smithi067 ceph-mon[29730]: min_mon_release 19 (squid) 2024-05-03T21:10:40.299 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:10:40 smithi067 ceph-mon[29730]: election_strategy: 1 2024-05-03T21:10:40.299 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:10:40 smithi067 ceph-mon[29730]: 0: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.a 2024-05-03T21:10:40.299 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:10:40 smithi067 ceph-mon[29730]: fsmap 2024-05-03T21:10:40.299 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:10:40 smithi067 ceph-mon[29730]: osdmap e1: 0 total, 0 up, 0 in 2024-05-03T21:10:40.299 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:10:40 smithi067 ceph-mon[29730]: mgrmap e1: no daemons active 2024-05-03T21:10:40.300 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:10:40 smithi067 ceph-mon[29730]: from='client.? 172.21.15.67:0/2621004565' entity='client.admin' 2024-05-03T21:10:40.302 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:10:39.676+0000 7f689bbf9700 1 Processor -- start 2024-05-03T21:10:40.306 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:10:39.676+0000 7f689bbf9700 1 -- start start 2024-05-03T21:10:40.306 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:10:39.676+0000 7f689bbf9700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f689410a800 0x7f689410ac20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:10:40.306 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:10:39.676+0000 7f689bbf9700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f689410b1f0 con 0x7f689410a800 2024-05-03T21:10:40.306 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:10:39.677+0000 7f6899995700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f689410a800 0x7f689410ac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:10:40.307 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:10:39.677+0000 7f6899995700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f689410a800 0x7f689410ac20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:59630/0 (socket says 172.21.15.67:59630) 2024-05-03T21:10:40.307 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:10:39.677+0000 7f6899995700 1 -- 172.21.15.67:0/1181880509 learned_addr learned my addr 172.21.15.67:0/1181880509 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-03T21:10:40.307 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:10:39.677+0000 7f6899995700 1 -- 172.21.15.67:0/1181880509 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f689410b370 con 0x7f689410a800 2024-05-03T21:10:40.307 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:10:39.677+0000 7f6899995700 1 --2- 172.21.15.67:0/1181880509 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f689410a800 0x7f689410ac20 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f6884009e80 tx=0x7f688400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e077a3981b4fca52 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T21:10:40.307 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:10:39.677+0000 7f6898993700 1 -- 172.21.15.67:0/1181880509 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6884017070 con 0x7f689410a800 2024-05-03T21:10:40.307 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:10:39.677+0000 7f6898993700 1 -- 172.21.15.67:0/1181880509 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(28 keys) ==== 1139+0+0 (secure 0 0 0) 0x7f688400b8c0 con 0x7f689410a800 2024-05-03T21:10:40.307 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:10:39.678+0000 7f6898993700 1 -- 172.21.15.67:0/1181880509 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6884004620 con 0x7f689410a800 2024-05-03T21:10:40.307 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:10:39.678+0000 7f689bbf9700 1 -- 172.21.15.67:0/1181880509 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f689410a800 msgr2=0x7f689410ac20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:10:40.307 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:10:39.678+0000 7f689bbf9700 1 --2- 172.21.15.67:0/1181880509 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f689410a800 0x7f689410ac20 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f6884009e80 tx=0x7f688400b330 comp rx=0 tx=0).stop 2024-05-03T21:10:40.307 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:10:39.678+0000 7f689bbf9700 1 -- 172.21.15.67:0/1181880509 shutdown_connections 2024-05-03T21:10:40.307 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:10:39.678+0000 7f689bbf9700 1 --2- 172.21.15.67:0/1181880509 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f689410a800 0x7f689410ac20 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:10:40.307 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:10:39.678+0000 7f689bbf9700 1 -- 172.21.15.67:0/1181880509 >> 172.21.15.67:0/1181880509 conn(0x7f6894102f40 msgr2=0x7f6894105360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T21:10:40.307 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:10:39.678+0000 7f689bbf9700 1 -- 172.21.15.67:0/1181880509 shutdown_connections 2024-05-03T21:10:40.307 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:10:39.678+0000 7f689bbf9700 1 -- 172.21.15.67:0/1181880509 wait complete. 2024-05-03T21:10:40.308 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:10:39.679+0000 7f689bbf9700 1 Processor -- start 2024-05-03T21:10:40.308 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:10:39.679+0000 7f689bbf9700 1 -- start start 2024-05-03T21:10:40.308 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:10:39.679+0000 7f689bbf9700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f689410a800 0x7f689419d240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:10:40.308 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:10:39.679+0000 7f689bbf9700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f68941080d0 con 0x7f689410a800 2024-05-03T21:10:40.308 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:10:39.679+0000 7f6899995700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f689410a800 0x7f689419d240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:10:40.308 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:10:39.679+0000 7f6899995700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f689410a800 0x7f689419d240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:59644/0 (socket says 172.21.15.67:59644) 2024-05-03T21:10:40.308 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:10:39.679+0000 7f6899995700 1 -- 172.21.15.67:0/2621004565 learned_addr learned my addr 172.21.15.67:0/2621004565 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-03T21:10:40.308 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:10:39.679+0000 7f6899995700 1 -- 172.21.15.67:0/2621004565 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f689419d780 con 0x7f689410a800 2024-05-03T21:10:40.308 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:10:39.680+0000 7f6899995700 1 --2- 172.21.15.67:0/2621004565 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f689410a800 0x7f689419d240 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f688400b7b0 tx=0x7f6884003d00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T21:10:40.308 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:10:39.680+0000 7f6892ffd700 1 -- 172.21.15.67:0/2621004565 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6884021020 con 0x7f689410a800 2024-05-03T21:10:40.308 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:10:39.680+0000 7f6892ffd700 1 -- 172.21.15.67:0/2621004565 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(28 keys) ==== 1139+0+0 (secure 0 0 0) 0x7f688400bde0 con 0x7f689410a800 2024-05-03T21:10:40.308 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:10:39.680+0000 7f689bbf9700 1 -- 172.21.15.67:0/2621004565 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f689419da10 con 0x7f689410a800 2024-05-03T21:10:40.308 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:10:39.680+0000 7f6892ffd700 1 -- 172.21.15.67:0/2621004565 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6884017040 con 0x7f689410a800 2024-05-03T21:10:40.308 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:10:39.680+0000 7f689bbf9700 1 -- 172.21.15.67:0/2621004565 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f68941a0700 con 0x7f689410a800 2024-05-03T21:10:40.308 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:10:39.680+0000 7f6892ffd700 1 -- 172.21.15.67:0/2621004565 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f6884024020 con 0x7f689410a800 2024-05-03T21:10:40.309 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:10:39.680+0000 7f6892ffd700 1 -- 172.21.15.67:0/2621004565 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f688400f050 con 0x7f689410a800 2024-05-03T21:10:40.309 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:10:39.681+0000 7f689bbf9700 1 -- 172.21.15.67:0/2621004565 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6894064b00 con 0x7f689410a800 2024-05-03T21:10:40.309 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:10:39.683+0000 7f6892ffd700 1 -- 172.21.15.67:0/2621004565 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7f688402b020 con 0x7f689410a800 2024-05-03T21:10:40.309 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:10:39.752+0000 7f689bbf9700 1 -- 172.21.15.67:0/2621004565 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command([{prefix=config set, name=public_network}] v 0) -- 0x7f68941a0970 con 0x7f689410a800 2024-05-03T21:10:40.309 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:10:39.754+0000 7f6892ffd700 1 -- 172.21.15.67:0/2621004565 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{prefix=config set, name=public_network}]=0 v3)=0 v3) ==== 124+0+0 (secure 0 0 0) 0x7f688400b860 con 0x7f689410a800 2024-05-03T21:10:40.309 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:10:39.755+0000 7f689bbf9700 1 -- 172.21.15.67:0/2621004565 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f689410a800 msgr2=0x7f689419d240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:10:40.309 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:10:39.755+0000 7f689bbf9700 1 --2- 172.21.15.67:0/2621004565 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f689410a800 0x7f689419d240 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f688400b7b0 tx=0x7f6884003d00 comp rx=0 tx=0).stop 2024-05-03T21:10:40.309 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:10:39.755+0000 7f689bbf9700 1 -- 172.21.15.67:0/2621004565 shutdown_connections 2024-05-03T21:10:40.309 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:10:39.755+0000 7f689bbf9700 1 --2- 172.21.15.67:0/2621004565 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f689410a800 0x7f689419d240 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:10:40.309 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:10:39.755+0000 7f689bbf9700 1 -- 172.21.15.67:0/2621004565 >> 172.21.15.67:0/2621004565 conn(0x7f6894102f40 msgr2=0x7f6894103b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T21:10:40.309 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:10:39.755+0000 7f689bbf9700 1 -- 172.21.15.67:0/2621004565 shutdown_connections 2024-05-03T21:10:40.309 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:10:39.755+0000 7f689bbf9700 1 -- 172.21.15.67:0/2621004565 wait complete. 2024-05-03T21:10:40.309 INFO:teuthology.orchestra.run.smithi067.stdout:Wrote config to /etc/ceph/ceph.conf 2024-05-03T21:10:40.348 INFO:teuthology.orchestra.run.smithi067.stdout:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2024-05-03T21:10:40.348 INFO:teuthology.orchestra.run.smithi067.stdout:Creating mgr... 2024-05-03T21:10:40.349 INFO:teuthology.orchestra.run.smithi067.stdout:Verifying port 0.0.0.0:9283 ... 2024-05-03T21:10:40.349 INFO:teuthology.orchestra.run.smithi067.stdout:Verifying port 0.0.0.0:8765 ... 2024-05-03T21:10:40.598 INFO:teuthology.orchestra.run.smithi067.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-713bc57a-0991-11ef-bc97-c7b262605968@mgr.x 2024-05-03T21:10:40.598 INFO:teuthology.orchestra.run.smithi067.stdout:systemctl: stderr Failed to reset failed state of unit ceph-713bc57a-0991-11ef-bc97-c7b262605968@mgr.x.service: Unit ceph-713bc57a-0991-11ef-bc97-c7b262605968@mgr.x.service not loaded. 2024-05-03T21:10:40.755 INFO:teuthology.orchestra.run.smithi067.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-713bc57a-0991-11ef-bc97-c7b262605968.target.wants/ceph-713bc57a-0991-11ef-bc97-c7b262605968@mgr.x.service → /etc/systemd/system/ceph-713bc57a-0991-11ef-bc97-c7b262605968@.service. 2024-05-03T21:10:40.971 INFO:journalctl@ceph.mgr.x.smithi067.stdout:May 03 21:10:40 smithi067 systemd[1]: Starting Ceph mgr.x for 713bc57a-0991-11ef-bc97-c7b262605968... 2024-05-03T21:10:41.253 INFO:journalctl@ceph.mgr.x.smithi067.stdout:May 03 21:10:40 smithi067 podman[29949]: 2024-05-03 21:10:40.991321103 +0000 UTC m=+0.114896876 container create c7ff7c041cc782e833bd5f17e8627af26a5d2cd4b083a8b7ac36999ce800629b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9a047fe7599a549042d06a3fb5b6813a203eabea, name=ceph-713bc57a-0991-11ef-bc97-c7b262605968-mgr-x, ceph=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, RELEASE=wip-rishabh-testing-20240501.193033-9a047fe, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240430) 2024-05-03T21:10:41.253 INFO:journalctl@ceph.mgr.x.smithi067.stdout:May 03 21:10:40 smithi067 podman[29949]: 2024-05-03 21:10:40.894851112 +0000 UTC m=+0.018426886 image pull ee4d846202496ba6096593e8e8becc0700a5eb7ebd95536031187f5f337d4012 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9a047fe7599a549042d06a3fb5b6813a203eabea 2024-05-03T21:10:41.253 INFO:journalctl@ceph.mgr.x.smithi067.stdout:May 03 21:10:41 smithi067 podman[29949]: 2024-05-03 21:10:41.201797248 +0000 UTC m=+0.325373025 container init c7ff7c041cc782e833bd5f17e8627af26a5d2cd4b083a8b7ac36999ce800629b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9a047fe7599a549042d06a3fb5b6813a203eabea, name=ceph-713bc57a-0991-11ef-bc97-c7b262605968-mgr-x, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, ceph=True, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240430, org.label-schema.vendor=CentOS, RELEASE=wip-rishabh-testing-20240501.193033-9a047fe) 2024-05-03T21:10:41.253 INFO:journalctl@ceph.mgr.x.smithi067.stdout:May 03 21:10:41 smithi067 podman[29949]: 2024-05-03 21:10:41.204398667 +0000 UTC m=+0.327974441 container start c7ff7c041cc782e833bd5f17e8627af26a5d2cd4b083a8b7ac36999ce800629b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9a047fe7599a549042d06a3fb5b6813a203eabea, name=ceph-713bc57a-0991-11ef-bc97-c7b262605968-mgr-x, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.build-date=20240430, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-rishabh-testing-20240501.193033-9a047fe, ceph=True) 2024-05-03T21:10:41.253 INFO:journalctl@ceph.mgr.x.smithi067.stdout:May 03 21:10:41 smithi067 bash[29949]: c7ff7c041cc782e833bd5f17e8627af26a5d2cd4b083a8b7ac36999ce800629b 2024-05-03T21:10:41.267 INFO:teuthology.orchestra.run.smithi067.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-05-03T21:10:41.268 INFO:teuthology.orchestra.run.smithi067.stdout:systemctl: stdout disabled 2024-05-03T21:10:41.275 INFO:teuthology.orchestra.run.smithi067.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-05-03T21:10:41.275 INFO:teuthology.orchestra.run.smithi067.stdout:systemctl: stdout inactive 2024-05-03T21:10:41.275 INFO:teuthology.orchestra.run.smithi067.stdout:firewalld.service is not enabled 2024-05-03T21:10:41.276 INFO:teuthology.orchestra.run.smithi067.stdout:Not possible to enable service . firewalld.service is not available 2024-05-03T21:10:41.284 INFO:teuthology.orchestra.run.smithi067.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-05-03T21:10:41.284 INFO:teuthology.orchestra.run.smithi067.stdout:systemctl: stdout disabled 2024-05-03T21:10:41.292 INFO:teuthology.orchestra.run.smithi067.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-05-03T21:10:41.292 INFO:teuthology.orchestra.run.smithi067.stdout:systemctl: stdout inactive 2024-05-03T21:10:41.292 INFO:teuthology.orchestra.run.smithi067.stdout:firewalld.service is not enabled 2024-05-03T21:10:41.292 INFO:teuthology.orchestra.run.smithi067.stdout:Not possible to open ports <[9283, 8765]>. firewalld.service is not available 2024-05-03T21:10:41.292 INFO:teuthology.orchestra.run.smithi067.stdout:Waiting for mgr to start... 2024-05-03T21:10:41.292 INFO:teuthology.orchestra.run.smithi067.stdout:Waiting for mgr... 2024-05-03T21:10:41.565 INFO:journalctl@ceph.mgr.x.smithi067.stdout:May 03 21:10:41 smithi067 systemd[1]: Started Ceph mgr.x for 713bc57a-0991-11ef-bc97-c7b262605968. 2024-05-03T21:10:41.565 INFO:journalctl@ceph.mgr.x.smithi067.stdout:May 03 21:10:41 smithi067 ceph-713bc57a-0991-11ef-bc97-c7b262605968-mgr-x[29964]: 2024-05-03T21:10:41.367+0000 7f0f6a9d4200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-05-03T21:10:41.565 INFO:journalctl@ceph.mgr.x.smithi067.stdout:May 03 21:10:41 smithi067 ceph-713bc57a-0991-11ef-bc97-c7b262605968-mgr-x[29964]: 2024-05-03T21:10:41.562+0000 7f0f6a9d4200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-05-03T21:10:41.970 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:10:41 smithi067 ceph-mon[29730]: from='client.? 172.21.15.67:0/1028439948' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-05-03T21:10:41.971 INFO:journalctl@ceph.mgr.x.smithi067.stdout:May 03 21:10:41 smithi067 ceph-713bc57a-0991-11ef-bc97-c7b262605968-mgr-x[29964]: 2024-05-03T21:10:41.671+0000 7f0f6a9d4200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-05-03T21:10:42.342 INFO:journalctl@ceph.mgr.x.smithi067.stdout:May 03 21:10:42 smithi067 ceph-713bc57a-0991-11ef-bc97-c7b262605968-mgr-x[29964]: 2024-05-03T21:10:42.010+0000 7f0f6a9d4200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-05-03T21:10:42.342 INFO:journalctl@ceph.mgr.x.smithi067.stdout:May 03 21:10:42 smithi067 ceph-713bc57a-0991-11ef-bc97-c7b262605968-mgr-x[29964]: 2024-05-03T21:10:42.098+0000 7f0f6a9d4200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-05-03T21:10:42.347 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout 2024-05-03T21:10:42.347 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout { 2024-05-03T21:10:42.347 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "fsid": "713bc57a-0991-11ef-bc97-c7b262605968", 2024-05-03T21:10:42.347 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "health": { 2024-05-03T21:10:42.347 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-05-03T21:10:42.347 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-05-03T21:10:42.347 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-05-03T21:10:42.348 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout }, 2024-05-03T21:10:42.348 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-05-03T21:10:42.348 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-05-03T21:10:42.348 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout 0 2024-05-03T21:10:42.348 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout ], 2024-05-03T21:10:42.348 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-05-03T21:10:42.348 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "a" 2024-05-03T21:10:42.348 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout ], 2024-05-03T21:10:42.348 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "quorum_age": 2, 2024-05-03T21:10:42.348 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "monmap": { 2024-05-03T21:10:42.348 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-05-03T21:10:42.348 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-05-03T21:10:42.348 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-05-03T21:10:42.348 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout }, 2024-05-03T21:10:42.348 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-05-03T21:10:42.348 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-05-03T21:10:42.349 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-05-03T21:10:42.349 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-05-03T21:10:42.349 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-05-03T21:10:42.360 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-05-03T21:10:42.360 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-05-03T21:10:42.360 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-05-03T21:10:42.361 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout }, 2024-05-03T21:10:42.361 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-05-03T21:10:42.361 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-05-03T21:10:42.361 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-05-03T21:10:42.361 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-05-03T21:10:42.361 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-05-03T21:10:42.361 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-05-03T21:10:42.361 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-05-03T21:10:42.361 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-05-03T21:10:42.361 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-05-03T21:10:42.361 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout }, 2024-05-03T21:10:42.362 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-05-03T21:10:42.362 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-05-03T21:10:42.362 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-05-03T21:10:42.362 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-05-03T21:10:42.363 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout }, 2024-05-03T21:10:42.363 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-05-03T21:10:42.363 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "available": false, 2024-05-03T21:10:42.363 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-05-03T21:10:42.363 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "modules": [ 2024-05-03T21:10:42.363 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "iostat", 2024-05-03T21:10:42.363 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "nfs", 2024-05-03T21:10:42.364 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "restful" 2024-05-03T21:10:42.364 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout ], 2024-05-03T21:10:42.364 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "services": {} 2024-05-03T21:10:42.364 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout }, 2024-05-03T21:10:42.364 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-05-03T21:10:42.364 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-05-03T21:10:42.364 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "modified": "2024-05-03T21:10:34.070672+0000", 2024-05-03T21:10:42.364 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "services": {} 2024-05-03T21:10:42.364 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout }, 2024-05-03T21:10:42.364 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-05-03T21:10:42.364 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout } 2024-05-03T21:10:42.364 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:10:41.673+0000 7f800b7f4700 1 Processor -- start 2024-05-03T21:10:42.364 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:10:41.673+0000 7f800b7f4700 1 -- start start 2024-05-03T21:10:42.364 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:10:41.673+0000 7f800b7f4700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8004108110 0x7f8004108530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:10:42.365 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:10:41.673+0000 7f800b7f4700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f8004108b00 con 0x7f8004108110 2024-05-03T21:10:42.365 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:10:41.674+0000 7f8009590700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8004108110 0x7f8004108530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:10:42.365 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:10:41.674+0000 7f8009590700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8004108110 0x7f8004108530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:59686/0 (socket says 172.21.15.67:59686) 2024-05-03T21:10:42.365 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:10:41.674+0000 7f8009590700 1 -- 172.21.15.67:0/1788843825 learned_addr learned my addr 172.21.15.67:0/1788843825 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-03T21:10:42.365 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:10:41.674+0000 7f8009590700 1 -- 172.21.15.67:0/1788843825 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8004109310 con 0x7f8004108110 2024-05-03T21:10:42.365 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:10:41.674+0000 7f8009590700 1 --2- 172.21.15.67:0/1788843825 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8004108110 0x7f8004108530 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f7ff400bb40 tx=0x7f7ff400be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b2db47c4e91f5299 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T21:10:42.365 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:10:41.675+0000 7f7ffbfff700 1 -- 172.21.15.67:0/1788843825 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7ff400dbe0 con 0x7f8004108110 2024-05-03T21:10:42.365 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:10:41.675+0000 7f7ffbfff700 1 -- 172.21.15.67:0/1788843825 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(28 keys) ==== 1139+0+0 (secure 0 0 0) 0x7f7ff4004510 con 0x7f8004108110 2024-05-03T21:10:42.365 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:10:41.675+0000 7f7ffbfff700 1 -- 172.21.15.67:0/1788843825 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7ff40104d0 con 0x7f8004108110 2024-05-03T21:10:42.365 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:10:41.675+0000 7f800b7f4700 1 -- 172.21.15.67:0/1788843825 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8004108110 msgr2=0x7f8004108530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:10:42.365 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:10:41.675+0000 7f800b7f4700 1 --2- 172.21.15.67:0/1788843825 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8004108110 0x7f8004108530 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f7ff400bb40 tx=0x7f7ff400be50 comp rx=0 tx=0).stop 2024-05-03T21:10:42.365 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:10:41.675+0000 7f800b7f4700 1 -- 172.21.15.67:0/1788843825 shutdown_connections 2024-05-03T21:10:42.365 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:10:41.675+0000 7f800b7f4700 1 --2- 172.21.15.67:0/1788843825 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8004108110 0x7f8004108530 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:10:42.365 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:10:41.675+0000 7f800b7f4700 1 -- 172.21.15.67:0/1788843825 >> 172.21.15.67:0/1788843825 conn(0x7f80041037b0 msgr2=0x7f8004105bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T21:10:42.365 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:10:41.675+0000 7f800b7f4700 1 -- 172.21.15.67:0/1788843825 shutdown_connections 2024-05-03T21:10:42.366 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:10:41.675+0000 7f800b7f4700 1 -- 172.21.15.67:0/1788843825 wait complete. 2024-05-03T21:10:42.366 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:10:41.676+0000 7f800b7f4700 1 Processor -- start 2024-05-03T21:10:42.366 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:10:41.676+0000 7f800b7f4700 1 -- start start 2024-05-03T21:10:42.366 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:10:41.676+0000 7f800b7f4700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8004108110 0x7f800419b050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:10:42.366 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:10:41.676+0000 7f800b7f4700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f8004076290 con 0x7f8004108110 2024-05-03T21:10:42.366 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:10:41.676+0000 7f8009590700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8004108110 0x7f800419b050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:10:42.366 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:10:41.676+0000 7f8009590700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8004108110 0x7f800419b050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:59700/0 (socket says 172.21.15.67:59700) 2024-05-03T21:10:42.366 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:10:41.676+0000 7f8009590700 1 -- 172.21.15.67:0/1028439948 learned_addr learned my addr 172.21.15.67:0/1028439948 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-03T21:10:42.366 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:10:41.677+0000 7f8009590700 1 -- 172.21.15.67:0/1028439948 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f800419b590 con 0x7f8004108110 2024-05-03T21:10:42.366 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:10:41.677+0000 7f8009590700 1 --2- 172.21.15.67:0/1028439948 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8004108110 0x7f800419b050 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f7ff4000c00 tx=0x7f7ff4003ac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T21:10:42.366 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:10:41.677+0000 7f7ffa7fc700 1 -- 172.21.15.67:0/1028439948 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7ff4027070 con 0x7f8004108110 2024-05-03T21:10:42.366 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:10:41.677+0000 7f800b7f4700 1 -- 172.21.15.67:0/1028439948 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f800419b820 con 0x7f8004108110 2024-05-03T21:10:42.366 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:10:41.677+0000 7f7ffa7fc700 1 -- 172.21.15.67:0/1028439948 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(28 keys) ==== 1139+0+0 (secure 0 0 0) 0x7f7ff4010c60 con 0x7f8004108110 2024-05-03T21:10:42.367 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:10:41.677+0000 7f7ffa7fc700 1 -- 172.21.15.67:0/1028439948 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7ff4022bd0 con 0x7f8004108110 2024-05-03T21:10:42.367 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:10:41.677+0000 7f800b7f4700 1 -- 172.21.15.67:0/1028439948 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f800419e510 con 0x7f8004108110 2024-05-03T21:10:42.367 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:10:41.678+0000 7f7ffa7fc700 1 -- 172.21.15.67:0/1028439948 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f7ff402a070 con 0x7f8004108110 2024-05-03T21:10:42.367 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:10:41.678+0000 7f800b7f4700 1 -- 172.21.15.67:0/1028439948 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8004064b00 con 0x7f8004108110 2024-05-03T21:10:42.367 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:10:41.678+0000 7f7ffa7fc700 1 -- 172.21.15.67:0/1028439948 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f7ff401f070 con 0x7f8004108110 2024-05-03T21:10:42.367 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:10:41.680+0000 7f7ffa7fc700 1 -- 172.21.15.67:0/1028439948 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7f7ff4031030 con 0x7f8004108110 2024-05-03T21:10:42.367 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:10:41.756+0000 7f800b7f4700 1 -- 172.21.15.67:0/1028439948 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7f80041945b0 con 0x7f8004108110 2024-05-03T21:10:42.367 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:10:41.756+0000 7f7ffa7fc700 1 -- 172.21.15.67:0/1028439948 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+1239 (secure 0 0 0) 0x7f80041945b0 con 0x7f8004108110 2024-05-03T21:10:42.367 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:10:41.757+0000 7f800b7f4700 1 -- 172.21.15.67:0/1028439948 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8004108110 msgr2=0x7f800419b050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:10:42.367 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:10:41.757+0000 7f800b7f4700 1 --2- 172.21.15.67:0/1028439948 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8004108110 0x7f800419b050 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f7ff4000c00 tx=0x7f7ff4003ac0 comp rx=0 tx=0).stop 2024-05-03T21:10:42.367 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:10:41.757+0000 7f800b7f4700 1 -- 172.21.15.67:0/1028439948 shutdown_connections 2024-05-03T21:10:42.367 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:10:41.757+0000 7f800b7f4700 1 --2- 172.21.15.67:0/1028439948 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8004108110 0x7f800419b050 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:10:42.367 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:10:41.757+0000 7f800b7f4700 1 -- 172.21.15.67:0/1028439948 >> 172.21.15.67:0/1028439948 conn(0x7f80041037b0 msgr2=0x7f8004191690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T21:10:42.367 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:10:41.757+0000 7f800b7f4700 1 -- 172.21.15.67:0/1028439948 shutdown_connections 2024-05-03T21:10:42.367 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:10:41.757+0000 7f800b7f4700 1 -- 172.21.15.67:0/1028439948 wait complete. 2024-05-03T21:10:42.368 INFO:teuthology.orchestra.run.smithi067.stdout:mgr not available, waiting (1/15)... 2024-05-03T21:10:43.720 INFO:journalctl@ceph.mgr.x.smithi067.stdout:May 03 21:10:43 smithi067 ceph-713bc57a-0991-11ef-bc97-c7b262605968-mgr-x[29964]: 2024-05-03T21:10:43.373+0000 7f0f6a9d4200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-05-03T21:10:43.721 INFO:journalctl@ceph.mgr.x.smithi067.stdout:May 03 21:10:43 smithi067 ceph-713bc57a-0991-11ef-bc97-c7b262605968-mgr-x[29964]: 2024-05-03T21:10:43.548+0000 7f0f6a9d4200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-05-03T21:10:44.040 INFO:journalctl@ceph.mgr.x.smithi067.stdout:May 03 21:10:43 smithi067 ceph-713bc57a-0991-11ef-bc97-c7b262605968-mgr-x[29964]: 2024-05-03T21:10:43.724+0000 7f0f6a9d4200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-05-03T21:10:44.041 INFO:journalctl@ceph.mgr.x.smithi067.stdout:May 03 21:10:44 smithi067 ceph-713bc57a-0991-11ef-bc97-c7b262605968-mgr-x[29964]: 2024-05-03T21:10:44.038+0000 7f0f6a9d4200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-05-03T21:10:44.345 INFO:journalctl@ceph.mgr.x.smithi067.stdout:May 03 21:10:44 smithi067 ceph-713bc57a-0991-11ef-bc97-c7b262605968-mgr-x[29964]: 2024-05-03T21:10:44.133+0000 7f0f6a9d4200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-05-03T21:10:44.345 INFO:journalctl@ceph.mgr.x.smithi067.stdout:May 03 21:10:44 smithi067 ceph-713bc57a-0991-11ef-bc97-c7b262605968-mgr-x[29964]: 2024-05-03T21:10:44.343+0000 7f0f6a9d4200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-05-03T21:10:44.721 INFO:journalctl@ceph.mgr.x.smithi067.stdout:May 03 21:10:44 smithi067 ceph-713bc57a-0991-11ef-bc97-c7b262605968-mgr-x[29964]: 2024-05-03T21:10:44.549+0000 7f0f6a9d4200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-05-03T21:10:45.410 INFO:journalctl@ceph.mgr.x.smithi067.stdout:May 03 21:10:45 smithi067 ceph-713bc57a-0991-11ef-bc97-c7b262605968-mgr-x[29964]: 2024-05-03T21:10:45.180+0000 7f0f6a9d4200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-05-03T21:10:45.410 INFO:journalctl@ceph.mgr.x.smithi067.stdout:May 03 21:10:45 smithi067 ceph-713bc57a-0991-11ef-bc97-c7b262605968-mgr-x[29964]: 2024-05-03T21:10:45.408+0000 7f0f6a9d4200 -1 mgr[py] Module smb has missing NOTIFY_TYPES member 2024-05-03T21:10:46.208 INFO:journalctl@ceph.mgr.x.smithi067.stdout:May 03 21:10:45 smithi067 ceph-713bc57a-0991-11ef-bc97-c7b262605968-mgr-x[29964]: 2024-05-03T21:10:45.848+0000 7f0f6a9d4200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-05-03T21:10:46.208 INFO:journalctl@ceph.mgr.x.smithi067.stdout:May 03 21:10:45 smithi067 ceph-713bc57a-0991-11ef-bc97-c7b262605968-mgr-x[29964]: 2024-05-03T21:10:45.948+0000 7f0f6a9d4200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-05-03T21:10:46.470 INFO:journalctl@ceph.mgr.x.smithi067.stdout:May 03 21:10:46 smithi067 ceph-713bc57a-0991-11ef-bc97-c7b262605968-mgr-x[29964]: 2024-05-03T21:10:46.349+0000 7f0f6a9d4200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-05-03T21:10:47.071 INFO:journalctl@ceph.mgr.x.smithi067.stdout:May 03 21:10:46 smithi067 ceph-713bc57a-0991-11ef-bc97-c7b262605968-mgr-x[29964]: 2024-05-03T21:10:46.720+0000 7f0f6a9d4200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-05-03T21:10:47.071 INFO:journalctl@ceph.mgr.x.smithi067.stdout:May 03 21:10:46 smithi067 ceph-713bc57a-0991-11ef-bc97-c7b262605968-mgr-x[29964]: 2024-05-03T21:10:46.816+0000 7f0f6a9d4200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-05-03T21:10:47.071 INFO:journalctl@ceph.mgr.x.smithi067.stdout:May 03 21:10:47 smithi067 ceph-713bc57a-0991-11ef-bc97-c7b262605968-mgr-x[29964]: 2024-05-03T21:10:47.069+0000 7f0f6a9d4200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-05-03T21:10:47.416 INFO:journalctl@ceph.mgr.x.smithi067.stdout:May 03 21:10:47 smithi067 ceph-713bc57a-0991-11ef-bc97-c7b262605968-mgr-x[29964]: 2024-05-03T21:10:47.207+0000 7f0f6a9d4200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-05-03T21:10:47.720 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:10:47 smithi067 ceph-mon[29730]: from='client.? 172.21.15.67:0/653678195' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-05-03T21:10:47.721 INFO:journalctl@ceph.mgr.x.smithi067.stdout:May 03 21:10:47 smithi067 ceph-713bc57a-0991-11ef-bc97-c7b262605968-mgr-x[29964]: 2024-05-03T21:10:47.528+0000 7f0f6a9d4200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-05-03T21:10:47.721 INFO:journalctl@ceph.mgr.x.smithi067.stdout:May 03 21:10:47 smithi067 ceph-713bc57a-0991-11ef-bc97-c7b262605968-mgr-x[29964]: 2024-05-03T21:10:47.649+0000 7f0f6a9d4200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-05-03T21:10:48.097 INFO:journalctl@ceph.mgr.x.smithi067.stdout:May 03 21:10:47 smithi067 ceph-713bc57a-0991-11ef-bc97-c7b262605968-mgr-x[29964]: 2024-05-03T21:10:47.745+0000 7f0f6a9d4200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-05-03T21:10:48.655 INFO:journalctl@ceph.mgr.x.smithi067.stdout:May 03 21:10:48 smithi067 ceph-713bc57a-0991-11ef-bc97-c7b262605968-mgr-x[29964]: 2024-05-03T21:10:48.373+0000 7f0f6a9d4200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-05-03T21:10:48.655 INFO:journalctl@ceph.mgr.x.smithi067.stdout:May 03 21:10:48 smithi067 ceph-713bc57a-0991-11ef-bc97-c7b262605968-mgr-x[29964]: 2024-05-03T21:10:48.598+0000 7f0f6a9d4200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-05-03T21:10:48.971 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:10:48 smithi067 ceph-mon[29730]: Activating manager daemon x 2024-05-03T21:10:48.971 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:10:48 smithi067 ceph-mon[29730]: mgrmap e2: x(active, starting, since 0.00355957s) 2024-05-03T21:10:48.971 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:10:48 smithi067 ceph-mon[29730]: from='mgr.14100 172.21.15.67:0/1576097845' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-05-03T21:10:48.971 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:10:48 smithi067 ceph-mon[29730]: from='mgr.14100 172.21.15.67:0/1576097845' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-05-03T21:10:48.971 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:10:48 smithi067 ceph-mon[29730]: from='mgr.14100 172.21.15.67:0/1576097845' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-05-03T21:10:48.971 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:10:48 smithi067 ceph-mon[29730]: from='mgr.14100 172.21.15.67:0/1576097845' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-03T21:10:48.971 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:10:48 smithi067 ceph-mon[29730]: from='mgr.14100 172.21.15.67:0/1576097845' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-05-03T21:10:48.971 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:10:48 smithi067 ceph-mon[29730]: Manager daemon x is now available 2024-05-03T21:10:48.971 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:10:48 smithi067 ceph-mon[29730]: from='mgr.14100 172.21.15.67:0/1576097845' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-05-03T21:10:48.971 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:10:48 smithi067 ceph-mon[29730]: from='mgr.14100 172.21.15.67:0/1576097845' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-05-03T21:10:48.972 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:10:48 smithi067 ceph-mon[29730]: from='mgr.14100 172.21.15.67:0/1576097845' entity='mgr.x' 2024-05-03T21:10:48.972 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:10:48 smithi067 ceph-mon[29730]: from='mgr.14100 172.21.15.67:0/1576097845' entity='mgr.x' 2024-05-03T21:10:48.972 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:10:48 smithi067 ceph-mon[29730]: from='mgr.14100 172.21.15.67:0/1576097845' entity='mgr.x' 2024-05-03T21:10:50.971 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:10:50 smithi067 ceph-mon[29730]: mgrmap e3: x(active, since 1.00766s) 2024-05-03T21:10:51.970 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:10:51 smithi067 ceph-mon[29730]: mgrmap e4: x(active, since 2s) 2024-05-03T21:10:53.141 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout 2024-05-03T21:10:53.141 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout { 2024-05-03T21:10:53.141 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "fsid": "713bc57a-0991-11ef-bc97-c7b262605968", 2024-05-03T21:10:53.141 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "health": { 2024-05-03T21:10:53.142 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-05-03T21:10:53.142 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-05-03T21:10:53.142 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-05-03T21:10:53.142 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout }, 2024-05-03T21:10:53.142 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-05-03T21:10:53.142 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-05-03T21:10:53.142 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout 0 2024-05-03T21:10:53.142 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout ], 2024-05-03T21:10:53.142 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-05-03T21:10:53.142 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "a" 2024-05-03T21:10:53.142 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout ], 2024-05-03T21:10:53.142 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "quorum_age": 8, 2024-05-03T21:10:53.142 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "monmap": { 2024-05-03T21:10:53.142 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-05-03T21:10:53.142 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-05-03T21:10:53.143 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-05-03T21:10:53.143 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout }, 2024-05-03T21:10:53.143 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-05-03T21:10:53.143 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-05-03T21:10:53.143 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-05-03T21:10:53.143 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-05-03T21:10:53.143 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-05-03T21:10:53.151 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-05-03T21:10:53.151 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-05-03T21:10:53.151 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-05-03T21:10:53.151 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout }, 2024-05-03T21:10:53.151 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-05-03T21:10:53.151 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-05-03T21:10:53.152 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-05-03T21:10:53.152 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-05-03T21:10:53.152 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-05-03T21:10:53.152 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-05-03T21:10:53.152 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-05-03T21:10:53.152 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-05-03T21:10:53.152 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-05-03T21:10:53.152 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout }, 2024-05-03T21:10:53.152 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-05-03T21:10:53.152 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-05-03T21:10:53.152 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-05-03T21:10:53.152 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-05-03T21:10:53.152 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout }, 2024-05-03T21:10:53.152 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-05-03T21:10:53.153 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "available": false, 2024-05-03T21:10:53.153 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-05-03T21:10:53.153 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "modules": [ 2024-05-03T21:10:53.153 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "iostat", 2024-05-03T21:10:53.153 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "nfs", 2024-05-03T21:10:53.153 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "restful" 2024-05-03T21:10:53.153 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout ], 2024-05-03T21:10:53.153 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "services": {} 2024-05-03T21:10:53.153 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout }, 2024-05-03T21:10:53.153 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-05-03T21:10:53.153 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-05-03T21:10:53.153 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "modified": "2024-05-03T21:10:34.070672+0000", 2024-05-03T21:10:53.153 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "services": {} 2024-05-03T21:10:53.153 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout }, 2024-05-03T21:10:53.153 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-05-03T21:10:53.153 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout } 2024-05-03T21:10:53.154 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:10:47.414+0000 7f25158ee700 1 Processor -- start 2024-05-03T21:10:53.154 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:10:47.414+0000 7f25158ee700 1 -- start start 2024-05-03T21:10:53.154 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:10:47.414+0000 7f25158ee700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2510108150 0x7f2510108570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:10:53.154 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:10:47.414+0000 7f25158ee700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f2510108b40 con 0x7f2510108150 2024-05-03T21:10:53.154 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:10:47.414+0000 7f250effd700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2510108150 0x7f2510108570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:10:53.154 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:10:47.414+0000 7f250effd700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2510108150 0x7f2510108570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:47256/0 (socket says 172.21.15.67:47256) 2024-05-03T21:10:53.154 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:10:47.414+0000 7f250effd700 1 -- 172.21.15.67:0/196351396 learned_addr learned my addr 172.21.15.67:0/196351396 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-03T21:10:53.154 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:10:47.414+0000 7f250effd700 1 -- 172.21.15.67:0/196351396 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2510109350 con 0x7f2510108150 2024-05-03T21:10:53.154 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:10:47.415+0000 7f250effd700 1 --2- 172.21.15.67:0/196351396 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2510108150 0x7f2510108570 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f250000bb40 tx=0x7f250000be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=df05ded978bf2ca0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T21:10:53.154 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:10:47.415+0000 7f250dffb700 1 -- 172.21.15.67:0/196351396 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f250000dbe0 con 0x7f2510108150 2024-05-03T21:10:53.154 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:10:47.415+0000 7f250dffb700 1 -- 172.21.15.67:0/196351396 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(28 keys) ==== 1139+0+0 (secure 0 0 0) 0x7f2500004510 con 0x7f2510108150 2024-05-03T21:10:53.154 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:10:47.415+0000 7f250dffb700 1 -- 172.21.15.67:0/196351396 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f25000104d0 con 0x7f2510108150 2024-05-03T21:10:53.154 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:10:47.415+0000 7f25158ee700 1 -- 172.21.15.67:0/196351396 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2510108150 msgr2=0x7f2510108570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:10:53.154 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:10:47.415+0000 7f25158ee700 1 --2- 172.21.15.67:0/196351396 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2510108150 0x7f2510108570 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f250000bb40 tx=0x7f250000be50 comp rx=0 tx=0).stop 2024-05-03T21:10:53.155 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:10:47.416+0000 7f25158ee700 1 -- 172.21.15.67:0/196351396 shutdown_connections 2024-05-03T21:10:53.155 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:10:47.416+0000 7f25158ee700 1 --2- 172.21.15.67:0/196351396 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2510108150 0x7f2510108570 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:10:53.155 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:10:47.416+0000 7f25158ee700 1 -- 172.21.15.67:0/196351396 >> 172.21.15.67:0/196351396 conn(0x7f2510103750 msgr2=0x7f2510105b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T21:10:53.155 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:10:47.416+0000 7f25158ee700 1 -- 172.21.15.67:0/196351396 shutdown_connections 2024-05-03T21:10:53.155 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:10:47.416+0000 7f25158ee700 1 -- 172.21.15.67:0/196351396 wait complete. 2024-05-03T21:10:53.155 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:10:47.416+0000 7f25158ee700 1 Processor -- start 2024-05-03T21:10:53.155 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:10:47.416+0000 7f25158ee700 1 -- start start 2024-05-03T21:10:53.155 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:10:47.417+0000 7f25158ee700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2510108150 0x7f251019b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:10:53.155 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:10:47.417+0000 7f25158ee700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f251007c170 con 0x7f2510108150 2024-05-03T21:10:53.155 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:10:47.417+0000 7f250effd700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2510108150 0x7f251019b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:10:53.155 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:10:47.417+0000 7f250effd700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2510108150 0x7f251019b0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:47270/0 (socket says 172.21.15.67:47270) 2024-05-03T21:10:53.155 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:10:47.417+0000 7f250effd700 1 -- 172.21.15.67:0/653678195 learned_addr learned my addr 172.21.15.67:0/653678195 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-03T21:10:53.155 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:10:47.417+0000 7f250effd700 1 -- 172.21.15.67:0/653678195 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f251019b610 con 0x7f2510108150 2024-05-03T21:10:53.155 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:10:47.417+0000 7f250effd700 1 --2- 172.21.15.67:0/653678195 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2510108150 0x7f251019b0d0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f250000d650 tx=0x7f25000041a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T21:10:53.155 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:10:47.417+0000 7f25148ec700 1 -- 172.21.15.67:0/653678195 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2500027070 con 0x7f2510108150 2024-05-03T21:10:53.156 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:10:47.417+0000 7f25148ec700 1 -- 172.21.15.67:0/653678195 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(28 keys) ==== 1139+0+0 (secure 0 0 0) 0x7f2500003a10 con 0x7f2510108150 2024-05-03T21:10:53.156 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:10:47.417+0000 7f25158ee700 1 -- 172.21.15.67:0/653678195 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f251019b8a0 con 0x7f2510108150 2024-05-03T21:10:53.156 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:10:47.417+0000 7f25148ec700 1 -- 172.21.15.67:0/653678195 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2500022cb0 con 0x7f2510108150 2024-05-03T21:10:53.156 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:10:47.417+0000 7f25158ee700 1 -- 172.21.15.67:0/653678195 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f251019e590 con 0x7f2510108150 2024-05-03T21:10:53.156 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:10:47.418+0000 7f25148ec700 1 -- 172.21.15.67:0/653678195 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f250002a070 con 0x7f2510108150 2024-05-03T21:10:53.156 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:10:47.418+0000 7f25148ec700 1 -- 172.21.15.67:0/653678195 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f250001f070 con 0x7f2510108150 2024-05-03T21:10:53.156 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:10:47.418+0000 7f25158ee700 1 -- 172.21.15.67:0/653678195 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2510064b00 con 0x7f2510108150 2024-05-03T21:10:53.156 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:10:47.421+0000 7f25148ec700 1 -- 172.21.15.67:0/653678195 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7f2500031030 con 0x7f2510108150 2024-05-03T21:10:53.156 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:10:47.497+0000 7f25158ee700 1 -- 172.21.15.67:0/653678195 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7f2510194630 con 0x7f2510108150 2024-05-03T21:10:53.156 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:10:47.498+0000 7f25148ec700 1 -- 172.21.15.67:0/653678195 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+1239 (secure 0 0 0) 0x7f2500015070 con 0x7f2510108150 2024-05-03T21:10:53.156 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:10:47.498+0000 7f25158ee700 1 -- 172.21.15.67:0/653678195 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2510108150 msgr2=0x7f251019b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:10:53.156 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:10:47.498+0000 7f25158ee700 1 --2- 172.21.15.67:0/653678195 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2510108150 0x7f251019b0d0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f250000d650 tx=0x7f25000041a0 comp rx=0 tx=0).stop 2024-05-03T21:10:53.156 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:10:47.499+0000 7f25158ee700 1 -- 172.21.15.67:0/653678195 shutdown_connections 2024-05-03T21:10:53.156 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:10:47.499+0000 7f25158ee700 1 --2- 172.21.15.67:0/653678195 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2510108150 0x7f251019b0d0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:10:53.156 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:10:47.499+0000 7f25158ee700 1 -- 172.21.15.67:0/653678195 >> 172.21.15.67:0/653678195 conn(0x7f2510103750 msgr2=0x7f25101916f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T21:10:53.157 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:10:47.499+0000 7f25158ee700 1 -- 172.21.15.67:0/653678195 shutdown_connections 2024-05-03T21:10:53.157 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:10:47.499+0000 7f25158ee700 1 -- 172.21.15.67:0/653678195 wait complete. 2024-05-03T21:10:53.157 INFO:teuthology.orchestra.run.smithi067.stdout:mgr not available, waiting (2/15)... 2024-05-03T21:10:56.177 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:10:55 smithi067 ceph-mon[29730]: from='client.? 172.21.15.67:0/2292547144' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-05-03T21:10:56.461 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout 2024-05-03T21:10:56.461 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout { 2024-05-03T21:10:56.461 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "fsid": "713bc57a-0991-11ef-bc97-c7b262605968", 2024-05-03T21:10:56.461 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "health": { 2024-05-03T21:10:56.461 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-05-03T21:10:56.461 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-05-03T21:10:56.461 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-05-03T21:10:56.461 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout }, 2024-05-03T21:10:56.461 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-05-03T21:10:56.461 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-05-03T21:10:56.461 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout 0 2024-05-03T21:10:56.462 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout ], 2024-05-03T21:10:56.462 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-05-03T21:10:56.462 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "a" 2024-05-03T21:10:56.462 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout ], 2024-05-03T21:10:56.462 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "quorum_age": 16, 2024-05-03T21:10:56.462 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "monmap": { 2024-05-03T21:10:56.462 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-05-03T21:10:56.462 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-05-03T21:10:56.462 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-05-03T21:10:56.462 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout }, 2024-05-03T21:10:56.462 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-05-03T21:10:56.462 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-05-03T21:10:56.462 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-05-03T21:10:56.462 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-05-03T21:10:56.463 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-05-03T21:10:56.463 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-05-03T21:10:56.463 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-05-03T21:10:56.463 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-05-03T21:10:56.463 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout }, 2024-05-03T21:10:56.463 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-05-03T21:10:56.463 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-05-03T21:10:56.463 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-05-03T21:10:56.463 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-05-03T21:10:56.463 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-05-03T21:10:56.463 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-05-03T21:10:56.463 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-05-03T21:10:56.463 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-05-03T21:10:56.463 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-05-03T21:10:56.464 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout }, 2024-05-03T21:10:56.464 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-05-03T21:10:56.464 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-05-03T21:10:56.464 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-05-03T21:10:56.464 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-05-03T21:10:56.464 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout }, 2024-05-03T21:10:56.464 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-05-03T21:10:56.464 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "available": true, 2024-05-03T21:10:56.464 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-05-03T21:10:56.464 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "modules": [ 2024-05-03T21:10:56.464 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "iostat", 2024-05-03T21:10:56.464 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "nfs", 2024-05-03T21:10:56.464 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "restful" 2024-05-03T21:10:56.464 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout ], 2024-05-03T21:10:56.464 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "services": {} 2024-05-03T21:10:56.465 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout }, 2024-05-03T21:10:56.465 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-05-03T21:10:56.465 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-05-03T21:10:56.465 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "modified": "2024-05-03T21:10:34.070672+0000", 2024-05-03T21:10:56.465 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "services": {} 2024-05-03T21:10:56.465 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout }, 2024-05-03T21:10:56.465 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-05-03T21:10:56.465 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout } 2024-05-03T21:10:56.465 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:10:55.551+0000 7fee03faf700 1 Processor -- start 2024-05-03T21:10:56.465 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:10:55.551+0000 7fee03faf700 1 -- start start 2024-05-03T21:10:56.465 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:10:55.551+0000 7fee03faf700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fedfc07ed70 0x7fedfc07d2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:10:56.465 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:10:55.551+0000 7fee03faf700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fedfc07f190 con 0x7fedfc07ed70 2024-05-03T21:10:56.465 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:10:55.551+0000 7fee01d4b700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fedfc07ed70 0x7fedfc07d2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:10:56.465 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:10:55.551+0000 7fee01d4b700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fedfc07ed70 0x7fedfc07d2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:47348/0 (socket says 172.21.15.67:47348) 2024-05-03T21:10:56.465 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:10:55.551+0000 7fee01d4b700 1 -- 172.21.15.67:0/1547578016 learned_addr learned my addr 172.21.15.67:0/1547578016 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-03T21:10:56.466 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:10:55.552+0000 7fee01d4b700 1 -- 172.21.15.67:0/1547578016 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fedfc07d820 con 0x7fedfc07ed70 2024-05-03T21:10:56.466 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:10:55.552+0000 7fee01d4b700 1 --2- 172.21.15.67:0/1547578016 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fedfc07ed70 0x7fedfc07d2e0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fedf80098a0 tx=0x7fedf8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=74fed7c7ba177811 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T21:10:56.466 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:10:55.552+0000 7fee00d49700 1 -- 172.21.15.67:0/1547578016 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fedf8004160 con 0x7fedfc07ed70 2024-05-03T21:10:56.466 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:10:55.552+0000 7fee00d49700 1 -- 172.21.15.67:0/1547578016 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(28 keys) ==== 1139+0+0 (secure 0 0 0) 0x7fedf8004300 con 0x7fedfc07ed70 2024-05-03T21:10:56.466 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:10:55.552+0000 7fee00d49700 1 -- 172.21.15.67:0/1547578016 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fedf8011c00 con 0x7fedfc07ed70 2024-05-03T21:10:56.466 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:10:55.553+0000 7fee03faf700 1 -- 172.21.15.67:0/1547578016 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fedfc07ed70 msgr2=0x7fedfc07d2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:10:56.466 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:10:55.553+0000 7fee03faf700 1 --2- 172.21.15.67:0/1547578016 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fedfc07ed70 0x7fedfc07d2e0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fedf80098a0 tx=0x7fedf8009bb0 comp rx=0 tx=0).stop 2024-05-03T21:10:56.466 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:10:55.553+0000 7fee03faf700 1 -- 172.21.15.67:0/1547578016 shutdown_connections 2024-05-03T21:10:56.466 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:10:55.553+0000 7fee03faf700 1 --2- 172.21.15.67:0/1547578016 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fedfc07ed70 0x7fedfc07d2e0 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:10:56.466 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:10:55.553+0000 7fee03faf700 1 -- 172.21.15.67:0/1547578016 >> 172.21.15.67:0/1547578016 conn(0x7fedfc104950 msgr2=0x7fedfc106d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T21:10:56.466 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:10:55.553+0000 7fee03faf700 1 -- 172.21.15.67:0/1547578016 shutdown_connections 2024-05-03T21:10:56.466 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:10:55.553+0000 7fee03faf700 1 -- 172.21.15.67:0/1547578016 wait complete. 2024-05-03T21:10:56.466 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:10:55.553+0000 7fee03faf700 1 Processor -- start 2024-05-03T21:10:56.467 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:10:55.553+0000 7fee03faf700 1 -- start start 2024-05-03T21:10:56.467 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:10:55.553+0000 7fee03faf700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fedfc07ed70 0x7fedfc19f410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:10:56.467 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:10:55.554+0000 7fee03faf700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fedfc10f780 con 0x7fedfc07ed70 2024-05-03T21:10:56.467 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:10:55.554+0000 7fee01d4b700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fedfc07ed70 0x7fedfc19f410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:10:56.467 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:10:55.554+0000 7fee01d4b700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fedfc07ed70 0x7fedfc19f410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:47354/0 (socket says 172.21.15.67:47354) 2024-05-03T21:10:56.467 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:10:55.554+0000 7fee01d4b700 1 -- 172.21.15.67:0/2292547144 learned_addr learned my addr 172.21.15.67:0/2292547144 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-03T21:10:56.467 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:10:55.554+0000 7fee01d4b700 1 -- 172.21.15.67:0/2292547144 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fedfc19f950 con 0x7fedfc07ed70 2024-05-03T21:10:56.467 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:10:55.554+0000 7fee01d4b700 1 --2- 172.21.15.67:0/2292547144 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fedfc07ed70 0x7fedfc19f410 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7fedf8011bd0 tx=0x7fedf8011cb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T21:10:56.467 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:10:55.554+0000 7fedf2ffd700 1 -- 172.21.15.67:0/2292547144 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fedf8003b20 con 0x7fedfc07ed70 2024-05-03T21:10:56.467 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:10:55.554+0000 7fedf2ffd700 1 -- 172.21.15.67:0/2292547144 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(28 keys) ==== 1139+0+0 (secure 0 0 0) 0x7fedf8003cc0 con 0x7fedfc07ed70 2024-05-03T21:10:56.467 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:10:55.554+0000 7fedf2ffd700 1 -- 172.21.15.67:0/2292547144 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fedf8024930 con 0x7fedfc07ed70 2024-05-03T21:10:56.467 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:10:55.554+0000 7fee03faf700 1 -- 172.21.15.67:0/2292547144 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fedfc19fbe0 con 0x7fedfc07ed70 2024-05-03T21:10:56.467 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:10:55.554+0000 7fee03faf700 1 -- 172.21.15.67:0/2292547144 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fedfc1a28d0 con 0x7fedfc07ed70 2024-05-03T21:10:56.467 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:10:55.555+0000 7fedf2ffd700 1 -- 172.21.15.67:0/2292547144 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 4) ==== 50211+0+0 (secure 0 0 0) 0x7fedf8022020 con 0x7fedfc07ed70 2024-05-03T21:10:56.468 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:10:55.555+0000 7fee03faf700 1 -- 172.21.15.67:0/2292547144 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fedfc064b00 con 0x7fedfc07ed70 2024-05-03T21:10:56.468 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:10:55.555+0000 7fedf2ffd700 1 --2- 172.21.15.67:0/2292547144 >> [v2:172.21.15.67:6800/1698171509,v1:172.21.15.67:6801/1698171509] conn(0x7fede803e470 0x7fede8040930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:10:56.468 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:10:55.556+0000 7fedf2ffd700 1 -- 172.21.15.67:0/2292547144 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7fedf8053330 con 0x7fedfc07ed70 2024-05-03T21:10:56.468 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:10:55.556+0000 7fee0154a700 1 --2- 172.21.15.67:0/2292547144 >> [v2:172.21.15.67:6800/1698171509,v1:172.21.15.67:6801/1698171509] conn(0x7fede803e470 0x7fede8040930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:10:56.468 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:10:55.556+0000 7fee0154a700 1 --2- 172.21.15.67:0/2292547144 >> [v2:172.21.15.67:6800/1698171509,v1:172.21.15.67:6801/1698171509] conn(0x7fede803e470 0x7fede8040930 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fedec006fd0 tx=0x7fedec006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T21:10:56.468 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:10:55.561+0000 7fedf2ffd700 1 -- 172.21.15.67:0/2292547144 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191839 (secure 0 0 0) 0x7fedf8053090 con 0x7fedfc07ed70 2024-05-03T21:10:56.468 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:10:55.874+0000 7fee03faf700 1 -- 172.21.15.67:0/2292547144 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7fedfc1a2aa0 con 0x7fedfc07ed70 2024-05-03T21:10:56.468 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:10:55.874+0000 7fedf2ffd700 1 -- 172.21.15.67:0/2292547144 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+1239 (secure 0 0 0) 0x7fedfc1a2aa0 con 0x7fedfc07ed70 2024-05-03T21:10:56.468 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:10:55.877+0000 7fee03faf700 1 -- 172.21.15.67:0/2292547144 >> [v2:172.21.15.67:6800/1698171509,v1:172.21.15.67:6801/1698171509] conn(0x7fede803e470 msgr2=0x7fede8040930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:10:56.468 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:10:55.877+0000 7fee03faf700 1 --2- 172.21.15.67:0/2292547144 >> [v2:172.21.15.67:6800/1698171509,v1:172.21.15.67:6801/1698171509] conn(0x7fede803e470 0x7fede8040930 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fedec006fd0 tx=0x7fedec006e40 comp rx=0 tx=0).stop 2024-05-03T21:10:56.468 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:10:55.877+0000 7fee03faf700 1 -- 172.21.15.67:0/2292547144 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fedfc07ed70 msgr2=0x7fedfc19f410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:10:56.468 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:10:55.877+0000 7fee03faf700 1 --2- 172.21.15.67:0/2292547144 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fedfc07ed70 0x7fedfc19f410 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7fedf8011bd0 tx=0x7fedf8011cb0 comp rx=0 tx=0).stop 2024-05-03T21:10:56.468 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:10:55.877+0000 7fee03faf700 1 -- 172.21.15.67:0/2292547144 shutdown_connections 2024-05-03T21:10:56.468 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:10:55.877+0000 7fee03faf700 1 --2- 172.21.15.67:0/2292547144 >> [v2:172.21.15.67:6800/1698171509,v1:172.21.15.67:6801/1698171509] conn(0x7fede803e470 0x7fede8040930 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:10:56.468 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:10:55.877+0000 7fee03faf700 1 --2- 172.21.15.67:0/2292547144 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fedfc07ed70 0x7fedfc19f410 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:10:56.469 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:10:55.877+0000 7fee03faf700 1 -- 172.21.15.67:0/2292547144 >> 172.21.15.67:0/2292547144 conn(0x7fedfc104950 msgr2=0x7fedfc105500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T21:10:56.469 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:10:55.877+0000 7fee03faf700 1 -- 172.21.15.67:0/2292547144 shutdown_connections 2024-05-03T21:10:56.469 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:10:55.877+0000 7fee03faf700 1 -- 172.21.15.67:0/2292547144 wait complete. 2024-05-03T21:10:56.469 INFO:teuthology.orchestra.run.smithi067.stdout:mgr is available 2024-05-03T21:10:57.471 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:10:57 smithi067 ceph-mon[29730]: from='client.? 172.21.15.67:0/10340640' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-05-03T21:10:59.074 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout 2024-05-03T21:10:59.074 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout [global] 2024-05-03T21:10:59.075 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout fsid = 713bc57a-0991-11ef-bc97-c7b262605968 2024-05-03T21:10:59.075 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout mon_cluster_log_file_level = debug 2024-05-03T21:10:59.075 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-05-03T21:10:59.075 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-05-03T21:10:59.075 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-05-03T21:10:59.075 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-05-03T21:10:59.075 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout 2024-05-03T21:10:59.075 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout [mgr] 2024-05-03T21:10:59.075 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-05-03T21:10:59.075 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout 2024-05-03T21:10:59.075 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout [osd] 2024-05-03T21:10:59.075 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout bdev_async_discard = True 2024-05-03T21:10:59.075 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2024-05-03T21:10:59.075 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2024-05-03T21:10:59.076 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2024-05-03T21:10:59.076 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-05-03T21:10:59.076 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-05-03T21:10:59.076 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:10:56.835+0000 7f1843c21700 1 Processor -- start 2024-05-03T21:10:59.076 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:10:56.835+0000 7f1843c21700 1 -- start start 2024-05-03T21:10:59.076 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:10:56.835+0000 7f1843c21700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f183c109cc0 0x7f183c10c0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:10:59.076 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:10:56.835+0000 7f1843c21700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f183c076ff0 con 0x7f183c109cc0 2024-05-03T21:10:59.076 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:10:56.835+0000 7f18419bd700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f183c109cc0 0x7f183c10c0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:10:59.076 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:10:56.835+0000 7f18419bd700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f183c109cc0 0x7f183c10c0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:46284/0 (socket says 172.21.15.67:46284) 2024-05-03T21:10:59.076 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:10:56.835+0000 7f18419bd700 1 -- 172.21.15.67:0/3374008645 learned_addr learned my addr 172.21.15.67:0/3374008645 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-03T21:10:59.076 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:10:56.835+0000 7f18419bd700 1 -- 172.21.15.67:0/3374008645 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f183c10c5f0 con 0x7f183c109cc0 2024-05-03T21:10:59.076 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:10:56.836+0000 7f18419bd700 1 --2- 172.21.15.67:0/3374008645 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f183c109cc0 0x7f183c10c0b0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f18380098a0 tx=0x7f1838009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=96f3fbcc7c62d4ff server_cookie=0 in_seq=0 out_seq=0 2024-05-03T21:10:59.076 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:10:56.836+0000 7f18409bb700 1 -- 172.21.15.67:0/3374008645 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1838004160 con 0x7f183c109cc0 2024-05-03T21:10:59.077 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:10:56.836+0000 7f18409bb700 1 -- 172.21.15.67:0/3374008645 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(28 keys) ==== 1139+0+0 (secure 0 0 0) 0x7f1838004300 con 0x7f183c109cc0 2024-05-03T21:10:59.077 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:10:56.836+0000 7f18409bb700 1 -- 172.21.15.67:0/3374008645 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1838011c00 con 0x7f183c109cc0 2024-05-03T21:10:59.077 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:10:56.836+0000 7f1843c21700 1 -- 172.21.15.67:0/3374008645 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f183c109cc0 msgr2=0x7f183c10c0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:10:59.077 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:10:56.836+0000 7f1843c21700 1 --2- 172.21.15.67:0/3374008645 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f183c109cc0 0x7f183c10c0b0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f18380098a0 tx=0x7f1838009bb0 comp rx=0 tx=0).stop 2024-05-03T21:10:59.077 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:10:56.837+0000 7f1843c21700 1 -- 172.21.15.67:0/3374008645 shutdown_connections 2024-05-03T21:10:59.077 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:10:56.837+0000 7f1843c21700 1 --2- 172.21.15.67:0/3374008645 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f183c109cc0 0x7f183c10c0b0 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:10:59.077 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:10:56.837+0000 7f1843c21700 1 -- 172.21.15.67:0/3374008645 >> 172.21.15.67:0/3374008645 conn(0x7f183c103750 msgr2=0x7f183c105b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T21:10:59.077 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:10:56.837+0000 7f1843c21700 1 -- 172.21.15.67:0/3374008645 shutdown_connections 2024-05-03T21:10:59.077 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:10:56.837+0000 7f1843c21700 1 -- 172.21.15.67:0/3374008645 wait complete. 2024-05-03T21:10:59.077 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:10:56.837+0000 7f1843c21700 1 Processor -- start 2024-05-03T21:10:59.077 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:10:56.837+0000 7f1843c21700 1 -- start start 2024-05-03T21:10:59.077 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:10:56.837+0000 7f1843c21700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f183c109cc0 0x7f183c19d280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:10:59.077 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:10:56.838+0000 7f1843c21700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f183c10d680 con 0x7f183c109cc0 2024-05-03T21:10:59.077 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:10:56.838+0000 7f18419bd700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f183c109cc0 0x7f183c19d280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:10:59.078 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:10:56.838+0000 7f18419bd700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f183c109cc0 0x7f183c19d280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:46300/0 (socket says 172.21.15.67:46300) 2024-05-03T21:10:59.078 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:10:56.838+0000 7f18419bd700 1 -- 172.21.15.67:0/10340640 learned_addr learned my addr 172.21.15.67:0/10340640 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-03T21:10:59.078 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:10:56.838+0000 7f18419bd700 1 -- 172.21.15.67:0/10340640 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f183c19d7c0 con 0x7f183c109cc0 2024-05-03T21:10:59.078 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:10:56.838+0000 7f18419bd700 1 --2- 172.21.15.67:0/10340640 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f183c109cc0 0x7f183c19d280 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f1838011bd0 tx=0x7f1838011cb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T21:10:59.078 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:10:56.838+0000 7f1832ffd700 1 -- 172.21.15.67:0/10340640 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1838003b20 con 0x7f183c109cc0 2024-05-03T21:10:59.078 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:10:56.838+0000 7f1832ffd700 1 -- 172.21.15.67:0/10340640 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(28 keys) ==== 1139+0+0 (secure 0 0 0) 0x7f1838003cc0 con 0x7f183c109cc0 2024-05-03T21:10:59.078 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:10:56.838+0000 7f1832ffd700 1 -- 172.21.15.67:0/10340640 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1838024930 con 0x7f183c109cc0 2024-05-03T21:10:59.078 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:10:56.838+0000 7f1843c21700 1 -- 172.21.15.67:0/10340640 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f183c19da50 con 0x7f183c109cc0 2024-05-03T21:10:59.078 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:10:56.838+0000 7f1843c21700 1 -- 172.21.15.67:0/10340640 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f183c1a0740 con 0x7f183c109cc0 2024-05-03T21:10:59.078 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:10:56.839+0000 7f1832ffd700 1 -- 172.21.15.67:0/10340640 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 4) ==== 50211+0+0 (secure 0 0 0) 0x7f1838022020 con 0x7f183c109cc0 2024-05-03T21:10:59.078 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:10:56.839+0000 7f1832ffd700 1 --2- 172.21.15.67:0/10340640 >> [v2:172.21.15.67:6800/1698171509,v1:172.21.15.67:6801/1698171509] conn(0x7f182803e420 0x7f18280408e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:10:59.078 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:10:56.839+0000 7f1832ffd700 1 -- 172.21.15.67:0/10340640 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f1838053380 con 0x7f183c109cc0 2024-05-03T21:10:59.078 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:10:56.839+0000 7f1843c21700 1 -- 172.21.15.67:0/10340640 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f183c064b00 con 0x7f183c109cc0 2024-05-03T21:10:59.078 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:10:56.840+0000 7f18411bc700 1 --2- 172.21.15.67:0/10340640 >> [v2:172.21.15.67:6800/1698171509,v1:172.21.15.67:6801/1698171509] conn(0x7f182803e420 0x7f18280408e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:10:59.079 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:10:56.840+0000 7f18411bc700 1 --2- 172.21.15.67:0/10340640 >> [v2:172.21.15.67:6800/1698171509,v1:172.21.15.67:6801/1698171509] conn(0x7f182803e420 0x7f18280408e0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f182c006fd0 tx=0x7f182c006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T21:10:59.079 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:10:56.845+0000 7f1832ffd700 1 -- 172.21.15.67:0/10340640 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191839 (secure 0 0 0) 0x7f183800e260 con 0x7f183c109cc0 2024-05-03T21:10:59.079 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:10:57.061+0000 7f1843c21700 1 -- 172.21.15.67:0/10340640 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) -- 0x7f183c1a1e90 con 0x7f183c109cc0 2024-05-03T21:10:59.079 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:10:57.062+0000 7f1832ffd700 1 -- 172.21.15.67:0/10340640 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v3) ==== 70+0+436 (secure 0 0 0) 0x7f18380530e0 con 0x7f183c109cc0 2024-05-03T21:10:59.079 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:10:57.065+0000 7f1843c21700 1 -- 172.21.15.67:0/10340640 >> [v2:172.21.15.67:6800/1698171509,v1:172.21.15.67:6801/1698171509] conn(0x7f182803e420 msgr2=0x7f18280408e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:10:59.079 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:10:57.065+0000 7f1843c21700 1 --2- 172.21.15.67:0/10340640 >> [v2:172.21.15.67:6800/1698171509,v1:172.21.15.67:6801/1698171509] conn(0x7f182803e420 0x7f18280408e0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f182c006fd0 tx=0x7f182c006e40 comp rx=0 tx=0).stop 2024-05-03T21:10:59.079 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:10:57.065+0000 7f1843c21700 1 -- 172.21.15.67:0/10340640 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f183c109cc0 msgr2=0x7f183c19d280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:10:59.079 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:10:57.065+0000 7f1843c21700 1 --2- 172.21.15.67:0/10340640 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f183c109cc0 0x7f183c19d280 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f1838011bd0 tx=0x7f1838011cb0 comp rx=0 tx=0).stop 2024-05-03T21:10:59.080 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:10:57.065+0000 7f1843c21700 1 -- 172.21.15.67:0/10340640 shutdown_connections 2024-05-03T21:10:59.080 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:10:57.065+0000 7f1843c21700 1 --2- 172.21.15.67:0/10340640 >> [v2:172.21.15.67:6800/1698171509,v1:172.21.15.67:6801/1698171509] conn(0x7f182803e420 0x7f18280408e0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:10:59.080 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:10:57.065+0000 7f1843c21700 1 --2- 172.21.15.67:0/10340640 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f183c109cc0 0x7f183c19d280 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:10:59.080 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:10:57.065+0000 7f1843c21700 1 -- 172.21.15.67:0/10340640 >> 172.21.15.67:0/10340640 conn(0x7f183c103750 msgr2=0x7f183c1042b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T21:10:59.080 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:10:57.065+0000 7f1843c21700 1 -- 172.21.15.67:0/10340640 shutdown_connections 2024-05-03T21:10:59.080 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:10:57.065+0000 7f1843c21700 1 -- 172.21.15.67:0/10340640 wait complete. 2024-05-03T21:10:59.080 INFO:teuthology.orchestra.run.smithi067.stdout:Enabling cephadm module... 2024-05-03T21:11:00.200 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:10:59 smithi067 ceph-mon[29730]: from='client.? 172.21.15.67:0/945518574' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2024-05-03T21:11:00.200 INFO:journalctl@ceph.mgr.x.smithi067.stdout:May 03 21:10:59 smithi067 ceph-713bc57a-0991-11ef-bc97-c7b262605968-mgr-x[29964]: ignoring --setuser ceph since I am not root 2024-05-03T21:11:00.201 INFO:journalctl@ceph.mgr.x.smithi067.stdout:May 03 21:10:59 smithi067 ceph-713bc57a-0991-11ef-bc97-c7b262605968-mgr-x[29964]: ignoring --setgroup ceph since I am not root 2024-05-03T21:11:00.201 INFO:journalctl@ceph.mgr.x.smithi067.stdout:May 03 21:11:00 smithi067 ceph-713bc57a-0991-11ef-bc97-c7b262605968-mgr-x[29964]: 2024-05-03T21:11:00.012+0000 7f4278f95200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-05-03T21:11:00.419 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:10:59.499+0000 7f1023b1f700 1 Processor -- start 2024-05-03T21:11:00.419 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:10:59.500+0000 7f1023b1f700 1 -- start start 2024-05-03T21:11:00.420 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:10:59.500+0000 7f1023b1f700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f101c109ce0 0x7f101c10c0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:11:00.420 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:10:59.500+0000 7f1023b1f700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f101c076f90 con 0x7f101c109ce0 2024-05-03T21:11:00.420 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:10:59.500+0000 7f10218bb700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f101c109ce0 0x7f101c10c0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:11:00.420 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:10:59.500+0000 7f10218bb700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f101c109ce0 0x7f101c10c0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:46310/0 (socket says 172.21.15.67:46310) 2024-05-03T21:11:00.420 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:10:59.500+0000 7f10218bb700 1 -- 172.21.15.67:0/1631938684 learned_addr learned my addr 172.21.15.67:0/1631938684 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-03T21:11:00.420 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:10:59.500+0000 7f10218bb700 1 -- 172.21.15.67:0/1631938684 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f101c10c610 con 0x7f101c109ce0 2024-05-03T21:11:00.420 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:10:59.501+0000 7f10218bb700 1 --2- 172.21.15.67:0/1631938684 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f101c109ce0 0x7f101c10c0d0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f10100098a0 tx=0x7f1010009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8774770bc097a24f server_cookie=0 in_seq=0 out_seq=0 2024-05-03T21:11:00.420 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:10:59.501+0000 7f10208b9700 1 -- 172.21.15.67:0/1631938684 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1010004160 con 0x7f101c109ce0 2024-05-03T21:11:00.420 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:10:59.501+0000 7f10208b9700 1 -- 172.21.15.67:0/1631938684 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(28 keys) ==== 1139+0+0 (secure 0 0 0) 0x7f1010004300 con 0x7f101c109ce0 2024-05-03T21:11:00.421 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:10:59.501+0000 7f10208b9700 1 -- 172.21.15.67:0/1631938684 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1010011c00 con 0x7f101c109ce0 2024-05-03T21:11:00.421 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:10:59.501+0000 7f1023b1f700 1 -- 172.21.15.67:0/1631938684 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f101c109ce0 msgr2=0x7f101c10c0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:11:00.421 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:10:59.501+0000 7f1023b1f700 1 --2- 172.21.15.67:0/1631938684 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f101c109ce0 0x7f101c10c0d0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f10100098a0 tx=0x7f1010009bb0 comp rx=0 tx=0).stop 2024-05-03T21:11:00.421 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:10:59.501+0000 7f1023b1f700 1 -- 172.21.15.67:0/1631938684 shutdown_connections 2024-05-03T21:11:00.421 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:10:59.501+0000 7f1023b1f700 1 --2- 172.21.15.67:0/1631938684 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f101c109ce0 0x7f101c10c0d0 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:11:00.421 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:10:59.501+0000 7f1023b1f700 1 -- 172.21.15.67:0/1631938684 >> 172.21.15.67:0/1631938684 conn(0x7f101c1037b0 msgr2=0x7f101c105bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T21:11:00.421 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:10:59.502+0000 7f1023b1f700 1 -- 172.21.15.67:0/1631938684 shutdown_connections 2024-05-03T21:11:00.422 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:10:59.502+0000 7f1023b1f700 1 -- 172.21.15.67:0/1631938684 wait complete. 2024-05-03T21:11:00.422 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:10:59.502+0000 7f1023b1f700 1 Processor -- start 2024-05-03T21:11:00.422 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:10:59.502+0000 7f1023b1f700 1 -- start start 2024-05-03T21:11:00.422 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:10:59.502+0000 7f1023b1f700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f101c109ce0 0x7f101c109560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:11:00.422 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:10:59.502+0000 7f1023b1f700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f101c10d640 con 0x7f101c109ce0 2024-05-03T21:11:00.422 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:10:59.502+0000 7f10218bb700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f101c109ce0 0x7f101c109560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:11:00.422 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:10:59.502+0000 7f10218bb700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f101c109ce0 0x7f101c109560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:46316/0 (socket says 172.21.15.67:46316) 2024-05-03T21:11:00.422 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:10:59.502+0000 7f10218bb700 1 -- 172.21.15.67:0/945518574 learned_addr learned my addr 172.21.15.67:0/945518574 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-03T21:11:00.422 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:10:59.503+0000 7f10218bb700 1 -- 172.21.15.67:0/945518574 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f101c109aa0 con 0x7f101c109ce0 2024-05-03T21:11:00.422 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:10:59.503+0000 7f10218bb700 1 --2- 172.21.15.67:0/945518574 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f101c109ce0 0x7f101c109560 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f1010011bd0 tx=0x7f1010011cb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T21:11:00.422 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:10:59.503+0000 7f100effd700 1 -- 172.21.15.67:0/945518574 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1010003b20 con 0x7f101c109ce0 2024-05-03T21:11:00.422 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:10:59.503+0000 7f100effd700 1 -- 172.21.15.67:0/945518574 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(28 keys) ==== 1139+0+0 (secure 0 0 0) 0x7f1010003cc0 con 0x7f101c109ce0 2024-05-03T21:11:00.422 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:10:59.503+0000 7f1023b1f700 1 -- 172.21.15.67:0/945518574 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f101c107c90 con 0x7f101c109ce0 2024-05-03T21:11:00.423 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:10:59.503+0000 7f100effd700 1 -- 172.21.15.67:0/945518574 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1010024930 con 0x7f101c109ce0 2024-05-03T21:11:00.423 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:10:59.503+0000 7f1023b1f700 1 -- 172.21.15.67:0/945518574 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f101c1081d0 con 0x7f101c109ce0 2024-05-03T21:11:00.423 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:10:59.504+0000 7f100effd700 1 -- 172.21.15.67:0/945518574 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 4) ==== 50211+0+0 (secure 0 0 0) 0x7f1010022020 con 0x7f101c109ce0 2024-05-03T21:11:00.423 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:10:59.504+0000 7f1023b1f700 1 -- 172.21.15.67:0/945518574 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f101c064b00 con 0x7f101c109ce0 2024-05-03T21:11:00.423 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:10:59.504+0000 7f100effd700 1 --2- 172.21.15.67:0/945518574 >> [v2:172.21.15.67:6800/1698171509,v1:172.21.15.67:6801/1698171509] conn(0x7f100803e470 0x7f1008040930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:11:00.423 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:10:59.504+0000 7f100effd700 1 -- 172.21.15.67:0/945518574 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f1010053440 con 0x7f101c109ce0 2024-05-03T21:11:00.423 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:10:59.504+0000 7f10210ba700 1 --2- 172.21.15.67:0/945518574 >> [v2:172.21.15.67:6800/1698171509,v1:172.21.15.67:6801/1698171509] conn(0x7f100803e470 0x7f1008040930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:11:00.424 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:10:59.505+0000 7f10210ba700 1 --2- 172.21.15.67:0/945518574 >> [v2:172.21.15.67:6800/1698171509,v1:172.21.15.67:6801/1698171509] conn(0x7f100803e470 0x7f1008040930 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f1018006fd0 tx=0x7f1018006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T21:11:00.424 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:10:59.509+0000 7f100effd700 1 -- 172.21.15.67:0/945518574 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191839 (secure 0 0 0) 0x7f10100531a0 con 0x7f101c109ce0 2024-05-03T21:11:00.424 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:10:59.785+0000 7f1023b1f700 1 -- 172.21.15.67:0/945518574 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "cephadm"} v 0) -- 0x7f101c108450 con 0x7f101c109ce0 2024-05-03T21:11:00.424 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:10:59.837+0000 7f100effd700 1 -- 172.21.15.67:0/945518574 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "cephadm"}]=0 v5) ==== 86+0+0 (secure 0 0 0) 0x7f101001aca0 con 0x7f101c109ce0 2024-05-03T21:11:00.424 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:10:59.839+0000 7f100effd700 1 -- 172.21.15.67:0/945518574 <== mon.0 v2:172.21.15.67:3300/0 8 ==== mgrmap(e 5) ==== 50222+0+0 (secure 0 0 0) 0x7f101002dc40 con 0x7f101c109ce0 2024-05-03T21:11:00.424 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:10:59.840+0000 7f1023b1f700 1 -- 172.21.15.67:0/945518574 >> [v2:172.21.15.67:6800/1698171509,v1:172.21.15.67:6801/1698171509] conn(0x7f100803e470 msgr2=0x7f1008040930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:11:00.425 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:10:59.840+0000 7f1023b1f700 1 --2- 172.21.15.67:0/945518574 >> [v2:172.21.15.67:6800/1698171509,v1:172.21.15.67:6801/1698171509] conn(0x7f100803e470 0x7f1008040930 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f1018006fd0 tx=0x7f1018006e40 comp rx=0 tx=0).stop 2024-05-03T21:11:00.425 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:10:59.840+0000 7f1023b1f700 1 -- 172.21.15.67:0/945518574 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f101c109ce0 msgr2=0x7f101c109560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:11:00.425 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:10:59.840+0000 7f1023b1f700 1 --2- 172.21.15.67:0/945518574 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f101c109ce0 0x7f101c109560 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f1010011bd0 tx=0x7f1010011cb0 comp rx=0 tx=0).stop 2024-05-03T21:11:00.425 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:10:59.840+0000 7f1023b1f700 1 -- 172.21.15.67:0/945518574 shutdown_connections 2024-05-03T21:11:00.425 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:10:59.840+0000 7f1023b1f700 1 --2- 172.21.15.67:0/945518574 >> [v2:172.21.15.67:6800/1698171509,v1:172.21.15.67:6801/1698171509] conn(0x7f100803e470 0x7f1008040930 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:11:00.425 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:10:59.840+0000 7f1023b1f700 1 --2- 172.21.15.67:0/945518574 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f101c109ce0 0x7f101c109560 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:11:00.425 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:10:59.840+0000 7f1023b1f700 1 -- 172.21.15.67:0/945518574 >> 172.21.15.67:0/945518574 conn(0x7f101c1037b0 msgr2=0x7f101c104360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T21:11:00.426 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:10:59.840+0000 7f1023b1f700 1 -- 172.21.15.67:0/945518574 shutdown_connections 2024-05-03T21:11:00.426 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:10:59.840+0000 7f1023b1f700 1 -- 172.21.15.67:0/945518574 wait complete. 2024-05-03T21:11:00.470 INFO:journalctl@ceph.mgr.x.smithi067.stdout:May 03 21:11:00 smithi067 ceph-713bc57a-0991-11ef-bc97-c7b262605968-mgr-x[29964]: 2024-05-03T21:11:00.205+0000 7f4278f95200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-05-03T21:11:00.471 INFO:journalctl@ceph.mgr.x.smithi067.stdout:May 03 21:11:00 smithi067 ceph-713bc57a-0991-11ef-bc97-c7b262605968-mgr-x[29964]: 2024-05-03T21:11:00.312+0000 7f4278f95200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-05-03T21:11:00.843 INFO:journalctl@ceph.mgr.x.smithi067.stdout:May 03 21:11:00 smithi067 ceph-713bc57a-0991-11ef-bc97-c7b262605968-mgr-x[29964]: 2024-05-03T21:11:00.644+0000 7f4278f95200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-05-03T21:11:00.843 INFO:journalctl@ceph.mgr.x.smithi067.stdout:May 03 21:11:00 smithi067 ceph-713bc57a-0991-11ef-bc97-c7b262605968-mgr-x[29964]: 2024-05-03T21:11:00.737+0000 7f4278f95200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-05-03T21:11:01.210 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:11:00 smithi067 ceph-mon[29730]: from='client.? 172.21.15.67:0/945518574' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2024-05-03T21:11:01.210 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:11:00 smithi067 ceph-mon[29730]: mgrmap e5: x(active, since 11s) 2024-05-03T21:11:01.754 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout { 2024-05-03T21:11:01.754 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "epoch": 5, 2024-05-03T21:11:01.754 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "available": true, 2024-05-03T21:11:01.754 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "active_name": "x", 2024-05-03T21:11:01.754 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-05-03T21:11:01.754 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout } 2024-05-03T21:11:01.755 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:00.881+0000 7f01b40d9700 1 Processor -- start 2024-05-03T21:11:01.755 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:00.881+0000 7f01b40d9700 1 -- start start 2024-05-03T21:11:01.755 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:00.881+0000 7f01b40d9700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f01ac107b30 0x7f01ac107f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:11:01.755 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:00.881+0000 7f01b40d9700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f01ac076f70 con 0x7f01ac107b30 2024-05-03T21:11:01.755 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:00.881+0000 7f01b1e75700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f01ac107b30 0x7f01ac107f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:11:01.755 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:00.882+0000 7f01b1e75700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f01ac107b30 0x7f01ac107f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:46334/0 (socket says 172.21.15.67:46334) 2024-05-03T21:11:01.755 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:00.882+0000 7f01b1e75700 1 -- 172.21.15.67:0/3607432991 learned_addr learned my addr 172.21.15.67:0/3607432991 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-03T21:11:01.755 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:00.882+0000 7f01b1e75700 1 -- 172.21.15.67:0/3607432991 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f01ac108490 con 0x7f01ac107b30 2024-05-03T21:11:01.756 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:00.882+0000 7f01b1e75700 1 --2- 172.21.15.67:0/3607432991 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f01ac107b30 0x7f01ac107f50 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f019c0098a0 tx=0x7f019c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e2fcfa33112c26b6 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T21:11:01.756 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:00.882+0000 7f01b0e73700 1 -- 172.21.15.67:0/3607432991 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f019c004160 con 0x7f01ac107b30 2024-05-03T21:11:01.756 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:00.882+0000 7f01b0e73700 1 -- 172.21.15.67:0/3607432991 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(28 keys) ==== 1139+0+0 (secure 0 0 0) 0x7f019c004300 con 0x7f01ac107b30 2024-05-03T21:11:01.756 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:00.882+0000 7f01b0e73700 1 -- 172.21.15.67:0/3607432991 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f019c011c00 con 0x7f01ac107b30 2024-05-03T21:11:01.756 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:00.883+0000 7f01b40d9700 1 -- 172.21.15.67:0/3607432991 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f01ac107b30 msgr2=0x7f01ac107f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:11:01.756 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:00.883+0000 7f01b40d9700 1 --2- 172.21.15.67:0/3607432991 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f01ac107b30 0x7f01ac107f50 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f019c0098a0 tx=0x7f019c009bb0 comp rx=0 tx=0).stop 2024-05-03T21:11:01.756 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:00.883+0000 7f01b40d9700 1 -- 172.21.15.67:0/3607432991 shutdown_connections 2024-05-03T21:11:01.756 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:00.883+0000 7f01b40d9700 1 --2- 172.21.15.67:0/3607432991 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f01ac107b30 0x7f01ac107f50 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:11:01.757 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:00.883+0000 7f01b40d9700 1 -- 172.21.15.67:0/3607432991 >> 172.21.15.67:0/3607432991 conn(0x7f01ac103750 msgr2=0x7f01ac105b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T21:11:01.757 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:00.883+0000 7f01b40d9700 1 -- 172.21.15.67:0/3607432991 shutdown_connections 2024-05-03T21:11:01.757 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:00.883+0000 7f01b40d9700 1 -- 172.21.15.67:0/3607432991 wait complete. 2024-05-03T21:11:01.757 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:00.884+0000 7f01b40d9700 1 Processor -- start 2024-05-03T21:11:01.757 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:00.884+0000 7f01b40d9700 1 -- start start 2024-05-03T21:11:01.757 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:00.884+0000 7f01b40d9700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f01ac107b30 0x7f01ac19f440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:11:01.757 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:00.884+0000 7f01b40d9700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f01ac109580 con 0x7f01ac107b30 2024-05-03T21:11:01.757 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:00.884+0000 7f01b1e75700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f01ac107b30 0x7f01ac19f440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:11:01.757 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:00.884+0000 7f01b1e75700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f01ac107b30 0x7f01ac19f440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:46342/0 (socket says 172.21.15.67:46342) 2024-05-03T21:11:01.757 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:00.884+0000 7f01b1e75700 1 -- 172.21.15.67:0/273734518 learned_addr learned my addr 172.21.15.67:0/273734518 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-03T21:11:01.758 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:00.884+0000 7f01b1e75700 1 -- 172.21.15.67:0/273734518 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f01ac19f980 con 0x7f01ac107b30 2024-05-03T21:11:01.758 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:00.884+0000 7f01b1e75700 1 --2- 172.21.15.67:0/273734518 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f01ac107b30 0x7f01ac19f440 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f019c011bd0 tx=0x7f019c011cb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T21:11:01.758 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:00.884+0000 7f01a2ffd700 1 -- 172.21.15.67:0/273734518 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f019c003b20 con 0x7f01ac107b30 2024-05-03T21:11:01.758 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:00.885+0000 7f01a2ffd700 1 -- 172.21.15.67:0/273734518 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(28 keys) ==== 1139+0+0 (secure 0 0 0) 0x7f019c003cc0 con 0x7f01ac107b30 2024-05-03T21:11:01.758 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:00.885+0000 7f01b40d9700 1 -- 172.21.15.67:0/273734518 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f01ac19fc10 con 0x7f01ac107b30 2024-05-03T21:11:01.758 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:00.885+0000 7f01a2ffd700 1 -- 172.21.15.67:0/273734518 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f019c024930 con 0x7f01ac107b30 2024-05-03T21:11:01.758 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:00.885+0000 7f01b40d9700 1 -- 172.21.15.67:0/273734518 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f01ac1a2900 con 0x7f01ac107b30 2024-05-03T21:11:01.758 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:00.885+0000 7f01a2ffd700 1 -- 172.21.15.67:0/273734518 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 50222+0+0 (secure 0 0 0) 0x7f019c022020 con 0x7f01ac107b30 2024-05-03T21:11:01.759 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:00.886+0000 7f01b40d9700 1 -- 172.21.15.67:0/273734518 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f01ac064b00 con 0x7f01ac107b30 2024-05-03T21:11:01.759 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:00.886+0000 7f01a2ffd700 1 --2- 172.21.15.67:0/273734518 >> [v2:172.21.15.67:6800/1698171509,v1:172.21.15.67:6801/1698171509] conn(0x7f019803e4c0 0x7f0198040980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:11:01.759 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:00.886+0000 7f01a2ffd700 1 -- 172.21.15.67:0/273734518 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f019c053380 con 0x7f01ac107b30 2024-05-03T21:11:01.759 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:00.886+0000 7f01b1674700 1 -- 172.21.15.67:0/273734518 >> [v2:172.21.15.67:6800/1698171509,v1:172.21.15.67:6801/1698171509] conn(0x7f019803e4c0 msgr2=0x7f0198040980 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.67:6800/1698171509 2024-05-03T21:11:01.759 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:00.886+0000 7f01b1674700 1 --2- 172.21.15.67:0/273734518 >> [v2:172.21.15.67:6800/1698171509,v1:172.21.15.67:6801/1698171509] conn(0x7f019803e4c0 0x7f0198040980 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-03T21:11:01.759 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:00.891+0000 7f01a2ffd700 1 -- 172.21.15.67:0/273734518 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191839 (secure 0 0 0) 0x7f019c00e260 con 0x7f01ac107b30 2024-05-03T21:11:01.759 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:01.087+0000 7f01b1674700 1 -- 172.21.15.67:0/273734518 >> [v2:172.21.15.67:6800/1698171509,v1:172.21.15.67:6801/1698171509] conn(0x7f019803e4c0 msgr2=0x7f0198040980 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.67:6800/1698171509 2024-05-03T21:11:01.759 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:01.087+0000 7f01b1674700 1 --2- 172.21.15.67:0/273734518 >> [v2:172.21.15.67:6800/1698171509,v1:172.21.15.67:6801/1698171509] conn(0x7f019803e4c0 0x7f0198040980 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-05-03T21:11:01.759 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:01.208+0000 7f01b40d9700 1 -- 172.21.15.67:0/273734518 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) -- 0x7f01ac0008d0 con 0x7f01ac107b30 2024-05-03T21:11:01.760 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:01.208+0000 7f01a2ffd700 1 -- 172.21.15.67:0/273734518 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v5) ==== 56+0+88 (secure 0 0 0) 0x7f019c0530e0 con 0x7f01ac107b30 2024-05-03T21:11:01.760 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:01.211+0000 7f01b40d9700 1 -- 172.21.15.67:0/273734518 >> [v2:172.21.15.67:6800/1698171509,v1:172.21.15.67:6801/1698171509] conn(0x7f019803e4c0 msgr2=0x7f0198040980 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-03T21:11:01.760 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:01.211+0000 7f01b40d9700 1 --2- 172.21.15.67:0/273734518 >> [v2:172.21.15.67:6800/1698171509,v1:172.21.15.67:6801/1698171509] conn(0x7f019803e4c0 0x7f0198040980 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:11:01.760 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:01.211+0000 7f01b40d9700 1 -- 172.21.15.67:0/273734518 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f01ac107b30 msgr2=0x7f01ac19f440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:11:01.760 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:01.211+0000 7f01b40d9700 1 --2- 172.21.15.67:0/273734518 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f01ac107b30 0x7f01ac19f440 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f019c011bd0 tx=0x7f019c011cb0 comp rx=0 tx=0).stop 2024-05-03T21:11:01.760 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:01.211+0000 7f01b40d9700 1 -- 172.21.15.67:0/273734518 shutdown_connections 2024-05-03T21:11:01.760 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:01.211+0000 7f01b40d9700 1 --2- 172.21.15.67:0/273734518 >> [v2:172.21.15.67:6800/1698171509,v1:172.21.15.67:6801/1698171509] conn(0x7f019803e4c0 0x7f0198040980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:11:01.760 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:01.211+0000 7f01b40d9700 1 --2- 172.21.15.67:0/273734518 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f01ac107b30 0x7f01ac19f440 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:11:01.760 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:01.211+0000 7f01b40d9700 1 -- 172.21.15.67:0/273734518 >> 172.21.15.67:0/273734518 conn(0x7f01ac103750 msgr2=0x7f01ac196080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T21:11:01.761 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:01.211+0000 7f01b40d9700 1 -- 172.21.15.67:0/273734518 shutdown_connections 2024-05-03T21:11:01.761 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:01.211+0000 7f01b40d9700 1 -- 172.21.15.67:0/273734518 wait complete. 2024-05-03T21:11:01.761 INFO:teuthology.orchestra.run.smithi067.stdout:Waiting for the mgr to restart... 2024-05-03T21:11:01.761 INFO:teuthology.orchestra.run.smithi067.stdout:Waiting for mgr epoch 5... 2024-05-03T21:11:02.136 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:11:01 smithi067 ceph-mon[29730]: from='client.? 172.21.15.67:0/273734518' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-05-03T21:11:02.136 INFO:journalctl@ceph.mgr.x.smithi067.stdout:May 03 21:11:02 smithi067 ceph-713bc57a-0991-11ef-bc97-c7b262605968-mgr-x[29964]: 2024-05-03T21:11:02.005+0000 7f4278f95200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-05-03T21:11:02.470 INFO:journalctl@ceph.mgr.x.smithi067.stdout:May 03 21:11:02 smithi067 ceph-713bc57a-0991-11ef-bc97-c7b262605968-mgr-x[29964]: 2024-05-03T21:11:02.188+0000 7f4278f95200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-05-03T21:11:02.471 INFO:journalctl@ceph.mgr.x.smithi067.stdout:May 03 21:11:02 smithi067 ceph-713bc57a-0991-11ef-bc97-c7b262605968-mgr-x[29964]: 2024-05-03T21:11:02.365+0000 7f4278f95200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-05-03T21:11:02.970 INFO:journalctl@ceph.mgr.x.smithi067.stdout:May 03 21:11:02 smithi067 ceph-713bc57a-0991-11ef-bc97-c7b262605968-mgr-x[29964]: 2024-05-03T21:11:02.667+0000 7f4278f95200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-05-03T21:11:02.971 INFO:journalctl@ceph.mgr.x.smithi067.stdout:May 03 21:11:02 smithi067 ceph-713bc57a-0991-11ef-bc97-c7b262605968-mgr-x[29964]: 2024-05-03T21:11:02.763+0000 7f4278f95200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-05-03T21:11:03.470 INFO:journalctl@ceph.mgr.x.smithi067.stdout:May 03 21:11:02 smithi067 ceph-713bc57a-0991-11ef-bc97-c7b262605968-mgr-x[29964]: 2024-05-03T21:11:02.974+0000 7f4278f95200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-05-03T21:11:03.471 INFO:journalctl@ceph.mgr.x.smithi067.stdout:May 03 21:11:03 smithi067 ceph-713bc57a-0991-11ef-bc97-c7b262605968-mgr-x[29964]: 2024-05-03T21:11:03.176+0000 7f4278f95200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-05-03T21:11:04.221 INFO:journalctl@ceph.mgr.x.smithi067.stdout:May 03 21:11:03 smithi067 ceph-713bc57a-0991-11ef-bc97-c7b262605968-mgr-x[29964]: 2024-05-03T21:11:03.831+0000 7f4278f95200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-05-03T21:11:04.221 INFO:journalctl@ceph.mgr.x.smithi067.stdout:May 03 21:11:04 smithi067 ceph-713bc57a-0991-11ef-bc97-c7b262605968-mgr-x[29964]: 2024-05-03T21:11:04.063+0000 7f4278f95200 -1 mgr[py] Module smb has missing NOTIFY_TYPES member 2024-05-03T21:11:04.970 INFO:journalctl@ceph.mgr.x.smithi067.stdout:May 03 21:11:04 smithi067 ceph-713bc57a-0991-11ef-bc97-c7b262605968-mgr-x[29964]: 2024-05-03T21:11:04.511+0000 7f4278f95200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-05-03T21:11:04.971 INFO:journalctl@ceph.mgr.x.smithi067.stdout:May 03 21:11:04 smithi067 ceph-713bc57a-0991-11ef-bc97-c7b262605968-mgr-x[29964]: 2024-05-03T21:11:04.614+0000 7f4278f95200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-05-03T21:11:05.401 INFO:journalctl@ceph.mgr.x.smithi067.stdout:May 03 21:11:05 smithi067 ceph-713bc57a-0991-11ef-bc97-c7b262605968-mgr-x[29964]: 2024-05-03T21:11:05.023+0000 7f4278f95200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-05-03T21:11:05.401 INFO:journalctl@ceph.mgr.x.smithi067.stdout:May 03 21:11:05 smithi067 ceph-713bc57a-0991-11ef-bc97-c7b262605968-mgr-x[29964]: 2024-05-03T21:11:05.399+0000 7f4278f95200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-05-03T21:11:05.720 INFO:journalctl@ceph.mgr.x.smithi067.stdout:May 03 21:11:05 smithi067 ceph-713bc57a-0991-11ef-bc97-c7b262605968-mgr-x[29964]: 2024-05-03T21:11:05.497+0000 7f4278f95200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-05-03T21:11:06.221 INFO:journalctl@ceph.mgr.x.smithi067.stdout:May 03 21:11:05 smithi067 ceph-713bc57a-0991-11ef-bc97-c7b262605968-mgr-x[29964]: 2024-05-03T21:11:05.756+0000 7f4278f95200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-05-03T21:11:06.221 INFO:journalctl@ceph.mgr.x.smithi067.stdout:May 03 21:11:05 smithi067 ceph-713bc57a-0991-11ef-bc97-c7b262605968-mgr-x[29964]: 2024-05-03T21:11:05.897+0000 7f4278f95200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-05-03T21:11:06.642 INFO:journalctl@ceph.mgr.x.smithi067.stdout:May 03 21:11:06 smithi067 ceph-713bc57a-0991-11ef-bc97-c7b262605968-mgr-x[29964]: 2024-05-03T21:11:06.242+0000 7f4278f95200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-05-03T21:11:06.642 INFO:journalctl@ceph.mgr.x.smithi067.stdout:May 03 21:11:06 smithi067 ceph-713bc57a-0991-11ef-bc97-c7b262605968-mgr-x[29964]: 2024-05-03T21:11:06.365+0000 7f4278f95200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-05-03T21:11:06.642 INFO:journalctl@ceph.mgr.x.smithi067.stdout:May 03 21:11:06 smithi067 ceph-713bc57a-0991-11ef-bc97-c7b262605968-mgr-x[29964]: 2024-05-03T21:11:06.462+0000 7f4278f95200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-05-03T21:11:07.387 INFO:journalctl@ceph.mgr.x.smithi067.stdout:May 03 21:11:07 smithi067 ceph-713bc57a-0991-11ef-bc97-c7b262605968-mgr-x[29964]: 2024-05-03T21:11:07.098+0000 7f4278f95200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-05-03T21:11:07.387 INFO:journalctl@ceph.mgr.x.smithi067.stdout:May 03 21:11:07 smithi067 ceph-713bc57a-0991-11ef-bc97-c7b262605968-mgr-x[29964]: 2024-05-03T21:11:07.331+0000 7f4278f95200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-05-03T21:11:07.721 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:11:07 smithi067 ceph-mon[29730]: Active manager daemon x restarted 2024-05-03T21:11:07.722 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:11:07 smithi067 ceph-mon[29730]: Activating manager daemon x 2024-05-03T21:11:07.722 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:11:07 smithi067 ceph-mon[29730]: osdmap e2: 0 total, 0 up, 0 in 2024-05-03T21:11:07.722 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:11:07 smithi067 ceph-mon[29730]: mgrmap e6: x(active, starting, since 0.00376817s) 2024-05-03T21:11:07.722 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:11:07 smithi067 ceph-mon[29730]: from='mgr.14118 172.21.15.67:0/4082808469' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-03T21:11:07.722 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:11:07 smithi067 ceph-mon[29730]: from='mgr.14118 172.21.15.67:0/4082808469' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-05-03T21:11:07.722 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:11:07 smithi067 ceph-mon[29730]: from='mgr.14118 172.21.15.67:0/4082808469' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-05-03T21:11:07.722 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:11:07 smithi067 ceph-mon[29730]: from='mgr.14118 172.21.15.67:0/4082808469' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-05-03T21:11:07.722 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:11:07 smithi067 ceph-mon[29730]: from='mgr.14118 172.21.15.67:0/4082808469' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-05-03T21:11:07.722 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:11:07 smithi067 ceph-mon[29730]: Manager daemon x is now available 2024-05-03T21:11:07.722 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:11:07 smithi067 ceph-mon[29730]: from='mgr.14118 172.21.15.67:0/4082808469' entity='mgr.x' 2024-05-03T21:11:07.722 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:11:07 smithi067 ceph-mon[29730]: from='mgr.14118 172.21.15.67:0/4082808469' entity='mgr.x' 2024-05-03T21:11:07.723 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:11:07 smithi067 ceph-mon[29730]: from='mgr.14118 172.21.15.67:0/4082808469' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T21:11:07.723 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:11:07 smithi067 ceph-mon[29730]: from='mgr.14118 172.21.15.67:0/4082808469' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T21:11:08.605 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:11:08 smithi067 ceph-mon[29730]: Found migration_current of "None". Setting to last migration. 2024-05-03T21:11:08.606 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:11:08 smithi067 ceph-mon[29730]: from='mgr.14118 172.21.15.67:0/4082808469' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-05-03T21:11:08.606 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:11:08 smithi067 ceph-mon[29730]: from='mgr.14118 172.21.15.67:0/4082808469' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-05-03T21:11:08.606 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:11:08 smithi067 ceph-mon[29730]: from='mgr.14118 172.21.15.67:0/4082808469' entity='mgr.x' 2024-05-03T21:11:08.606 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:11:08 smithi067 ceph-mon[29730]: from='mgr.14118 172.21.15.67:0/4082808469' entity='mgr.x' 2024-05-03T21:11:08.606 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:11:08 smithi067 ceph-mon[29730]: mgrmap e7: x(active, since 1.00601s) 2024-05-03T21:11:08.874 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout { 2024-05-03T21:11:08.875 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 7, 2024-05-03T21:11:08.875 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "initialized": true 2024-05-03T21:11:08.875 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout } 2024-05-03T21:11:08.875 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:02.161+0000 7f2ac49e5700 1 Processor -- start 2024-05-03T21:11:08.875 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:02.161+0000 7f2ac49e5700 1 -- start start 2024-05-03T21:11:08.875 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:02.162+0000 7f2ac49e5700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2ac0109dc0 0x7f2ac010c1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:11:08.875 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:02.162+0000 7f2ac49e5700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f2ac007ce30 con 0x7f2ac0109dc0 2024-05-03T21:11:08.875 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:02.162+0000 7f2abe59c700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2ac0109dc0 0x7f2ac010c1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:11:08.875 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:02.162+0000 7f2abe59c700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2ac0109dc0 0x7f2ac010c1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:46348/0 (socket says 172.21.15.67:46348) 2024-05-03T21:11:08.875 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:02.162+0000 7f2abe59c700 1 -- 172.21.15.67:0/4026012023 learned_addr learned my addr 172.21.15.67:0/4026012023 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-03T21:11:08.875 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:02.162+0000 7f2abe59c700 1 -- 172.21.15.67:0/4026012023 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2ac010c6f0 con 0x7f2ac0109dc0 2024-05-03T21:11:08.875 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:02.162+0000 7f2abe59c700 1 --2- 172.21.15.67:0/4026012023 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2ac0109dc0 0x7f2ac010c1b0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f2ab00098a0 tx=0x7f2ab0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b8b1872b3b7f1fb7 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T21:11:08.875 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:02.163+0000 7f2abd59a700 1 -- 172.21.15.67:0/4026012023 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2ab0004160 con 0x7f2ac0109dc0 2024-05-03T21:11:08.876 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:02.163+0000 7f2abd59a700 1 -- 172.21.15.67:0/4026012023 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(28 keys) ==== 1139+0+0 (secure 0 0 0) 0x7f2ab0004300 con 0x7f2ac0109dc0 2024-05-03T21:11:08.876 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:02.163+0000 7f2abd59a700 1 -- 172.21.15.67:0/4026012023 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2ab0011c00 con 0x7f2ac0109dc0 2024-05-03T21:11:08.876 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:02.163+0000 7f2ac49e5700 1 -- 172.21.15.67:0/4026012023 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2ac0109dc0 msgr2=0x7f2ac010c1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:11:08.876 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:02.163+0000 7f2ac49e5700 1 --2- 172.21.15.67:0/4026012023 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2ac0109dc0 0x7f2ac010c1b0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f2ab00098a0 tx=0x7f2ab0009bb0 comp rx=0 tx=0).stop 2024-05-03T21:11:08.876 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:02.164+0000 7f2ac49e5700 1 -- 172.21.15.67:0/4026012023 shutdown_connections 2024-05-03T21:11:08.876 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:02.164+0000 7f2ac49e5700 1 --2- 172.21.15.67:0/4026012023 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2ac0109dc0 0x7f2ac010c1b0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:11:08.876 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:02.164+0000 7f2ac49e5700 1 -- 172.21.15.67:0/4026012023 >> 172.21.15.67:0/4026012023 conn(0x7f2ac0103790 msgr2=0x7f2ac0105bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T21:11:08.876 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:02.164+0000 7f2ac49e5700 1 -- 172.21.15.67:0/4026012023 shutdown_connections 2024-05-03T21:11:08.876 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:02.164+0000 7f2ac49e5700 1 -- 172.21.15.67:0/4026012023 wait complete. 2024-05-03T21:11:08.876 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:02.164+0000 7f2ac49e5700 1 Processor -- start 2024-05-03T21:11:08.876 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:02.164+0000 7f2ac49e5700 1 -- start start 2024-05-03T21:11:08.876 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:02.165+0000 7f2ac49e5700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2ac0109dc0 0x7f2ac019d310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:11:08.876 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:02.165+0000 7f2ac49e5700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f2ac010d710 con 0x7f2ac0109dc0 2024-05-03T21:11:08.877 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:02.165+0000 7f2abe59c700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2ac0109dc0 0x7f2ac019d310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:11:08.877 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:02.165+0000 7f2abe59c700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2ac0109dc0 0x7f2ac019d310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:46360/0 (socket says 172.21.15.67:46360) 2024-05-03T21:11:08.877 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:02.165+0000 7f2abe59c700 1 -- 172.21.15.67:0/905055003 learned_addr learned my addr 172.21.15.67:0/905055003 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-03T21:11:08.877 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:02.165+0000 7f2abe59c700 1 -- 172.21.15.67:0/905055003 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2ac019d850 con 0x7f2ac0109dc0 2024-05-03T21:11:08.877 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:02.165+0000 7f2abe59c700 1 --2- 172.21.15.67:0/905055003 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2ac0109dc0 0x7f2ac019d310 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f2ab0011bd0 tx=0x7f2ab0011cb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T21:11:08.877 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:02.165+0000 7f2aaf7fe700 1 -- 172.21.15.67:0/905055003 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2ab0003970 con 0x7f2ac0109dc0 2024-05-03T21:11:08.877 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:02.165+0000 7f2aaf7fe700 1 -- 172.21.15.67:0/905055003 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(28 keys) ==== 1139+0+0 (secure 0 0 0) 0x7f2ab0003b10 con 0x7f2ac0109dc0 2024-05-03T21:11:08.877 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:02.165+0000 7f2ac49e5700 1 -- 172.21.15.67:0/905055003 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2ac019dae0 con 0x7f2ac0109dc0 2024-05-03T21:11:08.877 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:02.166+0000 7f2aaf7fe700 1 -- 172.21.15.67:0/905055003 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2ab00245a0 con 0x7f2ac0109dc0 2024-05-03T21:11:08.877 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:02.166+0000 7f2ac49e5700 1 -- 172.21.15.67:0/905055003 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2ac01a07d0 con 0x7f2ac0109dc0 2024-05-03T21:11:08.877 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:02.166+0000 7f2aaf7fe700 1 -- 172.21.15.67:0/905055003 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 50222+0+0 (secure 0 0 0) 0x7f2ab0024920 con 0x7f2ac0109dc0 2024-05-03T21:11:08.877 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:02.167+0000 7f2aaf7fe700 1 --2- 172.21.15.67:0/905055003 >> [v2:172.21.15.67:6800/1698171509,v1:172.21.15.67:6801/1698171509] conn(0x7f2aa803e470 0x7f2aa8040930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:11:08.877 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:02.167+0000 7f2ac49e5700 1 -- 172.21.15.67:0/905055003 --> [v2:172.21.15.67:6800/1698171509,v1:172.21.15.67:6801/1698171509] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f2ac0064b00 con 0x7f2aa803e470 2024-05-03T21:11:08.877 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:02.167+0000 7f2aaf7fe700 1 -- 172.21.15.67:0/905055003 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f2ab0054880 con 0x7f2ac0109dc0 2024-05-03T21:11:08.878 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:02.167+0000 7f2abdd9b700 1 -- 172.21.15.67:0/905055003 >> [v2:172.21.15.67:6800/1698171509,v1:172.21.15.67:6801/1698171509] conn(0x7f2aa803e470 msgr2=0x7f2aa8040930 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.67:6800/1698171509 2024-05-03T21:11:08.878 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:02.167+0000 7f2abdd9b700 1 --2- 172.21.15.67:0/905055003 >> [v2:172.21.15.67:6800/1698171509,v1:172.21.15.67:6801/1698171509] conn(0x7f2aa803e470 0x7f2aa8040930 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-03T21:11:08.878 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:02.368+0000 7f2abdd9b700 1 -- 172.21.15.67:0/905055003 >> [v2:172.21.15.67:6800/1698171509,v1:172.21.15.67:6801/1698171509] conn(0x7f2aa803e470 msgr2=0x7f2aa8040930 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.67:6800/1698171509 2024-05-03T21:11:08.878 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:02.368+0000 7f2abdd9b700 1 --2- 172.21.15.67:0/905055003 >> [v2:172.21.15.67:6800/1698171509,v1:172.21.15.67:6801/1698171509] conn(0x7f2aa803e470 0x7f2aa8040930 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-05-03T21:11:08.878 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:02.769+0000 7f2abdd9b700 1 -- 172.21.15.67:0/905055003 >> [v2:172.21.15.67:6800/1698171509,v1:172.21.15.67:6801/1698171509] conn(0x7f2aa803e470 msgr2=0x7f2aa8040930 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.67:6800/1698171509 2024-05-03T21:11:08.878 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:02.769+0000 7f2abdd9b700 1 --2- 172.21.15.67:0/905055003 >> [v2:172.21.15.67:6800/1698171509,v1:172.21.15.67:6801/1698171509] conn(0x7f2aa803e470 0x7f2aa8040930 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-05-03T21:11:08.878 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:03.570+0000 7f2abdd9b700 1 -- 172.21.15.67:0/905055003 >> [v2:172.21.15.67:6800/1698171509,v1:172.21.15.67:6801/1698171509] conn(0x7f2aa803e470 msgr2=0x7f2aa8040930 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.67:6800/1698171509 2024-05-03T21:11:08.878 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:03.570+0000 7f2abdd9b700 1 --2- 172.21.15.67:0/905055003 >> [v2:172.21.15.67:6800/1698171509,v1:172.21.15.67:6801/1698171509] conn(0x7f2aa803e470 0x7f2aa8040930 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-05-03T21:11:08.878 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:05.171+0000 7f2abdd9b700 1 -- 172.21.15.67:0/905055003 >> [v2:172.21.15.67:6800/1698171509,v1:172.21.15.67:6801/1698171509] conn(0x7f2aa803e470 msgr2=0x7f2aa8040930 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.67:6800/1698171509 2024-05-03T21:11:08.878 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:05.171+0000 7f2abdd9b700 1 --2- 172.21.15.67:0/905055003 >> [v2:172.21.15.67:6800/1698171509,v1:172.21.15.67:6801/1698171509] conn(0x7f2aa803e470 0x7f2aa8040930 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-05-03T21:11:08.878 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:07.335+0000 7f2aaf7fe700 1 -- 172.21.15.67:0/905055003 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mgrmap(e 6) ==== 49989+0+0 (secure 0 0 0) 0x7f2ab002d3f0 con 0x7f2ac0109dc0 2024-05-03T21:11:08.878 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:07.335+0000 7f2aaf7fe700 1 -- 172.21.15.67:0/905055003 >> [v2:172.21.15.67:6800/1698171509,v1:172.21.15.67:6801/1698171509] conn(0x7f2aa803e470 msgr2=0x7f2aa8040930 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-03T21:11:08.879 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:07.335+0000 7f2aaf7fe700 1 --2- 172.21.15.67:0/905055003 >> [v2:172.21.15.67:6800/1698171509,v1:172.21.15.67:6801/1698171509] conn(0x7f2aa803e470 0x7f2aa8040930 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:11:08.879 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:08.336+0000 7f2aaf7fe700 1 -- 172.21.15.67:0/905055003 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mgrmap(e 7) ==== 50116+0+0 (secure 0 0 0) 0x7f2ab0018350 con 0x7f2ac0109dc0 2024-05-03T21:11:08.879 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:08.336+0000 7f2aaf7fe700 1 --2- 172.21.15.67:0/905055003 >> [v2:172.21.15.67:6800/2576113038,v1:172.21.15.67:6801/2576113038] conn(0x7f2aa8041b70 0x7f2aa8043f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:11:08.879 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:08.336+0000 7f2aaf7fe700 1 -- 172.21.15.67:0/905055003 --> [v2:172.21.15.67:6800/2576113038,v1:172.21.15.67:6801/2576113038] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f2ac0064b00 con 0x7f2aa8041b70 2024-05-03T21:11:08.879 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:08.336+0000 7f2abdd9b700 1 --2- 172.21.15.67:0/905055003 >> [v2:172.21.15.67:6800/2576113038,v1:172.21.15.67:6801/2576113038] conn(0x7f2aa8041b70 0x7f2aa8043f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:11:08.879 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:08.337+0000 7f2abdd9b700 1 --2- 172.21.15.67:0/905055003 >> [v2:172.21.15.67:6800/2576113038,v1:172.21.15.67:6801/2576113038] conn(0x7f2aa8041b70 0x7f2aa8043f60 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f2ab4003a10 tx=0x7f2ab4009290 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T21:11:08.879 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:08.338+0000 7f2aaf7fe700 1 -- 172.21.15.67:0/905055003 <== mgr.14118 v2:172.21.15.67:6800/2576113038 1 ==== command_reply(tid 0: 0 ) ==== 8+0+9564 (secure 0 0 0) 0x7f2ac0064b00 con 0x7f2aa8041b70 2024-05-03T21:11:08.879 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:08.346+0000 7f2ac49e5700 1 -- 172.21.15.67:0/905055003 --> [v2:172.21.15.67:6800/2576113038,v1:172.21.15.67:6801/2576113038] -- command(tid 1: {"prefix": "mgr_status"}) -- 0x7f2ac0104aa0 con 0x7f2aa8041b70 2024-05-03T21:11:08.879 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:08.346+0000 7f2aaf7fe700 1 -- 172.21.15.67:0/905055003 <== mgr.14118 v2:172.21.15.67:6800/2576113038 2 ==== command_reply(tid 1: 0 ) ==== 8+0+51 (secure 0 0 0) 0x7f2ac0104aa0 con 0x7f2aa8041b70 2024-05-03T21:11:08.879 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:08.346+0000 7f2ac49e5700 1 -- 172.21.15.67:0/905055003 >> [v2:172.21.15.67:6800/2576113038,v1:172.21.15.67:6801/2576113038] conn(0x7f2aa8041b70 msgr2=0x7f2aa8043f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:11:08.879 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:08.346+0000 7f2ac49e5700 1 --2- 172.21.15.67:0/905055003 >> [v2:172.21.15.67:6800/2576113038,v1:172.21.15.67:6801/2576113038] conn(0x7f2aa8041b70 0x7f2aa8043f60 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f2ab4003a10 tx=0x7f2ab4009290 comp rx=0 tx=0).stop 2024-05-03T21:11:08.879 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:08.346+0000 7f2ac49e5700 1 -- 172.21.15.67:0/905055003 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2ac0109dc0 msgr2=0x7f2ac019d310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:11:08.879 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:08.346+0000 7f2ac49e5700 1 --2- 172.21.15.67:0/905055003 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2ac0109dc0 0x7f2ac019d310 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f2ab0011bd0 tx=0x7f2ab0011cb0 comp rx=0 tx=0).stop 2024-05-03T21:11:08.879 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:08.346+0000 7f2ac49e5700 1 -- 172.21.15.67:0/905055003 shutdown_connections 2024-05-03T21:11:08.880 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:08.346+0000 7f2ac49e5700 1 --2- 172.21.15.67:0/905055003 >> [v2:172.21.15.67:6800/2576113038,v1:172.21.15.67:6801/2576113038] conn(0x7f2aa8041b70 0x7f2aa8043f60 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:11:08.880 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:08.346+0000 7f2ac49e5700 1 --2- 172.21.15.67:0/905055003 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2ac0109dc0 0x7f2ac019d310 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:11:08.880 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:08.346+0000 7f2ac49e5700 1 --2- 172.21.15.67:0/905055003 >> [v2:172.21.15.67:6800/1698171509,v1:172.21.15.67:6801/1698171509] conn(0x7f2aa803e470 0x7f2aa8040930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:11:08.880 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:08.346+0000 7f2ac49e5700 1 -- 172.21.15.67:0/905055003 >> 172.21.15.67:0/905055003 conn(0x7f2ac0103790 msgr2=0x7f2ac0104390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T21:11:08.880 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:08.347+0000 7f2ac49e5700 1 -- 172.21.15.67:0/905055003 shutdown_connections 2024-05-03T21:11:08.880 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:08.347+0000 7f2ac49e5700 1 -- 172.21.15.67:0/905055003 wait complete. 2024-05-03T21:11:08.880 INFO:teuthology.orchestra.run.smithi067.stdout:mgr epoch 5 is available 2024-05-03T21:11:08.880 INFO:teuthology.orchestra.run.smithi067.stdout:Setting orchestrator backend to cephadm... 2024-05-03T21:11:09.396 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:11:09 smithi067 ceph-mon[29730]: [03/May/2024:21:11:08] ENGINE Bus STARTING 2024-05-03T21:11:09.397 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:11:09 smithi067 ceph-mon[29730]: [03/May/2024:21:11:08] ENGINE Serving on http://172.21.15.67:8765 2024-05-03T21:11:09.397 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:11:09 smithi067 ceph-mon[29730]: from='client.14122 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-05-03T21:11:09.397 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:11:09 smithi067 ceph-mon[29730]: from='client.14122 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-05-03T21:11:09.397 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:11:09 smithi067 ceph-mon[29730]: [03/May/2024:21:11:08] ENGINE Serving on https://172.21.15.67:7150 2024-05-03T21:11:09.397 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:11:09 smithi067 ceph-mon[29730]: [03/May/2024:21:11:08] ENGINE Bus STARTED 2024-05-03T21:11:09.397 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:11:09 smithi067 ceph-mon[29730]: from='mgr.14118 172.21.15.67:0/4082808469' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T21:11:10.038 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:09.243+0000 7fe51de6b700 1 Processor -- start 2024-05-03T21:11:10.038 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:09.243+0000 7fe51de6b700 1 -- start start 2024-05-03T21:11:10.038 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:09.243+0000 7fe51de6b700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe518109270 0x7fe518109690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:11:10.038 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:09.243+0000 7fe51de6b700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fe518109bd0 con 0x7fe518109270 2024-05-03T21:11:10.038 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:09.244+0000 7fe5177fe700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe518109270 0x7fe518109690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:11:10.038 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:09.244+0000 7fe5177fe700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe518109270 0x7fe518109690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:43776/0 (socket says 172.21.15.67:43776) 2024-05-03T21:11:10.038 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:09.244+0000 7fe5177fe700 1 -- 172.21.15.67:0/374382071 learned_addr learned my addr 172.21.15.67:0/374382071 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-03T21:11:10.038 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:09.244+0000 7fe5177fe700 1 -- 172.21.15.67:0/374382071 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe518109d50 con 0x7fe518109270 2024-05-03T21:11:10.038 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:09.244+0000 7fe5177fe700 1 --2- 172.21.15.67:0/374382071 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe518109270 0x7fe518109690 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fe508009e80 tx=0x7fe50800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6897f0b1fc185cfc server_cookie=0 in_seq=0 out_seq=0 2024-05-03T21:11:10.039 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:09.244+0000 7fe5167fc700 1 -- 172.21.15.67:0/374382071 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe508017070 con 0x7fe518109270 2024-05-03T21:11:10.039 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:09.244+0000 7fe5167fc700 1 -- 172.21.15.67:0/374382071 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(28 keys) ==== 1139+0+0 (secure 0 0 0) 0x7fe50800b8c0 con 0x7fe518109270 2024-05-03T21:11:10.039 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:09.244+0000 7fe5167fc700 1 -- 172.21.15.67:0/374382071 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe508004620 con 0x7fe518109270 2024-05-03T21:11:10.039 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:09.245+0000 7fe51de6b700 1 -- 172.21.15.67:0/374382071 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe518109270 msgr2=0x7fe518109690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:11:10.039 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:09.245+0000 7fe51de6b700 1 --2- 172.21.15.67:0/374382071 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe518109270 0x7fe518109690 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fe508009e80 tx=0x7fe50800b330 comp rx=0 tx=0).stop 2024-05-03T21:11:10.039 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:09.245+0000 7fe51de6b700 1 -- 172.21.15.67:0/374382071 shutdown_connections 2024-05-03T21:11:10.039 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:09.245+0000 7fe51de6b700 1 --2- 172.21.15.67:0/374382071 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe518109270 0x7fe518109690 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:11:10.039 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:09.245+0000 7fe51de6b700 1 -- 172.21.15.67:0/374382071 >> 172.21.15.67:0/374382071 conn(0x7fe518104950 msgr2=0x7fe518106d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T21:11:10.039 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:09.245+0000 7fe51de6b700 1 -- 172.21.15.67:0/374382071 shutdown_connections 2024-05-03T21:11:10.039 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:09.245+0000 7fe51de6b700 1 -- 172.21.15.67:0/374382071 wait complete. 2024-05-03T21:11:10.039 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:09.245+0000 7fe51de6b700 1 Processor -- start 2024-05-03T21:11:10.039 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:09.246+0000 7fe51de6b700 1 -- start start 2024-05-03T21:11:10.039 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:09.246+0000 7fe51de6b700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe518109270 0x7fe51819d1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:11:10.039 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:09.246+0000 7fe51de6b700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fe51810d6e0 con 0x7fe518109270 2024-05-03T21:11:10.040 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:09.246+0000 7fe5177fe700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe518109270 0x7fe51819d1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:11:10.040 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:09.246+0000 7fe5177fe700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe518109270 0x7fe51819d1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:43778/0 (socket says 172.21.15.67:43778) 2024-05-03T21:11:10.040 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:09.246+0000 7fe5177fe700 1 -- 172.21.15.67:0/2394836913 learned_addr learned my addr 172.21.15.67:0/2394836913 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-03T21:11:10.040 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:09.246+0000 7fe5177fe700 1 -- 172.21.15.67:0/2394836913 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe51819d6f0 con 0x7fe518109270 2024-05-03T21:11:10.040 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:09.246+0000 7fe5177fe700 1 --2- 172.21.15.67:0/2394836913 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe518109270 0x7fe51819d1b0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fe50800b7b0 tx=0x7fe508003d00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T21:11:10.040 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:09.247+0000 7fe514ff9700 1 -- 172.21.15.67:0/2394836913 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe508021020 con 0x7fe518109270 2024-05-03T21:11:10.040 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:09.247+0000 7fe514ff9700 1 -- 172.21.15.67:0/2394836913 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(28 keys) ==== 1139+0+0 (secure 0 0 0) 0x7fe50800be00 con 0x7fe518109270 2024-05-03T21:11:10.040 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:09.247+0000 7fe51de6b700 1 -- 172.21.15.67:0/2394836913 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe51819d980 con 0x7fe518109270 2024-05-03T21:11:10.040 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:09.247+0000 7fe51de6b700 1 -- 172.21.15.67:0/2394836913 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe5181a0670 con 0x7fe518109270 2024-05-03T21:11:10.040 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:09.247+0000 7fe514ff9700 1 -- 172.21.15.67:0/2394836913 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe508017040 con 0x7fe518109270 2024-05-03T21:11:10.041 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:09.248+0000 7fe514ff9700 1 -- 172.21.15.67:0/2394836913 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 7) ==== 50116+0+0 (secure 0 0 0) 0x7fe508024020 con 0x7fe518109270 2024-05-03T21:11:10.041 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:09.248+0000 7fe51de6b700 1 -- 172.21.15.67:0/2394836913 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe4f8005360 con 0x7fe518109270 2024-05-03T21:11:10.041 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:09.248+0000 7fe514ff9700 1 --2- 172.21.15.67:0/2394836913 >> [v2:172.21.15.67:6800/2576113038,v1:172.21.15.67:6801/2576113038] conn(0x7fe50403e350 0x7fe504040810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:11:10.041 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:09.248+0000 7fe514ff9700 1 -- 172.21.15.67:0/2394836913 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7fe50800f050 con 0x7fe518109270 2024-05-03T21:11:10.041 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:09.248+0000 7fe516ffd700 1 --2- 172.21.15.67:0/2394836913 >> [v2:172.21.15.67:6800/2576113038,v1:172.21.15.67:6801/2576113038] conn(0x7fe50403e350 0x7fe504040810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:11:10.041 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:09.248+0000 7fe516ffd700 1 --2- 172.21.15.67:0/2394836913 >> [v2:172.21.15.67:6800/2576113038,v1:172.21.15.67:6801/2576113038] conn(0x7fe50403e350 0x7fe504040810 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fe500006fd0 tx=0x7fe500006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T21:11:10.041 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:09.253+0000 7fe514ff9700 1 -- 172.21.15.67:0/2394836913 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191839 (secure 0 0 0) 0x7fe50802b020 con 0x7fe518109270 2024-05-03T21:11:10.041 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:09.394+0000 7fe514ff9700 1 -- 172.21.15.67:0/2394836913 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mgrmap(e 8) ==== 50222+0+0 (secure 0 0 0) 0x7fe50802ecf0 con 0x7fe518109270 2024-05-03T21:11:10.041 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:09.485+0000 7fe51de6b700 1 -- 172.21.15.67:0/2394836913 --> [v2:172.21.15.67:6800/2576113038,v1:172.21.15.67:6801/2576113038] -- mgr_command(tid 0: {"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}) -- 0x7fe4f8000bf0 con 0x7fe50403e350 2024-05-03T21:11:10.041 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:09.490+0000 7fe514ff9700 1 -- 172.21.15.67:0/2394836913 <== mgr.14118 v2:172.21.15.67:6800/2576113038 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7fe4f8000bf0 con 0x7fe50403e350 2024-05-03T21:11:10.041 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:09.493+0000 7fe51de6b700 1 -- 172.21.15.67:0/2394836913 >> [v2:172.21.15.67:6800/2576113038,v1:172.21.15.67:6801/2576113038] conn(0x7fe50403e350 msgr2=0x7fe504040810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:11:10.041 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:09.493+0000 7fe51de6b700 1 --2- 172.21.15.67:0/2394836913 >> [v2:172.21.15.67:6800/2576113038,v1:172.21.15.67:6801/2576113038] conn(0x7fe50403e350 0x7fe504040810 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fe500006fd0 tx=0x7fe500006e40 comp rx=0 tx=0).stop 2024-05-03T21:11:10.041 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:09.493+0000 7fe51de6b700 1 -- 172.21.15.67:0/2394836913 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe518109270 msgr2=0x7fe51819d1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:11:10.042 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:09.493+0000 7fe51de6b700 1 --2- 172.21.15.67:0/2394836913 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe518109270 0x7fe51819d1b0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fe50800b7b0 tx=0x7fe508003d00 comp rx=0 tx=0).stop 2024-05-03T21:11:10.042 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:09.493+0000 7fe51de6b700 1 -- 172.21.15.67:0/2394836913 shutdown_connections 2024-05-03T21:11:10.042 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:09.493+0000 7fe51de6b700 1 --2- 172.21.15.67:0/2394836913 >> [v2:172.21.15.67:6800/2576113038,v1:172.21.15.67:6801/2576113038] conn(0x7fe50403e350 0x7fe504040810 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:11:10.042 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:09.493+0000 7fe51de6b700 1 --2- 172.21.15.67:0/2394836913 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe518109270 0x7fe51819d1b0 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:11:10.042 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:09.493+0000 7fe51de6b700 1 -- 172.21.15.67:0/2394836913 >> 172.21.15.67:0/2394836913 conn(0x7fe518104950 msgr2=0x7fe5181053f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T21:11:10.042 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:09.493+0000 7fe51de6b700 1 -- 172.21.15.67:0/2394836913 shutdown_connections 2024-05-03T21:11:10.042 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:09.493+0000 7fe51de6b700 1 -- 172.21.15.67:0/2394836913 wait complete. 2024-05-03T21:11:10.428 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:11:10 smithi067 ceph-mon[29730]: mgrmap e8: x(active, since 2s) 2024-05-03T21:11:10.428 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:11:10 smithi067 ceph-mon[29730]: from='client.14130 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2024-05-03T21:11:10.428 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:11:10 smithi067 ceph-mon[29730]: from='mgr.14118 172.21.15.67:0/4082808469' entity='mgr.x' 2024-05-03T21:11:10.428 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:11:10 smithi067 ceph-mon[29730]: from='mgr.14118 172.21.15.67:0/4082808469' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T21:11:11.219 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout value unchanged 2024-05-03T21:11:11.219 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:10.426+0000 7f56caa9a700 1 Processor -- start 2024-05-03T21:11:11.219 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:10.426+0000 7f56caa9a700 1 -- start start 2024-05-03T21:11:11.219 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:10.426+0000 7f56caa9a700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f56c41081b0 0x7f56c41085d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:11:11.219 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:10.426+0000 7f56caa9a700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f56c4108ba0 con 0x7f56c41081b0 2024-05-03T21:11:11.219 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:10.427+0000 7f56c8836700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f56c41081b0 0x7f56c41085d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:11:11.219 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:10.427+0000 7f56c8836700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f56c41081b0 0x7f56c41085d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:43794/0 (socket says 172.21.15.67:43794) 2024-05-03T21:11:11.219 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:10.427+0000 7f56c8836700 1 -- 172.21.15.67:0/147210659 learned_addr learned my addr 172.21.15.67:0/147210659 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-03T21:11:11.219 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:10.427+0000 7f56c8836700 1 -- 172.21.15.67:0/147210659 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f56c41093b0 con 0x7f56c41081b0 2024-05-03T21:11:11.219 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:10.427+0000 7f56c8836700 1 --2- 172.21.15.67:0/147210659 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f56c41081b0 0x7f56c41085d0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f56b80098a0 tx=0x7f56b8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ef473e471f7a1d19 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T21:11:11.219 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:10.427+0000 7f56c37fe700 1 -- 172.21.15.67:0/147210659 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f56b8004160 con 0x7f56c41081b0 2024-05-03T21:11:11.220 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:10.427+0000 7f56c37fe700 1 -- 172.21.15.67:0/147210659 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(28 keys) ==== 1139+0+0 (secure 0 0 0) 0x7f56b8004300 con 0x7f56c41081b0 2024-05-03T21:11:11.220 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:10.427+0000 7f56c37fe700 1 -- 172.21.15.67:0/147210659 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f56b8011c00 con 0x7f56c41081b0 2024-05-03T21:11:11.220 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:10.428+0000 7f56caa9a700 1 -- 172.21.15.67:0/147210659 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f56c41081b0 msgr2=0x7f56c41085d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:11:11.220 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:10.428+0000 7f56caa9a700 1 --2- 172.21.15.67:0/147210659 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f56c41081b0 0x7f56c41085d0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f56b80098a0 tx=0x7f56b8009bb0 comp rx=0 tx=0).stop 2024-05-03T21:11:11.220 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:10.428+0000 7f56caa9a700 1 -- 172.21.15.67:0/147210659 shutdown_connections 2024-05-03T21:11:11.220 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:10.428+0000 7f56caa9a700 1 --2- 172.21.15.67:0/147210659 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f56c41081b0 0x7f56c41085d0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:11:11.220 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:10.428+0000 7f56caa9a700 1 -- 172.21.15.67:0/147210659 >> 172.21.15.67:0/147210659 conn(0x7f56c4103790 msgr2=0x7f56c4105bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T21:11:11.220 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:10.428+0000 7f56caa9a700 1 -- 172.21.15.67:0/147210659 shutdown_connections 2024-05-03T21:11:11.220 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:10.428+0000 7f56caa9a700 1 -- 172.21.15.67:0/147210659 wait complete. 2024-05-03T21:11:11.220 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:10.429+0000 7f56caa9a700 1 Processor -- start 2024-05-03T21:11:11.220 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:10.429+0000 7f56caa9a700 1 -- start start 2024-05-03T21:11:11.220 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:10.429+0000 7f56caa9a700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f56c41081b0 0x7f56c419f530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:11:11.220 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:10.429+0000 7f56caa9a700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f56c410c760 con 0x7f56c41081b0 2024-05-03T21:11:11.220 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:10.429+0000 7f56c8836700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f56c41081b0 0x7f56c419f530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:11:11.220 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:10.429+0000 7f56c8836700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f56c41081b0 0x7f56c419f530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:43802/0 (socket says 172.21.15.67:43802) 2024-05-03T21:11:11.221 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:10.429+0000 7f56c8836700 1 -- 172.21.15.67:0/3112173468 learned_addr learned my addr 172.21.15.67:0/3112173468 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-03T21:11:11.221 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:10.429+0000 7f56c8836700 1 -- 172.21.15.67:0/3112173468 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f56c419fa70 con 0x7f56c41081b0 2024-05-03T21:11:11.221 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:10.429+0000 7f56c8836700 1 --2- 172.21.15.67:0/3112173468 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f56c41081b0 0x7f56c419f530 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f56b8009fb0 tx=0x7f56b8011c60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T21:11:11.221 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:10.430+0000 7f56c1ffb700 1 -- 172.21.15.67:0/3112173468 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f56b80038b0 con 0x7f56c41081b0 2024-05-03T21:11:11.221 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:10.430+0000 7f56caa9a700 1 -- 172.21.15.67:0/3112173468 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f56c419fd00 con 0x7f56c41081b0 2024-05-03T21:11:11.221 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:10.430+0000 7f56c1ffb700 1 -- 172.21.15.67:0/3112173468 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(28 keys) ==== 1139+0+0 (secure 0 0 0) 0x7f56b801a430 con 0x7f56c41081b0 2024-05-03T21:11:11.221 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:10.430+0000 7f56c1ffb700 1 -- 172.21.15.67:0/3112173468 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f56b80195e0 con 0x7f56c41081b0 2024-05-03T21:11:11.221 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:10.430+0000 7f56caa9a700 1 -- 172.21.15.67:0/3112173468 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f56c41a29f0 con 0x7f56c41081b0 2024-05-03T21:11:11.221 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:10.431+0000 7f56c1ffb700 1 -- 172.21.15.67:0/3112173468 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 8) ==== 50222+0+0 (secure 0 0 0) 0x7f56b801aa50 con 0x7f56c41081b0 2024-05-03T21:11:11.221 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:10.431+0000 7f56caa9a700 1 -- 172.21.15.67:0/3112173468 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f56c410c8e0 con 0x7f56c41081b0 2024-05-03T21:11:11.221 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:10.431+0000 7f56c1ffb700 1 --2- 172.21.15.67:0/3112173468 >> [v2:172.21.15.67:6800/2576113038,v1:172.21.15.67:6801/2576113038] conn(0x7f56b403e470 0x7f56b4040930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:11:11.221 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:10.431+0000 7f56c1ffb700 1 -- 172.21.15.67:0/3112173468 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f56b8053920 con 0x7f56c41081b0 2024-05-03T21:11:11.222 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:10.431+0000 7f56c3fff700 1 --2- 172.21.15.67:0/3112173468 >> [v2:172.21.15.67:6800/2576113038,v1:172.21.15.67:6801/2576113038] conn(0x7f56b403e470 0x7f56b4040930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:11:11.222 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:10.431+0000 7f56c3fff700 1 --2- 172.21.15.67:0/3112173468 >> [v2:172.21.15.67:6800/2576113038,v1:172.21.15.67:6801/2576113038] conn(0x7f56b403e470 0x7f56b4040930 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f56b0006fd0 tx=0x7f56b0006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T21:11:11.222 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:10.436+0000 7f56c1ffb700 1 -- 172.21.15.67:0/3112173468 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191839 (secure 0 0 0) 0x7f56b8003be0 con 0x7f56c41081b0 2024-05-03T21:11:11.222 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:10.658+0000 7f56caa9a700 1 -- 172.21.15.67:0/3112173468 --> [v2:172.21.15.67:6800/2576113038,v1:172.21.15.67:6801/2576113038] -- mgr_command(tid 0: {"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}) -- 0x7f56c402e840 con 0x7f56b403e470 2024-05-03T21:11:11.222 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:10.659+0000 7f56c1ffb700 1 -- 172.21.15.67:0/3112173468 <== mgr.14118 v2:172.21.15.67:6800/2576113038 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+16 (secure 0 0 0) 0x7f56c402e840 con 0x7f56b403e470 2024-05-03T21:11:11.222 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:10.661+0000 7f56caa9a700 1 -- 172.21.15.67:0/3112173468 >> [v2:172.21.15.67:6800/2576113038,v1:172.21.15.67:6801/2576113038] conn(0x7f56b403e470 msgr2=0x7f56b4040930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:11:11.222 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:10.661+0000 7f56caa9a700 1 --2- 172.21.15.67:0/3112173468 >> [v2:172.21.15.67:6800/2576113038,v1:172.21.15.67:6801/2576113038] conn(0x7f56b403e470 0x7f56b4040930 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f56b0006fd0 tx=0x7f56b0006e40 comp rx=0 tx=0).stop 2024-05-03T21:11:11.222 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:10.661+0000 7f56caa9a700 1 -- 172.21.15.67:0/3112173468 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f56c41081b0 msgr2=0x7f56c419f530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:11:11.222 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:10.661+0000 7f56caa9a700 1 --2- 172.21.15.67:0/3112173468 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f56c41081b0 0x7f56c419f530 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f56b8009fb0 tx=0x7f56b8011c60 comp rx=0 tx=0).stop 2024-05-03T21:11:11.222 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:10.661+0000 7f56caa9a700 1 -- 172.21.15.67:0/3112173468 shutdown_connections 2024-05-03T21:11:11.222 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:10.661+0000 7f56caa9a700 1 --2- 172.21.15.67:0/3112173468 >> [v2:172.21.15.67:6800/2576113038,v1:172.21.15.67:6801/2576113038] conn(0x7f56b403e470 0x7f56b4040930 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:11:11.222 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:10.661+0000 7f56caa9a700 1 --2- 172.21.15.67:0/3112173468 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f56c41081b0 0x7f56c419f530 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:11:11.222 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:10.661+0000 7f56caa9a700 1 -- 172.21.15.67:0/3112173468 >> 172.21.15.67:0/3112173468 conn(0x7f56c4103790 msgr2=0x7f56c41046b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T21:11:11.222 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:10.661+0000 7f56caa9a700 1 -- 172.21.15.67:0/3112173468 shutdown_connections 2024-05-03T21:11:11.223 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:10.661+0000 7f56caa9a700 1 -- 172.21.15.67:0/3112173468 wait complete. 2024-05-03T21:11:11.223 INFO:teuthology.orchestra.run.smithi067.stdout:Generating ssh key... 2024-05-03T21:11:11.621 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:11:11 smithi067 ceph-mon[29730]: from='client.14132 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2024-05-03T21:11:12.221 INFO:journalctl@ceph.mgr.x.smithi067.stdout:May 03 21:11:12 smithi067 ceph-713bc57a-0991-11ef-bc97-c7b262605968-mgr-x[29964]: Generating public/private rsa key pair. 2024-05-03T21:11:12.221 INFO:journalctl@ceph.mgr.x.smithi067.stdout:May 03 21:11:12 smithi067 ceph-713bc57a-0991-11ef-bc97-c7b262605968-mgr-x[29964]: Your identification has been saved in /tmp/tmp1pr6gqk3/key. 2024-05-03T21:11:12.221 INFO:journalctl@ceph.mgr.x.smithi067.stdout:May 03 21:11:12 smithi067 ceph-713bc57a-0991-11ef-bc97-c7b262605968-mgr-x[29964]: Your public key has been saved in /tmp/tmp1pr6gqk3/key.pub. 2024-05-03T21:11:12.221 INFO:journalctl@ceph.mgr.x.smithi067.stdout:May 03 21:11:12 smithi067 ceph-713bc57a-0991-11ef-bc97-c7b262605968-mgr-x[29964]: The key fingerprint is: 2024-05-03T21:11:12.221 INFO:journalctl@ceph.mgr.x.smithi067.stdout:May 03 21:11:12 smithi067 ceph-713bc57a-0991-11ef-bc97-c7b262605968-mgr-x[29964]: SHA256:26IKJBGZC4R4a6SaZ5Z0lO7/Ug1EcnS7E6utR5h2YLM ceph-713bc57a-0991-11ef-bc97-c7b262605968 2024-05-03T21:11:12.221 INFO:journalctl@ceph.mgr.x.smithi067.stdout:May 03 21:11:12 smithi067 ceph-713bc57a-0991-11ef-bc97-c7b262605968-mgr-x[29964]: The key's randomart image is: 2024-05-03T21:11:12.222 INFO:journalctl@ceph.mgr.x.smithi067.stdout:May 03 21:11:12 smithi067 ceph-713bc57a-0991-11ef-bc97-c7b262605968-mgr-x[29964]: +---[RSA 3072]----+ 2024-05-03T21:11:12.222 INFO:journalctl@ceph.mgr.x.smithi067.stdout:May 03 21:11:12 smithi067 ceph-713bc57a-0991-11ef-bc97-c7b262605968-mgr-x[29964]: |=+ ..o+ . | 2024-05-03T21:11:12.222 INFO:journalctl@ceph.mgr.x.smithi067.stdout:May 03 21:11:12 smithi067 ceph-713bc57a-0991-11ef-bc97-c7b262605968-mgr-x[29964]: |*.o o o.. . | 2024-05-03T21:11:12.222 INFO:journalctl@ceph.mgr.x.smithi067.stdout:May 03 21:11:12 smithi067 ceph-713bc57a-0991-11ef-bc97-c7b262605968-mgr-x[29964]: |o= + . o | 2024-05-03T21:11:12.222 INFO:journalctl@ceph.mgr.x.smithi067.stdout:May 03 21:11:12 smithi067 ceph-713bc57a-0991-11ef-bc97-c7b262605968-mgr-x[29964]: |o.+ o = + | 2024-05-03T21:11:12.222 INFO:journalctl@ceph.mgr.x.smithi067.stdout:May 03 21:11:12 smithi067 ceph-713bc57a-0991-11ef-bc97-c7b262605968-mgr-x[29964]: |o+.+ .SO+ | 2024-05-03T21:11:12.222 INFO:journalctl@ceph.mgr.x.smithi067.stdout:May 03 21:11:12 smithi067 ceph-713bc57a-0991-11ef-bc97-c7b262605968-mgr-x[29964]: |oo= . E=+. | 2024-05-03T21:11:12.222 INFO:journalctl@ceph.mgr.x.smithi067.stdout:May 03 21:11:12 smithi067 ceph-713bc57a-0991-11ef-bc97-c7b262605968-mgr-x[29964]: | +. . o+oo | 2024-05-03T21:11:12.222 INFO:journalctl@ceph.mgr.x.smithi067.stdout:May 03 21:11:12 smithi067 ceph-713bc57a-0991-11ef-bc97-c7b262605968-mgr-x[29964]: | . o. o. | 2024-05-03T21:11:12.222 INFO:journalctl@ceph.mgr.x.smithi067.stdout:May 03 21:11:12 smithi067 ceph-713bc57a-0991-11ef-bc97-c7b262605968-mgr-x[29964]: | ...oo. | 2024-05-03T21:11:12.222 INFO:journalctl@ceph.mgr.x.smithi067.stdout:May 03 21:11:12 smithi067 ceph-713bc57a-0991-11ef-bc97-c7b262605968-mgr-x[29964]: +----[SHA256]-----+ 2024-05-03T21:11:12.608 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:11:12 smithi067 ceph-mon[29730]: from='client.14134 -' entity='client.admin' cmd=[{"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}]: dispatch 2024-05-03T21:11:12.608 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:11:12 smithi067 ceph-mon[29730]: Generating ssh key... 2024-05-03T21:11:12.608 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:11:12 smithi067 ceph-mon[29730]: from='mgr.14118 172.21.15.67:0/4082808469' entity='mgr.x' 2024-05-03T21:11:12.608 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:11:12 smithi067 ceph-mon[29730]: from='mgr.14118 172.21.15.67:0/4082808469' entity='mgr.x' 2024-05-03T21:11:12.616 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:11.657+0000 7f9e9421c700 1 Processor -- start 2024-05-03T21:11:12.616 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:11.657+0000 7f9e9421c700 1 -- start start 2024-05-03T21:11:12.616 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:11.657+0000 7f9e9421c700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9e8c107b30 0x7f9e8c107f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:11:12.616 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:11.658+0000 7f9e9421c700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f9e8c076f70 con 0x7f9e8c107b30 2024-05-03T21:11:12.616 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:11.658+0000 7f9e91fb8700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9e8c107b30 0x7f9e8c107f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:11:12.616 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:11.658+0000 7f9e91fb8700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9e8c107b30 0x7f9e8c107f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:43814/0 (socket says 172.21.15.67:43814) 2024-05-03T21:11:12.616 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:11.658+0000 7f9e91fb8700 1 -- 172.21.15.67:0/66785246 learned_addr learned my addr 172.21.15.67:0/66785246 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-03T21:11:12.616 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:11.658+0000 7f9e91fb8700 1 -- 172.21.15.67:0/66785246 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9e8c108490 con 0x7f9e8c107b30 2024-05-03T21:11:12.617 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:11.658+0000 7f9e91fb8700 1 --2- 172.21.15.67:0/66785246 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9e8c107b30 0x7f9e8c107f50 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f9e7c009e80 tx=0x7f9e7c00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ce4b71cd3d0e305a server_cookie=0 in_seq=0 out_seq=0 2024-05-03T21:11:12.617 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:11.658+0000 7f9e90fb6700 1 -- 172.21.15.67:0/66785246 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9e7c017070 con 0x7f9e8c107b30 2024-05-03T21:11:12.617 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:11.658+0000 7f9e90fb6700 1 -- 172.21.15.67:0/66785246 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(28 keys) ==== 1139+0+0 (secure 0 0 0) 0x7f9e7c00b8c0 con 0x7f9e8c107b30 2024-05-03T21:11:12.617 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:11.658+0000 7f9e90fb6700 1 -- 172.21.15.67:0/66785246 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9e7c004620 con 0x7f9e8c107b30 2024-05-03T21:11:12.617 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:11.659+0000 7f9e9421c700 1 -- 172.21.15.67:0/66785246 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9e8c107b30 msgr2=0x7f9e8c107f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:11:12.617 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:11.659+0000 7f9e9421c700 1 --2- 172.21.15.67:0/66785246 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9e8c107b30 0x7f9e8c107f50 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f9e7c009e80 tx=0x7f9e7c00b330 comp rx=0 tx=0).stop 2024-05-03T21:11:12.617 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:11.659+0000 7f9e9421c700 1 -- 172.21.15.67:0/66785246 shutdown_connections 2024-05-03T21:11:12.617 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:11.659+0000 7f9e9421c700 1 --2- 172.21.15.67:0/66785246 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9e8c107b30 0x7f9e8c107f50 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:11:12.617 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:11.659+0000 7f9e9421c700 1 -- 172.21.15.67:0/66785246 >> 172.21.15.67:0/66785246 conn(0x7f9e8c103750 msgr2=0x7f9e8c105b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T21:11:12.617 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:11.659+0000 7f9e9421c700 1 -- 172.21.15.67:0/66785246 shutdown_connections 2024-05-03T21:11:12.617 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:11.659+0000 7f9e9421c700 1 -- 172.21.15.67:0/66785246 wait complete. 2024-05-03T21:11:12.617 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:11.660+0000 7f9e9421c700 1 Processor -- start 2024-05-03T21:11:12.617 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:11.660+0000 7f9e9421c700 1 -- start start 2024-05-03T21:11:12.617 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:11.660+0000 7f9e9421c700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9e8c107b30 0x7f9e8c19f410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:11:12.618 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:11.660+0000 7f9e9421c700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f9e8c109580 con 0x7f9e8c107b30 2024-05-03T21:11:12.618 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:11.660+0000 7f9e91fb8700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9e8c107b30 0x7f9e8c19f410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:11:12.618 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:11.660+0000 7f9e91fb8700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9e8c107b30 0x7f9e8c19f410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:43826/0 (socket says 172.21.15.67:43826) 2024-05-03T21:11:12.618 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:11.660+0000 7f9e91fb8700 1 -- 172.21.15.67:0/1306175622 learned_addr learned my addr 172.21.15.67:0/1306175622 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-03T21:11:12.618 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:11.660+0000 7f9e91fb8700 1 -- 172.21.15.67:0/1306175622 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9e8c19f950 con 0x7f9e8c107b30 2024-05-03T21:11:12.618 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:11.661+0000 7f9e91fb8700 1 --2- 172.21.15.67:0/1306175622 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9e8c107b30 0x7f9e8c19f410 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f9e7c0043d0 tx=0x7f9e7c003d00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T21:11:12.618 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:11.661+0000 7f9e8affd700 1 -- 172.21.15.67:0/1306175622 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9e7c021020 con 0x7f9e8c107b30 2024-05-03T21:11:12.618 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:11.661+0000 7f9e8affd700 1 -- 172.21.15.67:0/1306175622 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(28 keys) ==== 1139+0+0 (secure 0 0 0) 0x7f9e7c0094d0 con 0x7f9e8c107b30 2024-05-03T21:11:12.618 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:11.661+0000 7f9e8affd700 1 -- 172.21.15.67:0/1306175622 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9e7c017040 con 0x7f9e8c107b30 2024-05-03T21:11:12.618 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:11.661+0000 7f9e9421c700 1 -- 172.21.15.67:0/1306175622 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9e8c19fbe0 con 0x7f9e8c107b30 2024-05-03T21:11:12.618 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:11.661+0000 7f9e9421c700 1 -- 172.21.15.67:0/1306175622 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9e8c1a28d0 con 0x7f9e8c107b30 2024-05-03T21:11:12.618 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:11.662+0000 7f9e8affd700 1 -- 172.21.15.67:0/1306175622 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 8) ==== 50222+0+0 (secure 0 0 0) 0x7f9e7c024020 con 0x7f9e8c107b30 2024-05-03T21:11:12.619 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:11.662+0000 7f9e9421c700 1 -- 172.21.15.67:0/1306175622 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9e8c064b00 con 0x7f9e8c107b30 2024-05-03T21:11:12.619 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:11.662+0000 7f9e8affd700 1 --2- 172.21.15.67:0/1306175622 >> [v2:172.21.15.67:6800/2576113038,v1:172.21.15.67:6801/2576113038] conn(0x7f9e7403e420 0x7f9e740408e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:11:12.619 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:11.662+0000 7f9e8affd700 1 -- 172.21.15.67:0/1306175622 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f9e7c00f050 con 0x7f9e8c107b30 2024-05-03T21:11:12.619 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:11.662+0000 7f9e917b7700 1 --2- 172.21.15.67:0/1306175622 >> [v2:172.21.15.67:6800/2576113038,v1:172.21.15.67:6801/2576113038] conn(0x7f9e7403e420 0x7f9e740408e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:11:12.619 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:11.663+0000 7f9e917b7700 1 --2- 172.21.15.67:0/1306175622 >> [v2:172.21.15.67:6800/2576113038,v1:172.21.15.67:6801/2576113038] conn(0x7f9e7403e420 0x7f9e740408e0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f9e80006fd0 tx=0x7f9e80006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T21:11:12.619 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:11.667+0000 7f9e8affd700 1 -- 172.21.15.67:0/1306175622 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191839 (secure 0 0 0) 0x7f9e7c02b020 con 0x7f9e8c107b30 2024-05-03T21:11:12.619 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:11.886+0000 7f9e9421c700 1 -- 172.21.15.67:0/1306175622 --> [v2:172.21.15.67:6800/2576113038,v1:172.21.15.67:6801/2576113038] -- mgr_command(tid 0: {"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}) -- 0x7f9e8c02e8f0 con 0x7f9e7403e420 2024-05-03T21:11:12.619 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:12.035+0000 7f9e8affd700 1 -- 172.21.15.67:0/1306175622 <== mgr.14118 v2:172.21.15.67:6800/2576113038 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f9e8c02e8f0 con 0x7f9e7403e420 2024-05-03T21:11:12.619 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:12.037+0000 7f9e9421c700 1 -- 172.21.15.67:0/1306175622 >> [v2:172.21.15.67:6800/2576113038,v1:172.21.15.67:6801/2576113038] conn(0x7f9e7403e420 msgr2=0x7f9e740408e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:11:12.619 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:12.037+0000 7f9e9421c700 1 --2- 172.21.15.67:0/1306175622 >> [v2:172.21.15.67:6800/2576113038,v1:172.21.15.67:6801/2576113038] conn(0x7f9e7403e420 0x7f9e740408e0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f9e80006fd0 tx=0x7f9e80006e40 comp rx=0 tx=0).stop 2024-05-03T21:11:12.619 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:12.037+0000 7f9e9421c700 1 -- 172.21.15.67:0/1306175622 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9e8c107b30 msgr2=0x7f9e8c19f410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:11:12.619 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:12.037+0000 7f9e9421c700 1 --2- 172.21.15.67:0/1306175622 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9e8c107b30 0x7f9e8c19f410 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f9e7c0043d0 tx=0x7f9e7c003d00 comp rx=0 tx=0).stop 2024-05-03T21:11:12.619 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:12.037+0000 7f9e9421c700 1 -- 172.21.15.67:0/1306175622 shutdown_connections 2024-05-03T21:11:12.620 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:12.037+0000 7f9e9421c700 1 --2- 172.21.15.67:0/1306175622 >> [v2:172.21.15.67:6800/2576113038,v1:172.21.15.67:6801/2576113038] conn(0x7f9e7403e420 0x7f9e740408e0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:11:12.620 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:12.037+0000 7f9e9421c700 1 --2- 172.21.15.67:0/1306175622 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9e8c107b30 0x7f9e8c19f410 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:11:12.620 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:12.037+0000 7f9e9421c700 1 -- 172.21.15.67:0/1306175622 >> 172.21.15.67:0/1306175622 conn(0x7f9e8c103750 msgr2=0x7f9e8c195fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T21:11:12.620 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:12.037+0000 7f9e9421c700 1 -- 172.21.15.67:0/1306175622 shutdown_connections 2024-05-03T21:11:12.620 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:12.037+0000 7f9e9421c700 1 -- 172.21.15.67:0/1306175622 wait complete. 2024-05-03T21:11:13.881 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC/J7oMsbQ7wmnz8fHnpQAQlJo7xVjtRYwju3LjA335iqv29dC8Yq4lVSe5WbJdwxhrNYDPzkAbdX8JqGo43dfJKXeZ/ThoEzTVypm0JJmLN/xTJ6fckSUSmuMXVZGoyUlVZLczOc81cal2gw64UrsR8/gmF9l11lonPXKDwqkFFyN63BgLtXeImIEwZI9jBdHr7LECXjEbJTgwjeb/CDP3SIOARk/Vx4s+GIxbMLNxZd0e+7r1+uIlkBNk5FZb10B0RI+KVPOUp85Wgr/BX7gJAItsQCbH/Io04CtlytXCDpBr8iXJynjU69xUTcQN2kNYPhmCulb2VCV0gcB/cfufcoj4oqTBcP0846E/DdNrQ1cVKM1kedZdFK37T3thRUx1LlSzrdqCxX4NJklj9p5yhEs4N2SO3gmS3i/vdNHcIJ61LZuFsKqOmINIyRqq0wkoRWhdLxJRp5l/pwMR9rW7Mukcqw+fM3KDga0wLgvTSGdBw5EkxZzJQnCD0FGuJYc= ceph-713bc57a-0991-11ef-bc97-c7b262605968 2024-05-03T21:11:13.881 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:13.080+0000 7f5fadc76700 1 Processor -- start 2024-05-03T21:11:13.881 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:13.080+0000 7f5fadc76700 1 -- start start 2024-05-03T21:11:13.882 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:13.080+0000 7f5fadc76700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5fa8109d30 0x7f5fa810c120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:11:13.882 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:13.080+0000 7f5fadc76700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f5fa807ce90 con 0x7f5fa8109d30 2024-05-03T21:11:13.882 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:13.080+0000 7f5fa77fe700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5fa8109d30 0x7f5fa810c120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:11:13.882 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:13.081+0000 7f5fa77fe700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5fa8109d30 0x7f5fa810c120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:43832/0 (socket says 172.21.15.67:43832) 2024-05-03T21:11:13.882 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:13.081+0000 7f5fa77fe700 1 -- 172.21.15.67:0/388217374 learned_addr learned my addr 172.21.15.67:0/388217374 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-03T21:11:13.882 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:13.081+0000 7f5fa77fe700 1 -- 172.21.15.67:0/388217374 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5fa810c660 con 0x7f5fa8109d30 2024-05-03T21:11:13.882 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:13.081+0000 7f5fa77fe700 1 --2- 172.21.15.67:0/388217374 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5fa8109d30 0x7f5fa810c120 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f5f98009e80 tx=0x7f5f9800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=381404b57432f9c9 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T21:11:13.882 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:13.081+0000 7f5fa67fc700 1 -- 172.21.15.67:0/388217374 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5f98017070 con 0x7f5fa8109d30 2024-05-03T21:11:13.882 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:13.081+0000 7f5fa67fc700 1 -- 172.21.15.67:0/388217374 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(28 keys) ==== 1139+0+0 (secure 0 0 0) 0x7f5f9800b8c0 con 0x7f5fa8109d30 2024-05-03T21:11:13.882 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:13.081+0000 7f5fa67fc700 1 -- 172.21.15.67:0/388217374 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5f98004620 con 0x7f5fa8109d30 2024-05-03T21:11:13.882 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:13.082+0000 7f5fadc76700 1 -- 172.21.15.67:0/388217374 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5fa8109d30 msgr2=0x7f5fa810c120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:11:13.882 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:13.082+0000 7f5fadc76700 1 --2- 172.21.15.67:0/388217374 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5fa8109d30 0x7f5fa810c120 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f5f98009e80 tx=0x7f5f9800b330 comp rx=0 tx=0).stop 2024-05-03T21:11:13.882 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:13.082+0000 7f5fadc76700 1 -- 172.21.15.67:0/388217374 shutdown_connections 2024-05-03T21:11:13.883 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:13.082+0000 7f5fadc76700 1 --2- 172.21.15.67:0/388217374 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5fa8109d30 0x7f5fa810c120 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:11:13.883 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:13.082+0000 7f5fadc76700 1 -- 172.21.15.67:0/388217374 >> 172.21.15.67:0/388217374 conn(0x7f5fa8103760 msgr2=0x7f5fa8105b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T21:11:13.883 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:13.082+0000 7f5fadc76700 1 -- 172.21.15.67:0/388217374 shutdown_connections 2024-05-03T21:11:13.883 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:13.082+0000 7f5fadc76700 1 -- 172.21.15.67:0/388217374 wait complete. 2024-05-03T21:11:13.883 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:13.083+0000 7f5fadc76700 1 Processor -- start 2024-05-03T21:11:13.883 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:13.083+0000 7f5fadc76700 1 -- start start 2024-05-03T21:11:13.883 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:13.083+0000 7f5fadc76700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5fa8109d30 0x7f5fa819d220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:11:13.883 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:13.083+0000 7f5fadc76700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f5fa810d6e0 con 0x7f5fa8109d30 2024-05-03T21:11:13.883 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:13.083+0000 7f5fa77fe700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5fa8109d30 0x7f5fa819d220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:11:13.883 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:13.083+0000 7f5fa77fe700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5fa8109d30 0x7f5fa819d220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:43842/0 (socket says 172.21.15.67:43842) 2024-05-03T21:11:13.883 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:13.083+0000 7f5fa77fe700 1 -- 172.21.15.67:0/2973202293 learned_addr learned my addr 172.21.15.67:0/2973202293 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-03T21:11:13.883 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:13.083+0000 7f5fa77fe700 1 -- 172.21.15.67:0/2973202293 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5fa819d760 con 0x7f5fa8109d30 2024-05-03T21:11:13.884 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:13.084+0000 7f5fa77fe700 1 --2- 172.21.15.67:0/2973202293 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5fa8109d30 0x7f5fa819d220 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f5f980043d0 tx=0x7f5f98003d00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T21:11:13.884 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:13.084+0000 7f5fa4ff9700 1 -- 172.21.15.67:0/2973202293 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5f98021020 con 0x7f5fa8109d30 2024-05-03T21:11:13.884 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:13.084+0000 7f5fadc76700 1 -- 172.21.15.67:0/2973202293 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5fa819d9f0 con 0x7f5fa8109d30 2024-05-03T21:11:13.884 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:13.084+0000 7f5fa4ff9700 1 -- 172.21.15.67:0/2973202293 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(28 keys) ==== 1139+0+0 (secure 0 0 0) 0x7f5f980094d0 con 0x7f5fa8109d30 2024-05-03T21:11:13.884 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:13.084+0000 7f5fa4ff9700 1 -- 172.21.15.67:0/2973202293 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5f98017040 con 0x7f5fa8109d30 2024-05-03T21:11:13.884 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:13.084+0000 7f5fadc76700 1 -- 172.21.15.67:0/2973202293 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5fa81a06e0 con 0x7f5fa8109d30 2024-05-03T21:11:13.884 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:13.085+0000 7f5fa4ff9700 1 -- 172.21.15.67:0/2973202293 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 8) ==== 50222+0+0 (secure 0 0 0) 0x7f5f98024020 con 0x7f5fa8109d30 2024-05-03T21:11:13.884 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:13.085+0000 7f5fa4ff9700 1 --2- 172.21.15.67:0/2973202293 >> [v2:172.21.15.67:6800/2576113038,v1:172.21.15.67:6801/2576113038] conn(0x7f5f9403e420 0x7f5f940408e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:11:13.884 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:13.085+0000 7f5fa4ff9700 1 -- 172.21.15.67:0/2973202293 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f5f9800f050 con 0x7f5fa8109d30 2024-05-03T21:11:13.884 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:13.085+0000 7f5fadc76700 1 -- 172.21.15.67:0/2973202293 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5fa8064b00 con 0x7f5fa8109d30 2024-05-03T21:11:13.884 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:13.085+0000 7f5fa6ffd700 1 --2- 172.21.15.67:0/2973202293 >> [v2:172.21.15.67:6800/2576113038,v1:172.21.15.67:6801/2576113038] conn(0x7f5f9403e420 0x7f5f940408e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:11:13.884 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:13.086+0000 7f5fa6ffd700 1 --2- 172.21.15.67:0/2973202293 >> [v2:172.21.15.67:6800/2576113038,v1:172.21.15.67:6801/2576113038] conn(0x7f5f9403e420 0x7f5f940408e0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f5f90006fd0 tx=0x7f5f90006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T21:11:13.884 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:13.090+0000 7f5fa4ff9700 1 -- 172.21.15.67:0/2973202293 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191839 (secure 0 0 0) 0x7f5f9802b020 con 0x7f5fa8109d30 2024-05-03T21:11:13.885 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:13.306+0000 7f5fadc76700 1 -- 172.21.15.67:0/2973202293 --> [v2:172.21.15.67:6800/2576113038,v1:172.21.15.67:6801/2576113038] -- mgr_command(tid 0: {"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}) -- 0x7f5fa8104940 con 0x7f5f9403e420 2024-05-03T21:11:13.885 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:13.307+0000 7f5fa4ff9700 1 -- 172.21.15.67:0/2973202293 <== mgr.14118 v2:172.21.15.67:6800/2576113038 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+595 (secure 0 0 0) 0x7f5fa8104940 con 0x7f5f9403e420 2024-05-03T21:11:13.885 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:13.309+0000 7f5fadc76700 1 -- 172.21.15.67:0/2973202293 >> [v2:172.21.15.67:6800/2576113038,v1:172.21.15.67:6801/2576113038] conn(0x7f5f9403e420 msgr2=0x7f5f940408e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:11:13.885 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:13.309+0000 7f5fadc76700 1 --2- 172.21.15.67:0/2973202293 >> [v2:172.21.15.67:6800/2576113038,v1:172.21.15.67:6801/2576113038] conn(0x7f5f9403e420 0x7f5f940408e0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f5f90006fd0 tx=0x7f5f90006e40 comp rx=0 tx=0).stop 2024-05-03T21:11:13.885 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:13.309+0000 7f5fadc76700 1 -- 172.21.15.67:0/2973202293 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5fa8109d30 msgr2=0x7f5fa819d220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:11:13.885 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:13.309+0000 7f5fadc76700 1 --2- 172.21.15.67:0/2973202293 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5fa8109d30 0x7f5fa819d220 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f5f980043d0 tx=0x7f5f98003d00 comp rx=0 tx=0).stop 2024-05-03T21:11:13.885 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:13.309+0000 7f5fadc76700 1 -- 172.21.15.67:0/2973202293 shutdown_connections 2024-05-03T21:11:13.885 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:13.309+0000 7f5fadc76700 1 --2- 172.21.15.67:0/2973202293 >> [v2:172.21.15.67:6800/2576113038,v1:172.21.15.67:6801/2576113038] conn(0x7f5f9403e420 0x7f5f940408e0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:11:13.885 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:13.309+0000 7f5fadc76700 1 --2- 172.21.15.67:0/2973202293 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5fa8109d30 0x7f5fa819d220 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:11:13.885 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:13.309+0000 7f5fadc76700 1 -- 172.21.15.67:0/2973202293 >> 172.21.15.67:0/2973202293 conn(0x7f5fa8103760 msgr2=0x7f5fa8104230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T21:11:13.885 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:13.310+0000 7f5fadc76700 1 -- 172.21.15.67:0/2973202293 shutdown_connections 2024-05-03T21:11:13.885 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:13.310+0000 7f5fadc76700 1 -- 172.21.15.67:0/2973202293 wait complete. 2024-05-03T21:11:13.885 INFO:teuthology.orchestra.run.smithi067.stdout:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2024-05-03T21:11:13.886 INFO:teuthology.orchestra.run.smithi067.stdout:Adding key to root@localhost authorized_keys... 2024-05-03T21:11:13.886 INFO:teuthology.orchestra.run.smithi067.stdout:Adding host smithi067... 2024-05-03T21:11:14.133 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:11:13 smithi067 ceph-mon[29730]: from='client.14136 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2024-05-03T21:11:15.080 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:11:14 smithi067 ceph-mon[29730]: from='client.14138 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi067", "addr": "172.21.15.67", "target": ["mon-mgr", ""]}]: dispatch 2024-05-03T21:11:16.220 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:11:15 smithi067 ceph-mon[29730]: Deploying cephadm binary to smithi067 2024-05-03T21:11:19.288 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout Added host 'smithi067' with addr '172.21.15.67' 2024-05-03T21:11:19.289 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:14.236+0000 7f06d5381700 1 Processor -- start 2024-05-03T21:11:19.289 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:14.236+0000 7f06d5381700 1 -- start start 2024-05-03T21:11:19.289 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:14.237+0000 7f06d5381700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f06d0109d20 0x7f06d010c110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:11:19.289 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:14.237+0000 7f06d5381700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f06d007ce90 con 0x7f06d0109d20 2024-05-03T21:11:19.289 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:14.237+0000 7f06ceffd700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f06d0109d20 0x7f06d010c110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:11:19.289 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:14.237+0000 7f06ceffd700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f06d0109d20 0x7f06d010c110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:43858/0 (socket says 172.21.15.67:43858) 2024-05-03T21:11:19.290 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:14.237+0000 7f06ceffd700 1 -- 172.21.15.67:0/1927405292 learned_addr learned my addr 172.21.15.67:0/1927405292 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-03T21:11:19.290 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:14.237+0000 7f06ceffd700 1 -- 172.21.15.67:0/1927405292 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f06d010c650 con 0x7f06d0109d20 2024-05-03T21:11:19.290 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:14.237+0000 7f06ceffd700 1 --2- 172.21.15.67:0/1927405292 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f06d0109d20 0x7f06d010c110 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f06c00098a0 tx=0x7f06c0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8efa3389dcb4e218 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T21:11:19.290 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:14.238+0000 7f06cdffb700 1 -- 172.21.15.67:0/1927405292 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f06c0004160 con 0x7f06d0109d20 2024-05-03T21:11:19.290 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:14.238+0000 7f06cdffb700 1 -- 172.21.15.67:0/1927405292 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(28 keys) ==== 1139+0+0 (secure 0 0 0) 0x7f06c0004300 con 0x7f06d0109d20 2024-05-03T21:11:19.290 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:14.238+0000 7f06cdffb700 1 -- 172.21.15.67:0/1927405292 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f06c0011c00 con 0x7f06d0109d20 2024-05-03T21:11:19.290 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:14.238+0000 7f06d5381700 1 -- 172.21.15.67:0/1927405292 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f06d0109d20 msgr2=0x7f06d010c110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:11:19.290 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:14.238+0000 7f06d5381700 1 --2- 172.21.15.67:0/1927405292 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f06d0109d20 0x7f06d010c110 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f06c00098a0 tx=0x7f06c0009bb0 comp rx=0 tx=0).stop 2024-05-03T21:11:19.290 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:14.238+0000 7f06d5381700 1 -- 172.21.15.67:0/1927405292 shutdown_connections 2024-05-03T21:11:19.290 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:14.238+0000 7f06d5381700 1 --2- 172.21.15.67:0/1927405292 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f06d0109d20 0x7f06d010c110 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:11:19.290 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:14.238+0000 7f06d5381700 1 -- 172.21.15.67:0/1927405292 >> 172.21.15.67:0/1927405292 conn(0x7f06d0103770 msgr2=0x7f06d0105b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T21:11:19.290 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:14.238+0000 7f06d5381700 1 -- 172.21.15.67:0/1927405292 shutdown_connections 2024-05-03T21:11:19.291 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:14.238+0000 7f06d5381700 1 -- 172.21.15.67:0/1927405292 wait complete. 2024-05-03T21:11:19.291 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:14.239+0000 7f06d5381700 1 Processor -- start 2024-05-03T21:11:19.291 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:14.239+0000 7f06d5381700 1 -- start start 2024-05-03T21:11:19.291 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:14.239+0000 7f06d5381700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f06d0109d20 0x7f06d019d260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:11:19.291 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:14.239+0000 7f06d5381700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f06d010d6d0 con 0x7f06d0109d20 2024-05-03T21:11:19.291 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:14.239+0000 7f06ceffd700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f06d0109d20 0x7f06d019d260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:11:19.291 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:14.239+0000 7f06ceffd700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f06d0109d20 0x7f06d019d260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:43868/0 (socket says 172.21.15.67:43868) 2024-05-03T21:11:19.291 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:14.239+0000 7f06ceffd700 1 -- 172.21.15.67:0/179524466 learned_addr learned my addr 172.21.15.67:0/179524466 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-03T21:11:19.291 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:14.240+0000 7f06ceffd700 1 -- 172.21.15.67:0/179524466 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f06d019d7a0 con 0x7f06d0109d20 2024-05-03T21:11:19.291 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:14.240+0000 7f06ceffd700 1 --2- 172.21.15.67:0/179524466 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f06d0109d20 0x7f06d019d260 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f06c0011bd0 tx=0x7f06c0011cb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T21:11:19.291 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:14.240+0000 7f06b7fff700 1 -- 172.21.15.67:0/179524466 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f06c0003b20 con 0x7f06d0109d20 2024-05-03T21:11:19.291 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:14.240+0000 7f06d5381700 1 -- 172.21.15.67:0/179524466 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f06d019da30 con 0x7f06d0109d20 2024-05-03T21:11:19.291 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:14.240+0000 7f06b7fff700 1 -- 172.21.15.67:0/179524466 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(28 keys) ==== 1139+0+0 (secure 0 0 0) 0x7f06c0003cc0 con 0x7f06d0109d20 2024-05-03T21:11:19.291 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:14.240+0000 7f06d5381700 1 -- 172.21.15.67:0/179524466 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f06d01a0720 con 0x7f06d0109d20 2024-05-03T21:11:19.291 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:14.240+0000 7f06b7fff700 1 -- 172.21.15.67:0/179524466 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f06c0024930 con 0x7f06d0109d20 2024-05-03T21:11:19.292 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:14.241+0000 7f06b7fff700 1 -- 172.21.15.67:0/179524466 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 8) ==== 50222+0+0 (secure 0 0 0) 0x7f06c0022020 con 0x7f06d0109d20 2024-05-03T21:11:19.292 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:14.241+0000 7f06d5381700 1 -- 172.21.15.67:0/179524466 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f06d0064b00 con 0x7f06d0109d20 2024-05-03T21:11:19.292 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:14.241+0000 7f06b7fff700 1 --2- 172.21.15.67:0/179524466 >> [v2:172.21.15.67:6800/2576113038,v1:172.21.15.67:6801/2576113038] conn(0x7f06b803e4c0 0x7f06b8040980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:11:19.292 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:14.241+0000 7f06b7fff700 1 -- 172.21.15.67:0/179524466 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f06c00536c0 con 0x7f06d0109d20 2024-05-03T21:11:19.292 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:14.241+0000 7f06ce7fc700 1 --2- 172.21.15.67:0/179524466 >> [v2:172.21.15.67:6800/2576113038,v1:172.21.15.67:6801/2576113038] conn(0x7f06b803e4c0 0x7f06b8040980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:11:19.292 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:14.242+0000 7f06ce7fc700 1 --2- 172.21.15.67:0/179524466 >> [v2:172.21.15.67:6800/2576113038,v1:172.21.15.67:6801/2576113038] conn(0x7f06b803e4c0 0x7f06b8040980 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f06c4006fd0 tx=0x7f06c4006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T21:11:19.292 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:14.246+0000 7f06b7fff700 1 -- 172.21.15.67:0/179524466 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191839 (secure 0 0 0) 0x7f06c001a950 con 0x7f06d0109d20 2024-05-03T21:11:19.292 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:14.464+0000 7f06d5381700 1 -- 172.21.15.67:0/179524466 --> [v2:172.21.15.67:6800/2576113038,v1:172.21.15.67:6801/2576113038] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi067", "addr": "172.21.15.67", "target": ["mon-mgr", ""]}) -- 0x7f06d0104970 con 0x7f06b803e4c0 2024-05-03T21:11:19.292 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:18.714+0000 7f06b7fff700 1 -- 172.21.15.67:0/179524466 <== mgr.14118 v2:172.21.15.67:6800/2576113038 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+48 (secure 0 0 0) 0x7f06d0104970 con 0x7f06b803e4c0 2024-05-03T21:11:19.292 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:18.716+0000 7f06d5381700 1 -- 172.21.15.67:0/179524466 >> [v2:172.21.15.67:6800/2576113038,v1:172.21.15.67:6801/2576113038] conn(0x7f06b803e4c0 msgr2=0x7f06b8040980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:11:19.292 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:18.716+0000 7f06d5381700 1 --2- 172.21.15.67:0/179524466 >> [v2:172.21.15.67:6800/2576113038,v1:172.21.15.67:6801/2576113038] conn(0x7f06b803e4c0 0x7f06b8040980 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f06c4006fd0 tx=0x7f06c4006e40 comp rx=0 tx=0).stop 2024-05-03T21:11:19.292 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:18.716+0000 7f06d5381700 1 -- 172.21.15.67:0/179524466 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f06d0109d20 msgr2=0x7f06d019d260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:11:19.292 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:18.716+0000 7f06d5381700 1 --2- 172.21.15.67:0/179524466 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f06d0109d20 0x7f06d019d260 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f06c0011bd0 tx=0x7f06c0011cb0 comp rx=0 tx=0).stop 2024-05-03T21:11:19.292 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:18.716+0000 7f06d5381700 1 -- 172.21.15.67:0/179524466 shutdown_connections 2024-05-03T21:11:19.293 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:18.716+0000 7f06d5381700 1 --2- 172.21.15.67:0/179524466 >> [v2:172.21.15.67:6800/2576113038,v1:172.21.15.67:6801/2576113038] conn(0x7f06b803e4c0 0x7f06b8040980 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:11:19.293 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:18.716+0000 7f06d5381700 1 --2- 172.21.15.67:0/179524466 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f06d0109d20 0x7f06d019d260 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:11:19.293 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:18.716+0000 7f06d5381700 1 -- 172.21.15.67:0/179524466 >> 172.21.15.67:0/179524466 conn(0x7f06d0103770 msgr2=0x7f06d0104260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T21:11:19.293 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:18.717+0000 7f06d5381700 1 -- 172.21.15.67:0/179524466 shutdown_connections 2024-05-03T21:11:19.293 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:18.717+0000 7f06d5381700 1 -- 172.21.15.67:0/179524466 wait complete. 2024-05-03T21:11:19.293 INFO:teuthology.orchestra.run.smithi067.stdout:Deploying unmanaged mon service... 2024-05-03T21:11:19.970 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:11:19 smithi067 ceph-mon[29730]: from='mgr.14118 172.21.15.67:0/4082808469' entity='mgr.x' 2024-05-03T21:11:19.971 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:11:19 smithi067 ceph-mon[29730]: Added host smithi067 2024-05-03T21:11:19.971 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:11:19 smithi067 ceph-mon[29730]: from='mgr.14118 172.21.15.67:0/4082808469' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T21:11:20.648 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout Scheduled mon update... 2024-05-03T21:11:20.648 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:19.754+0000 7f516968b700 1 Processor -- start 2024-05-03T21:11:20.649 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:19.754+0000 7f516968b700 1 -- start start 2024-05-03T21:11:20.649 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:19.754+0000 7f516968b700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5164109cf0 0x7f516410c0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:11:20.649 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:19.754+0000 7f516968b700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f5164076ff0 con 0x7f5164109cf0 2024-05-03T21:11:20.649 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:19.755+0000 7f5162ffd700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5164109cf0 0x7f516410c0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:11:20.649 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:19.755+0000 7f5162ffd700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5164109cf0 0x7f516410c0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:57444/0 (socket says 172.21.15.67:57444) 2024-05-03T21:11:20.649 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:19.755+0000 7f5162ffd700 1 -- 172.21.15.67:0/4001970837 learned_addr learned my addr 172.21.15.67:0/4001970837 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-03T21:11:20.649 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:19.755+0000 7f5162ffd700 1 -- 172.21.15.67:0/4001970837 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f516410c620 con 0x7f5164109cf0 2024-05-03T21:11:20.649 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:19.755+0000 7f5162ffd700 1 --2- 172.21.15.67:0/4001970837 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5164109cf0 0x7f516410c0e0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f515c009e80 tx=0x7f515c00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1f7fcbdd8043a87d server_cookie=0 in_seq=0 out_seq=0 2024-05-03T21:11:20.649 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:19.755+0000 7f5161ffb700 1 -- 172.21.15.67:0/4001970837 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f515c017070 con 0x7f5164109cf0 2024-05-03T21:11:20.649 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:19.755+0000 7f5161ffb700 1 -- 172.21.15.67:0/4001970837 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(28 keys) ==== 1139+0+0 (secure 0 0 0) 0x7f515c00b8c0 con 0x7f5164109cf0 2024-05-03T21:11:20.649 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:19.755+0000 7f5161ffb700 1 -- 172.21.15.67:0/4001970837 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f515c004620 con 0x7f5164109cf0 2024-05-03T21:11:20.649 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:19.756+0000 7f516968b700 1 -- 172.21.15.67:0/4001970837 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5164109cf0 msgr2=0x7f516410c0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:11:20.649 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:19.756+0000 7f516968b700 1 --2- 172.21.15.67:0/4001970837 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5164109cf0 0x7f516410c0e0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f515c009e80 tx=0x7f515c00b330 comp rx=0 tx=0).stop 2024-05-03T21:11:20.650 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:19.756+0000 7f516968b700 1 -- 172.21.15.67:0/4001970837 shutdown_connections 2024-05-03T21:11:20.650 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:19.756+0000 7f516968b700 1 --2- 172.21.15.67:0/4001970837 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5164109cf0 0x7f516410c0e0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:11:20.650 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:19.756+0000 7f516968b700 1 -- 172.21.15.67:0/4001970837 >> 172.21.15.67:0/4001970837 conn(0x7f51641037e0 msgr2=0x7f5164105c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T21:11:20.650 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:19.756+0000 7f516968b700 1 -- 172.21.15.67:0/4001970837 shutdown_connections 2024-05-03T21:11:20.650 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:19.756+0000 7f516968b700 1 -- 172.21.15.67:0/4001970837 wait complete. 2024-05-03T21:11:20.650 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:19.756+0000 7f516968b700 1 Processor -- start 2024-05-03T21:11:20.650 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:19.757+0000 7f516968b700 1 -- start start 2024-05-03T21:11:20.650 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:19.757+0000 7f516968b700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5164109cf0 0x7f5164109570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:11:20.650 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:19.757+0000 7f516968b700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f516410d6b0 con 0x7f5164109cf0 2024-05-03T21:11:20.650 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:19.757+0000 7f5162ffd700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5164109cf0 0x7f5164109570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:11:20.650 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:19.757+0000 7f5162ffd700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5164109cf0 0x7f5164109570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:57458/0 (socket says 172.21.15.67:57458) 2024-05-03T21:11:20.650 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:19.757+0000 7f5162ffd700 1 -- 172.21.15.67:0/3472108786 learned_addr learned my addr 172.21.15.67:0/3472108786 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-03T21:11:20.650 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:19.757+0000 7f5162ffd700 1 -- 172.21.15.67:0/3472108786 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5164109ab0 con 0x7f5164109cf0 2024-05-03T21:11:20.650 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:19.757+0000 7f5162ffd700 1 --2- 172.21.15.67:0/3472108786 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5164109cf0 0x7f5164109570 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f515c0043d0 tx=0x7f515c003d00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T21:11:20.651 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:19.757+0000 7f515bfff700 1 -- 172.21.15.67:0/3472108786 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f515c021020 con 0x7f5164109cf0 2024-05-03T21:11:20.651 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:19.757+0000 7f516968b700 1 -- 172.21.15.67:0/3472108786 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5164107ca0 con 0x7f5164109cf0 2024-05-03T21:11:20.651 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:19.757+0000 7f515bfff700 1 -- 172.21.15.67:0/3472108786 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(28 keys) ==== 1139+0+0 (secure 0 0 0) 0x7f515c0094d0 con 0x7f5164109cf0 2024-05-03T21:11:20.651 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:19.757+0000 7f515bfff700 1 -- 172.21.15.67:0/3472108786 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f515c017040 con 0x7f5164109cf0 2024-05-03T21:11:20.651 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:19.757+0000 7f516968b700 1 -- 172.21.15.67:0/3472108786 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f51641081e0 con 0x7f5164109cf0 2024-05-03T21:11:20.651 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:19.758+0000 7f515bfff700 1 -- 172.21.15.67:0/3472108786 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 8) ==== 50222+0+0 (secure 0 0 0) 0x7f515c024020 con 0x7f5164109cf0 2024-05-03T21:11:20.651 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:19.758+0000 7f516968b700 1 -- 172.21.15.67:0/3472108786 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5164064b00 con 0x7f5164109cf0 2024-05-03T21:11:20.651 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:19.759+0000 7f515bfff700 1 --2- 172.21.15.67:0/3472108786 >> [v2:172.21.15.67:6800/2576113038,v1:172.21.15.67:6801/2576113038] conn(0x7f515003e420 0x7f51500408e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:11:20.651 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:19.759+0000 7f515bfff700 1 -- 172.21.15.67:0/3472108786 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f515c00f050 con 0x7f5164109cf0 2024-05-03T21:11:20.652 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:19.759+0000 7f51627fc700 1 --2- 172.21.15.67:0/3472108786 >> [v2:172.21.15.67:6800/2576113038,v1:172.21.15.67:6801/2576113038] conn(0x7f515003e420 0x7f51500408e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:11:20.652 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:19.759+0000 7f51627fc700 1 --2- 172.21.15.67:0/3472108786 >> [v2:172.21.15.67:6800/2576113038,v1:172.21.15.67:6801/2576113038] conn(0x7f515003e420 0x7f51500408e0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f514c006fd0 tx=0x7f514c006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T21:11:20.652 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:19.764+0000 7f515bfff700 1 -- 172.21.15.67:0/3472108786 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191839 (secure 0 0 0) 0x7f515c02b020 con 0x7f5164109cf0 2024-05-03T21:11:20.652 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:20.009+0000 7f516968b700 1 -- 172.21.15.67:0/3472108786 --> [v2:172.21.15.67:6800/2576113038,v1:172.21.15.67:6801/2576113038] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}) -- 0x7f5164104910 con 0x7f515003e420 2024-05-03T21:11:20.652 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:20.014+0000 7f515bfff700 1 -- 172.21.15.67:0/3472108786 <== mgr.14118 v2:172.21.15.67:6800/2576113038 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7f5164104910 con 0x7f515003e420 2024-05-03T21:11:20.653 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:20.016+0000 7f516968b700 1 -- 172.21.15.67:0/3472108786 >> [v2:172.21.15.67:6800/2576113038,v1:172.21.15.67:6801/2576113038] conn(0x7f515003e420 msgr2=0x7f51500408e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:11:20.653 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:20.016+0000 7f516968b700 1 --2- 172.21.15.67:0/3472108786 >> [v2:172.21.15.67:6800/2576113038,v1:172.21.15.67:6801/2576113038] conn(0x7f515003e420 0x7f51500408e0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f514c006fd0 tx=0x7f514c006e40 comp rx=0 tx=0).stop 2024-05-03T21:11:20.653 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:20.016+0000 7f516968b700 1 -- 172.21.15.67:0/3472108786 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5164109cf0 msgr2=0x7f5164109570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:11:20.653 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:20.016+0000 7f516968b700 1 --2- 172.21.15.67:0/3472108786 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5164109cf0 0x7f5164109570 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f515c0043d0 tx=0x7f515c003d00 comp rx=0 tx=0).stop 2024-05-03T21:11:20.653 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:20.016+0000 7f516968b700 1 -- 172.21.15.67:0/3472108786 shutdown_connections 2024-05-03T21:11:20.653 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:20.016+0000 7f516968b700 1 --2- 172.21.15.67:0/3472108786 >> [v2:172.21.15.67:6800/2576113038,v1:172.21.15.67:6801/2576113038] conn(0x7f515003e420 0x7f51500408e0 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:11:20.653 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:20.016+0000 7f516968b700 1 --2- 172.21.15.67:0/3472108786 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5164109cf0 0x7f5164109570 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:11:20.653 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:20.016+0000 7f516968b700 1 -- 172.21.15.67:0/3472108786 >> 172.21.15.67:0/3472108786 conn(0x7f51641037e0 msgr2=0x7f5164104200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T21:11:20.653 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:20.017+0000 7f516968b700 1 -- 172.21.15.67:0/3472108786 shutdown_connections 2024-05-03T21:11:20.654 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:20.017+0000 7f516968b700 1 -- 172.21.15.67:0/3472108786 wait complete. 2024-05-03T21:11:20.654 INFO:teuthology.orchestra.run.smithi067.stdout:Deploying unmanaged mgr service... 2024-05-03T21:11:21.269 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:11:21 smithi067 ceph-mon[29730]: from='client.14140 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-05-03T21:11:21.269 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:11:21 smithi067 ceph-mon[29730]: Saving service mon spec with placement count:5 2024-05-03T21:11:21.269 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:11:21 smithi067 ceph-mon[29730]: from='mgr.14118 172.21.15.67:0/4082808469' entity='mgr.x' 2024-05-03T21:11:21.965 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout Scheduled mgr update... 2024-05-03T21:11:21.965 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:21.099+0000 7f2dd1ae5700 1 Processor -- start 2024-05-03T21:11:21.965 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:21.099+0000 7f2dd1ae5700 1 -- start start 2024-05-03T21:11:21.965 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:21.099+0000 7f2dd1ae5700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2dcc1078f0 0x7f2dcc107d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:11:21.966 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:21.099+0000 7f2dd1ae5700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f2dcc1082e0 con 0x7f2dcc1078f0 2024-05-03T21:11:21.966 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:21.100+0000 7f2dcb7fe700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2dcc1078f0 0x7f2dcc107d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:11:21.966 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:21.100+0000 7f2dcb7fe700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2dcc1078f0 0x7f2dcc107d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:57462/0 (socket says 172.21.15.67:57462) 2024-05-03T21:11:21.966 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:21.100+0000 7f2dcb7fe700 1 -- 172.21.15.67:0/786878318 learned_addr learned my addr 172.21.15.67:0/786878318 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-03T21:11:21.966 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:21.100+0000 7f2dcb7fe700 1 -- 172.21.15.67:0/786878318 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2dcc108af0 con 0x7f2dcc1078f0 2024-05-03T21:11:21.966 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:21.100+0000 7f2dcb7fe700 1 --2- 172.21.15.67:0/786878318 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2dcc1078f0 0x7f2dcc107d10 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f2dbc0098a0 tx=0x7f2dbc009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eb4a14326bc37195 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T21:11:21.966 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:21.100+0000 7f2dca7fc700 1 -- 172.21.15.67:0/786878318 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2dbc004160 con 0x7f2dcc1078f0 2024-05-03T21:11:21.966 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:21.100+0000 7f2dca7fc700 1 -- 172.21.15.67:0/786878318 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(28 keys) ==== 1139+0+0 (secure 0 0 0) 0x7f2dbc004300 con 0x7f2dcc1078f0 2024-05-03T21:11:21.966 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:21.100+0000 7f2dca7fc700 1 -- 172.21.15.67:0/786878318 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2dbc011c00 con 0x7f2dcc1078f0 2024-05-03T21:11:21.966 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:21.101+0000 7f2dd1ae5700 1 -- 172.21.15.67:0/786878318 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2dcc1078f0 msgr2=0x7f2dcc107d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:11:21.966 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:21.101+0000 7f2dd1ae5700 1 --2- 172.21.15.67:0/786878318 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2dcc1078f0 0x7f2dcc107d10 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f2dbc0098a0 tx=0x7f2dbc009bb0 comp rx=0 tx=0).stop 2024-05-03T21:11:21.966 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:21.101+0000 7f2dd1ae5700 1 -- 172.21.15.67:0/786878318 shutdown_connections 2024-05-03T21:11:21.967 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:21.101+0000 7f2dd1ae5700 1 --2- 172.21.15.67:0/786878318 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2dcc1078f0 0x7f2dcc107d10 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:11:21.967 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:21.101+0000 7f2dd1ae5700 1 -- 172.21.15.67:0/786878318 >> 172.21.15.67:0/786878318 conn(0x7f2dcc102f50 msgr2=0x7f2dcc105370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T21:11:21.967 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:21.101+0000 7f2dd1ae5700 1 -- 172.21.15.67:0/786878318 shutdown_connections 2024-05-03T21:11:21.967 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:21.101+0000 7f2dd1ae5700 1 -- 172.21.15.67:0/786878318 wait complete. 2024-05-03T21:11:21.967 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:21.101+0000 7f2dd1ae5700 1 Processor -- start 2024-05-03T21:11:21.967 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:21.102+0000 7f2dd1ae5700 1 -- start start 2024-05-03T21:11:21.967 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:21.102+0000 7f2dd1ae5700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2dcc1078f0 0x7f2dcc1043a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:11:21.967 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:21.102+0000 7f2dd1ae5700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f2dcc10bea0 con 0x7f2dcc1078f0 2024-05-03T21:11:21.967 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:21.102+0000 7f2dcb7fe700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2dcc1078f0 0x7f2dcc1043a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:11:21.967 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:21.102+0000 7f2dcb7fe700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2dcc1078f0 0x7f2dcc1043a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:57474/0 (socket says 172.21.15.67:57474) 2024-05-03T21:11:21.967 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:21.102+0000 7f2dcb7fe700 1 -- 172.21.15.67:0/526284879 learned_addr learned my addr 172.21.15.67:0/526284879 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-03T21:11:21.967 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:21.102+0000 7f2dcb7fe700 1 -- 172.21.15.67:0/526284879 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2dcc1a0000 con 0x7f2dcc1078f0 2024-05-03T21:11:21.967 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:21.102+0000 7f2dcb7fe700 1 --2- 172.21.15.67:0/526284879 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2dcc1078f0 0x7f2dcc1043a0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f2dbc003fb0 tx=0x7f2dbc011bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T21:11:21.968 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:21.102+0000 7f2dc8ff9700 1 -- 172.21.15.67:0/526284879 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2dbc0039c0 con 0x7f2dcc1078f0 2024-05-03T21:11:21.968 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:21.103+0000 7f2dc8ff9700 1 -- 172.21.15.67:0/526284879 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(28 keys) ==== 1139+0+0 (secure 0 0 0) 0x7f2dbc003b60 con 0x7f2dcc1078f0 2024-05-03T21:11:21.968 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:21.103+0000 7f2dd1ae5700 1 -- 172.21.15.67:0/526284879 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2dcc1a01d0 con 0x7f2dcc1078f0 2024-05-03T21:11:21.968 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:21.103+0000 7f2dc8ff9700 1 -- 172.21.15.67:0/526284879 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2dbc01a6d0 con 0x7f2dcc1078f0 2024-05-03T21:11:21.968 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:21.103+0000 7f2dd1ae5700 1 -- 172.21.15.67:0/526284879 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2dcc1a0570 con 0x7f2dcc1078f0 2024-05-03T21:11:21.968 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:21.103+0000 7f2dc8ff9700 1 -- 172.21.15.67:0/526284879 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 8) ==== 50222+0+0 (secure 0 0 0) 0x7f2dbc01a9a0 con 0x7f2dcc1078f0 2024-05-03T21:11:21.968 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:21.104+0000 7f2dd1ae5700 1 -- 172.21.15.67:0/526284879 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2dcc10c020 con 0x7f2dcc1078f0 2024-05-03T21:11:21.968 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:21.104+0000 7f2dc8ff9700 1 --2- 172.21.15.67:0/526284879 >> [v2:172.21.15.67:6800/2576113038,v1:172.21.15.67:6801/2576113038] conn(0x7f2db403e420 0x7f2db40408e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:11:21.968 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:21.104+0000 7f2dc8ff9700 1 -- 172.21.15.67:0/526284879 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f2dbc053a80 con 0x7f2dcc1078f0 2024-05-03T21:11:21.968 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:21.104+0000 7f2dcaffd700 1 --2- 172.21.15.67:0/526284879 >> [v2:172.21.15.67:6800/2576113038,v1:172.21.15.67:6801/2576113038] conn(0x7f2db403e420 0x7f2db40408e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:11:21.968 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:21.104+0000 7f2dcaffd700 1 --2- 172.21.15.67:0/526284879 >> [v2:172.21.15.67:6800/2576113038,v1:172.21.15.67:6801/2576113038] conn(0x7f2db403e420 0x7f2db40408e0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f2dc0006fd0 tx=0x7f2dc0006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T21:11:21.968 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:21.109+0000 7f2dc8ff9700 1 -- 172.21.15.67:0/526284879 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191839 (secure 0 0 0) 0x7f2dbc02c350 con 0x7f2dcc1078f0 2024-05-03T21:11:21.969 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:21.341+0000 7f2dd1ae5700 1 -- 172.21.15.67:0/526284879 --> [v2:172.21.15.67:6800/2576113038,v1:172.21.15.67:6801/2576113038] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}) -- 0x7f2dcc02e960 con 0x7f2db403e420 2024-05-03T21:11:21.969 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:21.348+0000 7f2dc8ff9700 1 -- 172.21.15.67:0/526284879 <== mgr.14118 v2:172.21.15.67:6800/2576113038 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7f2dcc02e960 con 0x7f2db403e420 2024-05-03T21:11:21.969 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:21.350+0000 7f2dd1ae5700 1 -- 172.21.15.67:0/526284879 >> [v2:172.21.15.67:6800/2576113038,v1:172.21.15.67:6801/2576113038] conn(0x7f2db403e420 msgr2=0x7f2db40408e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:11:21.969 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:21.350+0000 7f2dd1ae5700 1 --2- 172.21.15.67:0/526284879 >> [v2:172.21.15.67:6800/2576113038,v1:172.21.15.67:6801/2576113038] conn(0x7f2db403e420 0x7f2db40408e0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f2dc0006fd0 tx=0x7f2dc0006e40 comp rx=0 tx=0).stop 2024-05-03T21:11:21.969 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:21.350+0000 7f2dd1ae5700 1 -- 172.21.15.67:0/526284879 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2dcc1078f0 msgr2=0x7f2dcc1043a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:11:21.969 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:21.350+0000 7f2dd1ae5700 1 --2- 172.21.15.67:0/526284879 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2dcc1078f0 0x7f2dcc1043a0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f2dbc003fb0 tx=0x7f2dbc011bd0 comp rx=0 tx=0).stop 2024-05-03T21:11:21.969 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:21.351+0000 7f2dd1ae5700 1 -- 172.21.15.67:0/526284879 shutdown_connections 2024-05-03T21:11:21.969 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:21.351+0000 7f2dd1ae5700 1 --2- 172.21.15.67:0/526284879 >> [v2:172.21.15.67:6800/2576113038,v1:172.21.15.67:6801/2576113038] conn(0x7f2db403e420 0x7f2db40408e0 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:11:21.969 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:21.351+0000 7f2dd1ae5700 1 --2- 172.21.15.67:0/526284879 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2dcc1078f0 0x7f2dcc1043a0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:11:21.969 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:21.351+0000 7f2dd1ae5700 1 -- 172.21.15.67:0/526284879 >> 172.21.15.67:0/526284879 conn(0x7f2dcc102f50 msgr2=0x7f2dcc1052c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T21:11:21.969 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:21.351+0000 7f2dd1ae5700 1 -- 172.21.15.67:0/526284879 shutdown_connections 2024-05-03T21:11:21.969 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:21.351+0000 7f2dd1ae5700 1 -- 172.21.15.67:0/526284879 wait complete. 2024-05-03T21:11:22.607 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:11:22 smithi067 ceph-mon[29730]: from='client.14142 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-05-03T21:11:22.607 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:11:22 smithi067 ceph-mon[29730]: Saving service mgr spec with placement count:2 2024-05-03T21:11:22.608 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:11:22 smithi067 ceph-mon[29730]: from='mgr.14118 172.21.15.67:0/4082808469' entity='mgr.x' 2024-05-03T21:11:23.356 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:22.378+0000 7f7812f53700 1 Processor -- start 2024-05-03T21:11:23.356 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:22.378+0000 7f7812f53700 1 -- start start 2024-05-03T21:11:23.357 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:22.378+0000 7f7812f53700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f780c108110 0x7f780c108530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:11:23.357 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:22.378+0000 7f7812f53700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f780c108b00 con 0x7f780c108110 2024-05-03T21:11:23.357 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:22.379+0000 7f7810cef700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f780c108110 0x7f780c108530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:11:23.357 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:22.379+0000 7f7810cef700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f780c108110 0x7f780c108530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:57488/0 (socket says 172.21.15.67:57488) 2024-05-03T21:11:23.357 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:22.379+0000 7f7810cef700 1 -- 172.21.15.67:0/1086626798 learned_addr learned my addr 172.21.15.67:0/1086626798 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-03T21:11:23.357 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:22.379+0000 7f7810cef700 1 -- 172.21.15.67:0/1086626798 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f780c109310 con 0x7f780c108110 2024-05-03T21:11:23.357 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:22.379+0000 7f7810cef700 1 --2- 172.21.15.67:0/1086626798 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f780c108110 0x7f780c108530 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f77fc0098a0 tx=0x7f77fc009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=396a03e3d8a38a19 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T21:11:23.357 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:22.379+0000 7f780b7fe700 1 -- 172.21.15.67:0/1086626798 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f77fc004160 con 0x7f780c108110 2024-05-03T21:11:23.357 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:22.379+0000 7f780b7fe700 1 -- 172.21.15.67:0/1086626798 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(28 keys) ==== 1139+0+0 (secure 0 0 0) 0x7f77fc004300 con 0x7f780c108110 2024-05-03T21:11:23.357 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:22.379+0000 7f780b7fe700 1 -- 172.21.15.67:0/1086626798 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f77fc011c00 con 0x7f780c108110 2024-05-03T21:11:23.357 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:22.380+0000 7f7812f53700 1 -- 172.21.15.67:0/1086626798 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f780c108110 msgr2=0x7f780c108530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:11:23.357 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:22.380+0000 7f7812f53700 1 --2- 172.21.15.67:0/1086626798 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f780c108110 0x7f780c108530 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f77fc0098a0 tx=0x7f77fc009bb0 comp rx=0 tx=0).stop 2024-05-03T21:11:23.357 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:22.380+0000 7f7812f53700 1 -- 172.21.15.67:0/1086626798 shutdown_connections 2024-05-03T21:11:23.358 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:22.380+0000 7f7812f53700 1 --2- 172.21.15.67:0/1086626798 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f780c108110 0x7f780c108530 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:11:23.358 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:22.380+0000 7f7812f53700 1 -- 172.21.15.67:0/1086626798 >> 172.21.15.67:0/1086626798 conn(0x7f780c1037b0 msgr2=0x7f780c105bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T21:11:23.358 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:22.380+0000 7f7812f53700 1 -- 172.21.15.67:0/1086626798 shutdown_connections 2024-05-03T21:11:23.358 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:22.380+0000 7f7812f53700 1 -- 172.21.15.67:0/1086626798 wait complete. 2024-05-03T21:11:23.358 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:22.381+0000 7f7812f53700 1 Processor -- start 2024-05-03T21:11:23.358 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:22.381+0000 7f7812f53700 1 -- start start 2024-05-03T21:11:23.358 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:22.381+0000 7f7812f53700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f780c108110 0x7f780c19d2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:11:23.358 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:22.381+0000 7f7812f53700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f780c10c6c0 con 0x7f780c108110 2024-05-03T21:11:23.358 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:22.381+0000 7f7810cef700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f780c108110 0x7f780c19d2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:11:23.358 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:22.381+0000 7f7810cef700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f780c108110 0x7f780c19d2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:57502/0 (socket says 172.21.15.67:57502) 2024-05-03T21:11:23.358 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:22.381+0000 7f7810cef700 1 -- 172.21.15.67:0/65542663 learned_addr learned my addr 172.21.15.67:0/65542663 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-03T21:11:23.358 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:22.381+0000 7f7810cef700 1 -- 172.21.15.67:0/65542663 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f780c19d7e0 con 0x7f780c108110 2024-05-03T21:11:23.358 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:22.381+0000 7f7810cef700 1 --2- 172.21.15.67:0/65542663 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f780c108110 0x7f780c19d2a0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f77fc003fb0 tx=0x7f77fc011bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T21:11:23.359 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:22.381+0000 7f7809ffb700 1 -- 172.21.15.67:0/65542663 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f77fc0039c0 con 0x7f780c108110 2024-05-03T21:11:23.359 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:22.382+0000 7f7809ffb700 1 -- 172.21.15.67:0/65542663 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(28 keys) ==== 1139+0+0 (secure 0 0 0) 0x7f77fc003b60 con 0x7f780c108110 2024-05-03T21:11:23.359 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:22.382+0000 7f7812f53700 1 -- 172.21.15.67:0/65542663 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f780c19da70 con 0x7f780c108110 2024-05-03T21:11:23.359 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:22.382+0000 7f7809ffb700 1 -- 172.21.15.67:0/65542663 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f77fc01a6d0 con 0x7f780c108110 2024-05-03T21:11:23.359 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:22.382+0000 7f7812f53700 1 -- 172.21.15.67:0/65542663 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f780c1a0760 con 0x7f780c108110 2024-05-03T21:11:23.359 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:22.383+0000 7f7809ffb700 1 -- 172.21.15.67:0/65542663 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 8) ==== 50222+0+0 (secure 0 0 0) 0x7f77fc01a8f0 con 0x7f780c108110 2024-05-03T21:11:23.359 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:22.383+0000 7f7809ffb700 1 --2- 172.21.15.67:0/65542663 >> [v2:172.21.15.67:6800/2576113038,v1:172.21.15.67:6801/2576113038] conn(0x7f77f403e470 0x7f77f4040930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:11:23.359 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:22.383+0000 7f7809ffb700 1 -- 172.21.15.67:0/65542663 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f77fc053ad0 con 0x7f780c108110 2024-05-03T21:11:23.359 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:22.383+0000 7f780bfff700 1 --2- 172.21.15.67:0/65542663 >> [v2:172.21.15.67:6800/2576113038,v1:172.21.15.67:6801/2576113038] conn(0x7f77f403e470 0x7f77f4040930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:11:23.359 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:22.383+0000 7f7812f53700 1 -- 172.21.15.67:0/65542663 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f780c10c840 con 0x7f780c108110 2024-05-03T21:11:23.360 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:22.383+0000 7f780bfff700 1 --2- 172.21.15.67:0/65542663 >> [v2:172.21.15.67:6800/2576113038,v1:172.21.15.67:6801/2576113038] conn(0x7f77f403e470 0x7f77f4040930 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f7800006fd0 tx=0x7f7800006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T21:11:23.360 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:22.388+0000 7f7809ffb700 1 -- 172.21.15.67:0/65542663 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191839 (secure 0 0 0) 0x7f77fc017e90 con 0x7f780c108110 2024-05-03T21:11:23.360 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:22.605+0000 7f7812f53700 1 -- 172.21.15.67:0/65542663 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/container_init}] v 0) -- 0x7f780c02e760 con 0x7f780c108110 2024-05-03T21:11:23.360 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:22.610+0000 7f7809ffb700 1 -- 172.21.15.67:0/65542663 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/container_init}]=0 v6)=0 v6) ==== 142+0+0 (secure 0 0 0) 0x7f77fc02c350 con 0x7f780c108110 2024-05-03T21:11:23.360 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:22.613+0000 7f7812f53700 1 -- 172.21.15.67:0/65542663 >> [v2:172.21.15.67:6800/2576113038,v1:172.21.15.67:6801/2576113038] conn(0x7f77f403e470 msgr2=0x7f77f4040930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:11:23.360 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:22.613+0000 7f7812f53700 1 --2- 172.21.15.67:0/65542663 >> [v2:172.21.15.67:6800/2576113038,v1:172.21.15.67:6801/2576113038] conn(0x7f77f403e470 0x7f77f4040930 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f7800006fd0 tx=0x7f7800006e40 comp rx=0 tx=0).stop 2024-05-03T21:11:23.360 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:22.613+0000 7f7812f53700 1 -- 172.21.15.67:0/65542663 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f780c108110 msgr2=0x7f780c19d2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:11:23.360 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:22.613+0000 7f7812f53700 1 --2- 172.21.15.67:0/65542663 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f780c108110 0x7f780c19d2a0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f77fc003fb0 tx=0x7f77fc011bd0 comp rx=0 tx=0).stop 2024-05-03T21:11:23.360 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:22.613+0000 7f7812f53700 1 -- 172.21.15.67:0/65542663 shutdown_connections 2024-05-03T21:11:23.360 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:22.613+0000 7f7812f53700 1 --2- 172.21.15.67:0/65542663 >> [v2:172.21.15.67:6800/2576113038,v1:172.21.15.67:6801/2576113038] conn(0x7f77f403e470 0x7f77f4040930 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:11:23.360 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:22.613+0000 7f7812f53700 1 --2- 172.21.15.67:0/65542663 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f780c108110 0x7f780c19d2a0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:11:23.360 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:22.613+0000 7f7812f53700 1 -- 172.21.15.67:0/65542663 >> 172.21.15.67:0/65542663 conn(0x7f780c1037b0 msgr2=0x7f780c104680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T21:11:23.360 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:22.613+0000 7f7812f53700 1 -- 172.21.15.67:0/65542663 shutdown_connections 2024-05-03T21:11:23.360 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:22.613+0000 7f7812f53700 1 -- 172.21.15.67:0/65542663 wait complete. 2024-05-03T21:11:23.971 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:11:23 smithi067 ceph-mon[29730]: from='client.? 172.21.15.67:0/65542663' entity='client.admin' 2024-05-03T21:11:23.971 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:11:23 smithi067 ceph-mon[29730]: from='mgr.14118 172.21.15.67:0/4082808469' entity='mgr.x' 2024-05-03T21:11:23.971 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:11:23 smithi067 ceph-mon[29730]: from='mgr.14118 172.21.15.67:0/4082808469' entity='mgr.x' 2024-05-03T21:11:24.820 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:23.822+0000 7fdfd1916700 1 Processor -- start 2024-05-03T21:11:24.820 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:23.823+0000 7fdfd1916700 1 -- start start 2024-05-03T21:11:24.820 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:23.823+0000 7fdfd1916700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdfcc107b30 0x7fdfcc107f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:11:24.820 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:23.823+0000 7fdfd1916700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fdfcc07ce50 con 0x7fdfcc107b30 2024-05-03T21:11:24.821 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:23.823+0000 7fdfcaffd700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdfcc107b30 0x7fdfcc107f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:11:24.821 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:23.823+0000 7fdfcaffd700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdfcc107b30 0x7fdfcc107f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:57518/0 (socket says 172.21.15.67:57518) 2024-05-03T21:11:24.821 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:23.823+0000 7fdfcaffd700 1 -- 172.21.15.67:0/1807533714 learned_addr learned my addr 172.21.15.67:0/1807533714 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-03T21:11:24.821 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:23.823+0000 7fdfcaffd700 1 -- 172.21.15.67:0/1807533714 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdfcc108490 con 0x7fdfcc107b30 2024-05-03T21:11:24.821 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:23.823+0000 7fdfcaffd700 1 --2- 172.21.15.67:0/1807533714 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdfcc107b30 0x7fdfcc107f50 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fdfbc0098a0 tx=0x7fdfbc009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8fe4bf96262b8280 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T21:11:24.821 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:23.824+0000 7fdfc9ffb700 1 -- 172.21.15.67:0/1807533714 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fdfbc004160 con 0x7fdfcc107b30 2024-05-03T21:11:24.821 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:23.824+0000 7fdfc9ffb700 1 -- 172.21.15.67:0/1807533714 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(28 keys) ==== 1170+0+0 (secure 0 0 0) 0x7fdfbc004300 con 0x7fdfcc107b30 2024-05-03T21:11:24.821 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:23.824+0000 7fdfc9ffb700 1 -- 172.21.15.67:0/1807533714 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fdfbc011c20 con 0x7fdfcc107b30 2024-05-03T21:11:24.821 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:23.824+0000 7fdfd1916700 1 -- 172.21.15.67:0/1807533714 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdfcc107b30 msgr2=0x7fdfcc107f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:11:24.821 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:23.824+0000 7fdfd1916700 1 --2- 172.21.15.67:0/1807533714 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdfcc107b30 0x7fdfcc107f50 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fdfbc0098a0 tx=0x7fdfbc009bb0 comp rx=0 tx=0).stop 2024-05-03T21:11:24.821 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:23.824+0000 7fdfd1916700 1 -- 172.21.15.67:0/1807533714 shutdown_connections 2024-05-03T21:11:24.821 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:23.824+0000 7fdfd1916700 1 --2- 172.21.15.67:0/1807533714 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdfcc107b30 0x7fdfcc107f50 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:11:24.821 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:23.824+0000 7fdfd1916700 1 -- 172.21.15.67:0/1807533714 >> 172.21.15.67:0/1807533714 conn(0x7fdfcc103750 msgr2=0x7fdfcc105b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T21:11:24.822 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:23.824+0000 7fdfd1916700 1 -- 172.21.15.67:0/1807533714 shutdown_connections 2024-05-03T21:11:24.822 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:23.824+0000 7fdfd1916700 1 -- 172.21.15.67:0/1807533714 wait complete. 2024-05-03T21:11:24.822 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:23.825+0000 7fdfd1916700 1 Processor -- start 2024-05-03T21:11:24.822 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:23.825+0000 7fdfd1916700 1 -- start start 2024-05-03T21:11:24.822 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:23.825+0000 7fdfd1916700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdfcc107b30 0x7fdfcc19f4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:11:24.822 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:23.825+0000 7fdfd1916700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fdfcc1095c0 con 0x7fdfcc107b30 2024-05-03T21:11:24.822 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:23.825+0000 7fdfcaffd700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdfcc107b30 0x7fdfcc19f4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:11:24.822 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:23.826+0000 7fdfcaffd700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdfcc107b30 0x7fdfcc19f4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:57530/0 (socket says 172.21.15.67:57530) 2024-05-03T21:11:24.822 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:23.826+0000 7fdfcaffd700 1 -- 172.21.15.67:0/1646863410 learned_addr learned my addr 172.21.15.67:0/1646863410 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-03T21:11:24.822 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:23.826+0000 7fdfcaffd700 1 -- 172.21.15.67:0/1646863410 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdfcc19fa00 con 0x7fdfcc107b30 2024-05-03T21:11:24.822 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:23.826+0000 7fdfcaffd700 1 --2- 172.21.15.67:0/1646863410 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdfcc107b30 0x7fdfcc19f4c0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fdfbc011670 tx=0x7fdfbc011a70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T21:11:24.822 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:23.826+0000 7fdfd0914700 1 -- 172.21.15.67:0/1646863410 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fdfbc0036a0 con 0x7fdfcc107b30 2024-05-03T21:11:24.822 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:23.826+0000 7fdfd1916700 1 -- 172.21.15.67:0/1646863410 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdfcc19fc90 con 0x7fdfcc107b30 2024-05-03T21:11:24.823 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:23.826+0000 7fdfd0914700 1 -- 172.21.15.67:0/1646863410 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(28 keys) ==== 1170+0+0 (secure 0 0 0) 0x7fdfbc003d40 con 0x7fdfcc107b30 2024-05-03T21:11:24.823 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:23.826+0000 7fdfd0914700 1 -- 172.21.15.67:0/1646863410 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fdfbc0193c0 con 0x7fdfcc107b30 2024-05-03T21:11:24.823 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:23.826+0000 7fdfd1916700 1 -- 172.21.15.67:0/1646863410 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdfcc1a2980 con 0x7fdfcc107b30 2024-05-03T21:11:24.823 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:23.827+0000 7fdfd0914700 1 -- 172.21.15.67:0/1646863410 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 8) ==== 50222+0+0 (secure 0 0 0) 0x7fdfbc0196c0 con 0x7fdfcc107b30 2024-05-03T21:11:24.823 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:23.827+0000 7fdfd1916700 1 -- 172.21.15.67:0/1646863410 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdfcc064b00 con 0x7fdfcc107b30 2024-05-03T21:11:24.823 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:23.828+0000 7fdfd0914700 1 --2- 172.21.15.67:0/1646863410 >> [v2:172.21.15.67:6800/2576113038,v1:172.21.15.67:6801/2576113038] conn(0x7fdfb403e420 0x7fdfb40408e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:11:24.823 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:23.828+0000 7fdfd0914700 1 -- 172.21.15.67:0/1646863410 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7fdfbc053770 con 0x7fdfcc107b30 2024-05-03T21:11:24.823 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:23.828+0000 7fdfca7fc700 1 --2- 172.21.15.67:0/1646863410 >> [v2:172.21.15.67:6800/2576113038,v1:172.21.15.67:6801/2576113038] conn(0x7fdfb403e420 0x7fdfb40408e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:11:24.823 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:23.828+0000 7fdfca7fc700 1 --2- 172.21.15.67:0/1646863410 >> [v2:172.21.15.67:6800/2576113038,v1:172.21.15.67:6801/2576113038] conn(0x7fdfb403e420 0x7fdfb40408e0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fdfc0006fd0 tx=0x7fdfc0006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T21:11:24.823 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:23.833+0000 7fdfd0914700 1 -- 172.21.15.67:0/1646863410 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191839 (secure 0 0 0) 0x7fdfbc017e90 con 0x7fdfcc107b30 2024-05-03T21:11:24.823 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:24.071+0000 7fdfd1916700 1 -- 172.21.15.67:0/1646863410 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command([{prefix=config set, name=mgr/dashboard/ssl_server_port}] v 0) -- 0x7fdfcc02e7d0 con 0x7fdfcc107b30 2024-05-03T21:11:24.823 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:24.074+0000 7fdfd0914700 1 -- 172.21.15.67:0/1646863410 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/dashboard/ssl_server_port}]=0 v8)=0 v8) ==== 130+0+0 (secure 0 0 0) 0x7fdfbc055cf0 con 0x7fdfcc107b30 2024-05-03T21:11:24.823 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:24.076+0000 7fdfd1916700 1 -- 172.21.15.67:0/1646863410 >> [v2:172.21.15.67:6800/2576113038,v1:172.21.15.67:6801/2576113038] conn(0x7fdfb403e420 msgr2=0x7fdfb40408e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:11:24.823 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:24.076+0000 7fdfd1916700 1 --2- 172.21.15.67:0/1646863410 >> [v2:172.21.15.67:6800/2576113038,v1:172.21.15.67:6801/2576113038] conn(0x7fdfb403e420 0x7fdfb40408e0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fdfc0006fd0 tx=0x7fdfc0006e40 comp rx=0 tx=0).stop 2024-05-03T21:11:24.824 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:24.076+0000 7fdfd1916700 1 -- 172.21.15.67:0/1646863410 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdfcc107b30 msgr2=0x7fdfcc19f4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:11:24.824 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:24.076+0000 7fdfd1916700 1 --2- 172.21.15.67:0/1646863410 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdfcc107b30 0x7fdfcc19f4c0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fdfbc011670 tx=0x7fdfbc011a70 comp rx=0 tx=0).stop 2024-05-03T21:11:24.824 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:24.076+0000 7fdfd1916700 1 -- 172.21.15.67:0/1646863410 shutdown_connections 2024-05-03T21:11:24.824 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:24.076+0000 7fdfd1916700 1 --2- 172.21.15.67:0/1646863410 >> [v2:172.21.15.67:6800/2576113038,v1:172.21.15.67:6801/2576113038] conn(0x7fdfb403e420 0x7fdfb40408e0 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:11:24.824 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:24.076+0000 7fdfd1916700 1 --2- 172.21.15.67:0/1646863410 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdfcc107b30 0x7fdfcc19f4c0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:11:24.824 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:24.076+0000 7fdfd1916700 1 -- 172.21.15.67:0/1646863410 >> 172.21.15.67:0/1646863410 conn(0x7fdfcc103750 msgr2=0x7fdfcc196130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T21:11:24.824 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:24.076+0000 7fdfd1916700 1 -- 172.21.15.67:0/1646863410 shutdown_connections 2024-05-03T21:11:24.824 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:24.076+0000 7fdfd1916700 1 -- 172.21.15.67:0/1646863410 wait complete. 2024-05-03T21:11:24.824 INFO:teuthology.orchestra.run.smithi067.stdout:Enabling the dashboard module... 2024-05-03T21:11:25.417 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:11:25 smithi067 ceph-mon[29730]: from='client.? 172.21.15.67:0/1646863410' entity='client.admin' 2024-05-03T21:11:26.618 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:11:26 smithi067 ceph-mon[29730]: from='mgr.14118 172.21.15.67:0/4082808469' entity='mgr.x' 2024-05-03T21:11:26.618 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:11:26 smithi067 ceph-mon[29730]: from='client.? 172.21.15.67:0/1571740953' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2024-05-03T21:11:26.618 INFO:journalctl@ceph.mgr.x.smithi067.stdout:May 03 21:11:26 smithi067 ceph-713bc57a-0991-11ef-bc97-c7b262605968-mgr-x[29964]: ignoring --setuser ceph since I am not root 2024-05-03T21:11:26.618 INFO:journalctl@ceph.mgr.x.smithi067.stdout:May 03 21:11:26 smithi067 ceph-713bc57a-0991-11ef-bc97-c7b262605968-mgr-x[29964]: ignoring --setgroup ceph since I am not root 2024-05-03T21:11:26.619 INFO:journalctl@ceph.mgr.x.smithi067.stdout:May 03 21:11:26 smithi067 ceph-713bc57a-0991-11ef-bc97-c7b262605968-mgr-x[29964]: 2024-05-03T21:11:26.435+0000 7f14295b8200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-05-03T21:11:26.926 INFO:journalctl@ceph.mgr.x.smithi067.stdout:May 03 21:11:26 smithi067 ceph-713bc57a-0991-11ef-bc97-c7b262605968-mgr-x[29964]: 2024-05-03T21:11:26.644+0000 7f14295b8200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-05-03T21:11:26.926 INFO:journalctl@ceph.mgr.x.smithi067.stdout:May 03 21:11:26 smithi067 ceph-713bc57a-0991-11ef-bc97-c7b262605968-mgr-x[29964]: 2024-05-03T21:11:26.754+0000 7f14295b8200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-05-03T21:11:26.929 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:25.601+0000 7fe749959700 1 Processor -- start 2024-05-03T21:11:26.935 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:25.601+0000 7fe749959700 1 -- start start 2024-05-03T21:11:26.935 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:25.601+0000 7fe749959700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe744109d20 0x7fe74410c110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:11:26.935 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:25.601+0000 7fe749959700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fe74407ce70 con 0x7fe744109d20 2024-05-03T21:11:26.935 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:25.601+0000 7fe742ffd700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe744109d20 0x7fe74410c110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:11:26.935 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:25.601+0000 7fe742ffd700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe744109d20 0x7fe74410c110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:57532/0 (socket says 172.21.15.67:57532) 2024-05-03T21:11:26.935 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:25.601+0000 7fe742ffd700 1 -- 172.21.15.67:0/846833222 learned_addr learned my addr 172.21.15.67:0/846833222 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-03T21:11:26.935 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:25.602+0000 7fe742ffd700 1 -- 172.21.15.67:0/846833222 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe74410c650 con 0x7fe744109d20 2024-05-03T21:11:26.935 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:25.602+0000 7fe742ffd700 1 --2- 172.21.15.67:0/846833222 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe744109d20 0x7fe74410c110 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fe7340098a0 tx=0x7fe734009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e5a2d44f33b98985 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T21:11:26.935 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:25.602+0000 7fe741ffb700 1 -- 172.21.15.67:0/846833222 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe734004160 con 0x7fe744109d20 2024-05-03T21:11:26.935 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:25.602+0000 7fe741ffb700 1 -- 172.21.15.67:0/846833222 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(28 keys) ==== 1170+0+0 (secure 0 0 0) 0x7fe734004300 con 0x7fe744109d20 2024-05-03T21:11:26.935 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:25.602+0000 7fe741ffb700 1 -- 172.21.15.67:0/846833222 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe734011c20 con 0x7fe744109d20 2024-05-03T21:11:26.935 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:25.603+0000 7fe749959700 1 -- 172.21.15.67:0/846833222 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe744109d20 msgr2=0x7fe74410c110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:11:26.935 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:25.603+0000 7fe749959700 1 --2- 172.21.15.67:0/846833222 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe744109d20 0x7fe74410c110 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fe7340098a0 tx=0x7fe734009bb0 comp rx=0 tx=0).stop 2024-05-03T21:11:26.936 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:25.603+0000 7fe749959700 1 -- 172.21.15.67:0/846833222 shutdown_connections 2024-05-03T21:11:26.936 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:25.603+0000 7fe749959700 1 --2- 172.21.15.67:0/846833222 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe744109d20 0x7fe74410c110 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:11:26.936 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:25.603+0000 7fe749959700 1 -- 172.21.15.67:0/846833222 >> 172.21.15.67:0/846833222 conn(0x7fe744103750 msgr2=0x7fe744105b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T21:11:26.936 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:25.603+0000 7fe749959700 1 -- 172.21.15.67:0/846833222 shutdown_connections 2024-05-03T21:11:26.936 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:25.603+0000 7fe749959700 1 -- 172.21.15.67:0/846833222 wait complete. 2024-05-03T21:11:26.936 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:25.603+0000 7fe749959700 1 Processor -- start 2024-05-03T21:11:26.936 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:25.603+0000 7fe749959700 1 -- start start 2024-05-03T21:11:26.936 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:25.604+0000 7fe749959700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe744109d20 0x7fe74419f4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:11:26.936 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:25.604+0000 7fe749959700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fe74410d6c0 con 0x7fe744109d20 2024-05-03T21:11:26.936 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:25.604+0000 7fe742ffd700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe744109d20 0x7fe74419f4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:11:26.936 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:25.604+0000 7fe742ffd700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe744109d20 0x7fe74419f4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:57548/0 (socket says 172.21.15.67:57548) 2024-05-03T21:11:26.936 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:25.604+0000 7fe742ffd700 1 -- 172.21.15.67:0/1571740953 learned_addr learned my addr 172.21.15.67:0/1571740953 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-03T21:11:26.936 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:25.604+0000 7fe742ffd700 1 -- 172.21.15.67:0/1571740953 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe74419f9f0 con 0x7fe744109d20 2024-05-03T21:11:26.936 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:25.604+0000 7fe742ffd700 1 --2- 172.21.15.67:0/1571740953 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe744109d20 0x7fe74419f4b0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fe734011670 tx=0x7fe734011a70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T21:11:26.936 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:25.604+0000 7fe748957700 1 -- 172.21.15.67:0/1571740953 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe7340036a0 con 0x7fe744109d20 2024-05-03T21:11:26.937 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:25.604+0000 7fe749959700 1 -- 172.21.15.67:0/1571740953 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe74419fc80 con 0x7fe744109d20 2024-05-03T21:11:26.937 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:25.604+0000 7fe748957700 1 -- 172.21.15.67:0/1571740953 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(28 keys) ==== 1170+0+0 (secure 0 0 0) 0x7fe734003d40 con 0x7fe744109d20 2024-05-03T21:11:26.937 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:25.605+0000 7fe748957700 1 -- 172.21.15.67:0/1571740953 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe7340193c0 con 0x7fe744109d20 2024-05-03T21:11:26.937 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:25.605+0000 7fe749959700 1 -- 172.21.15.67:0/1571740953 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe7441a2970 con 0x7fe744109d20 2024-05-03T21:11:26.937 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:25.605+0000 7fe749959700 1 -- 172.21.15.67:0/1571740953 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe744064b00 con 0x7fe744109d20 2024-05-03T21:11:26.937 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:25.606+0000 7fe748957700 1 -- 172.21.15.67:0/1571740953 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 8) ==== 50222+0+0 (secure 0 0 0) 0x7fe7340196e0 con 0x7fe744109d20 2024-05-03T21:11:26.937 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:25.606+0000 7fe748957700 1 --2- 172.21.15.67:0/1571740953 >> [v2:172.21.15.67:6800/2576113038,v1:172.21.15.67:6801/2576113038] conn(0x7fe72c03e420 0x7fe72c0408e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:11:26.937 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:25.606+0000 7fe748957700 1 -- 172.21.15.67:0/1571740953 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7fe734053940 con 0x7fe744109d20 2024-05-03T21:11:26.937 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:25.606+0000 7fe7427fc700 1 --2- 172.21.15.67:0/1571740953 >> [v2:172.21.15.67:6800/2576113038,v1:172.21.15.67:6801/2576113038] conn(0x7fe72c03e420 0x7fe72c0408e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:11:26.937 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:25.606+0000 7fe7427fc700 1 --2- 172.21.15.67:0/1571740953 >> [v2:172.21.15.67:6800/2576113038,v1:172.21.15.67:6801/2576113038] conn(0x7fe72c03e420 0x7fe72c0408e0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fe738006fd0 tx=0x7fe738006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T21:11:26.937 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:25.616+0000 7fe748957700 1 -- 172.21.15.67:0/1571740953 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191839 (secure 0 0 0) 0x7fe734023390 con 0x7fe744109d20 2024-05-03T21:11:26.937 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:25.898+0000 7fe749959700 1 -- 172.21.15.67:0/1571740953 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "dashboard"} v 0) -- 0x7fe7441a2b40 con 0x7fe744109d20 2024-05-03T21:11:26.937 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:26.234+0000 7fe748957700 1 -- 172.21.15.67:0/1571740953 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "dashboard"}]=0 v9) ==== 88+0+0 (secure 0 0 0) 0x7fe73401a580 con 0x7fe744109d20 2024-05-03T21:11:26.938 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:26.236+0000 7fe748957700 1 -- 172.21.15.67:0/1571740953 <== mon.0 v2:172.21.15.67:3300/0 8 ==== mgrmap(e 9) ==== 50235+0+0 (secure 0 0 0) 0x7fe734018350 con 0x7fe744109d20 2024-05-03T21:11:26.938 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:26.237+0000 7fe749959700 1 -- 172.21.15.67:0/1571740953 >> [v2:172.21.15.67:6800/2576113038,v1:172.21.15.67:6801/2576113038] conn(0x7fe72c03e420 msgr2=0x7fe72c0408e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:11:26.938 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:26.237+0000 7fe749959700 1 --2- 172.21.15.67:0/1571740953 >> [v2:172.21.15.67:6800/2576113038,v1:172.21.15.67:6801/2576113038] conn(0x7fe72c03e420 0x7fe72c0408e0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fe738006fd0 tx=0x7fe738006e40 comp rx=0 tx=0).stop 2024-05-03T21:11:26.938 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:26.237+0000 7fe749959700 1 -- 172.21.15.67:0/1571740953 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe744109d20 msgr2=0x7fe74419f4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:11:26.938 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:26.238+0000 7fe749959700 1 --2- 172.21.15.67:0/1571740953 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe744109d20 0x7fe74419f4b0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fe734011670 tx=0x7fe734011a70 comp rx=0 tx=0).stop 2024-05-03T21:11:26.938 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:26.238+0000 7fe749959700 1 -- 172.21.15.67:0/1571740953 shutdown_connections 2024-05-03T21:11:26.938 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:26.238+0000 7fe749959700 1 --2- 172.21.15.67:0/1571740953 >> [v2:172.21.15.67:6800/2576113038,v1:172.21.15.67:6801/2576113038] conn(0x7fe72c03e420 0x7fe72c0408e0 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:11:26.938 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:26.238+0000 7fe749959700 1 --2- 172.21.15.67:0/1571740953 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe744109d20 0x7fe74419f4b0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:11:26.938 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:26.238+0000 7fe749959700 1 -- 172.21.15.67:0/1571740953 >> 172.21.15.67:0/1571740953 conn(0x7fe744103750 msgr2=0x7fe7441042c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T21:11:26.938 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:26.238+0000 7fe749959700 1 -- 172.21.15.67:0/1571740953 shutdown_connections 2024-05-03T21:11:26.938 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:26.238+0000 7fe749959700 1 -- 172.21.15.67:0/1571740953 wait complete. 2024-05-03T21:11:27.201 INFO:journalctl@ceph.mgr.x.smithi067.stdout:May 03 21:11:27 smithi067 ceph-713bc57a-0991-11ef-bc97-c7b262605968-mgr-x[29964]: 2024-05-03T21:11:27.109+0000 7f14295b8200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-05-03T21:11:27.201 INFO:journalctl@ceph.mgr.x.smithi067.stdout:May 03 21:11:27 smithi067 ceph-713bc57a-0991-11ef-bc97-c7b262605968-mgr-x[29964]: 2024-05-03T21:11:27.198+0000 7f14295b8200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-05-03T21:11:27.453 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:11:27 smithi067 ceph-mon[29730]: from='client.? 172.21.15.67:0/1571740953' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2024-05-03T21:11:27.453 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:11:27 smithi067 ceph-mon[29730]: mgrmap e9: x(active, since 18s) 2024-05-03T21:11:28.275 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout { 2024-05-03T21:11:28.275 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "epoch": 9, 2024-05-03T21:11:28.275 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "available": true, 2024-05-03T21:11:28.275 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "active_name": "x", 2024-05-03T21:11:28.275 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-05-03T21:11:28.275 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout } 2024-05-03T21:11:28.275 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:27.450+0000 7fa4f7103700 1 Processor -- start 2024-05-03T21:11:28.275 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:27.451+0000 7fa4f7103700 1 -- start start 2024-05-03T21:11:28.275 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:27.451+0000 7fa4f7103700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa4f0108110 0x7fa4f0108530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:11:28.275 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:27.451+0000 7fa4f7103700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fa4f0108b00 con 0x7fa4f0108110 2024-05-03T21:11:28.276 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:27.451+0000 7fa4f4e9f700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa4f0108110 0x7fa4f0108530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:11:28.276 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:27.451+0000 7fa4f4e9f700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa4f0108110 0x7fa4f0108530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:33746/0 (socket says 172.21.15.67:33746) 2024-05-03T21:11:28.276 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:27.451+0000 7fa4f4e9f700 1 -- 172.21.15.67:0/4289061819 learned_addr learned my addr 172.21.15.67:0/4289061819 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-03T21:11:28.276 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:27.451+0000 7fa4f4e9f700 1 -- 172.21.15.67:0/4289061819 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa4f0109310 con 0x7fa4f0108110 2024-05-03T21:11:28.276 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:27.452+0000 7fa4f4e9f700 1 --2- 172.21.15.67:0/4289061819 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa4f0108110 0x7fa4f0108530 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fa4e40098a0 tx=0x7fa4e4009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1b1c28e7bf12e886 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T21:11:28.276 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:27.452+0000 7fa4ef7fe700 1 -- 172.21.15.67:0/4289061819 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa4e4004160 con 0x7fa4f0108110 2024-05-03T21:11:28.276 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:27.452+0000 7fa4ef7fe700 1 -- 172.21.15.67:0/4289061819 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(28 keys) ==== 1170+0+0 (secure 0 0 0) 0x7fa4e4004300 con 0x7fa4f0108110 2024-05-03T21:11:28.276 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:27.452+0000 7fa4ef7fe700 1 -- 172.21.15.67:0/4289061819 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa4e4011c20 con 0x7fa4f0108110 2024-05-03T21:11:28.276 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:27.453+0000 7fa4f7103700 1 -- 172.21.15.67:0/4289061819 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa4f0108110 msgr2=0x7fa4f0108530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:11:28.276 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:27.453+0000 7fa4f7103700 1 --2- 172.21.15.67:0/4289061819 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa4f0108110 0x7fa4f0108530 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fa4e40098a0 tx=0x7fa4e4009bb0 comp rx=0 tx=0).stop 2024-05-03T21:11:28.276 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:27.453+0000 7fa4f7103700 1 -- 172.21.15.67:0/4289061819 shutdown_connections 2024-05-03T21:11:28.276 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:27.453+0000 7fa4f7103700 1 --2- 172.21.15.67:0/4289061819 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa4f0108110 0x7fa4f0108530 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:11:28.276 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:27.453+0000 7fa4f7103700 1 -- 172.21.15.67:0/4289061819 >> 172.21.15.67:0/4289061819 conn(0x7fa4f01037b0 msgr2=0x7fa4f0105bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T21:11:28.276 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:27.453+0000 7fa4f7103700 1 -- 172.21.15.67:0/4289061819 shutdown_connections 2024-05-03T21:11:28.277 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:27.453+0000 7fa4f7103700 1 -- 172.21.15.67:0/4289061819 wait complete. 2024-05-03T21:11:28.277 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:27.454+0000 7fa4f7103700 1 Processor -- start 2024-05-03T21:11:28.277 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:27.454+0000 7fa4f7103700 1 -- start start 2024-05-03T21:11:28.277 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:27.454+0000 7fa4f7103700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa4f0108110 0x7fa4f019b070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:11:28.277 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:27.454+0000 7fa4f7103700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fa4f0076290 con 0x7fa4f0108110 2024-05-03T21:11:28.277 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:27.454+0000 7fa4f4e9f700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa4f0108110 0x7fa4f019b070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:11:28.277 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:27.454+0000 7fa4f4e9f700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa4f0108110 0x7fa4f019b070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:33750/0 (socket says 172.21.15.67:33750) 2024-05-03T21:11:28.277 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:27.454+0000 7fa4f4e9f700 1 -- 172.21.15.67:0/796014281 learned_addr learned my addr 172.21.15.67:0/796014281 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-03T21:11:28.277 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:27.454+0000 7fa4f4e9f700 1 -- 172.21.15.67:0/796014281 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa4f019b5b0 con 0x7fa4f0108110 2024-05-03T21:11:28.277 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:27.455+0000 7fa4f4e9f700 1 --2- 172.21.15.67:0/796014281 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa4f0108110 0x7fa4f019b070 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fa4e40118e0 tx=0x7fa4e4011ca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T21:11:28.277 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:27.455+0000 7fa4edffb700 1 -- 172.21.15.67:0/796014281 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa4e40038b0 con 0x7fa4f0108110 2024-05-03T21:11:28.277 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:27.455+0000 7fa4f7103700 1 -- 172.21.15.67:0/796014281 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa4f019b840 con 0x7fa4f0108110 2024-05-03T21:11:28.277 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:27.455+0000 7fa4edffb700 1 -- 172.21.15.67:0/796014281 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(28 keys) ==== 1170+0+0 (secure 0 0 0) 0x7fa4e4020070 con 0x7fa4f0108110 2024-05-03T21:11:28.277 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:27.455+0000 7fa4edffb700 1 -- 172.21.15.67:0/796014281 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa4e4019420 con 0x7fa4f0108110 2024-05-03T21:11:28.277 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:27.455+0000 7fa4f7103700 1 -- 172.21.15.67:0/796014281 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa4f019e530 con 0x7fa4f0108110 2024-05-03T21:11:28.278 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:27.456+0000 7fa4f7103700 1 -- 172.21.15.67:0/796014281 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa4f0064b00 con 0x7fa4f0108110 2024-05-03T21:11:28.278 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:27.456+0000 7fa4edffb700 1 -- 172.21.15.67:0/796014281 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 9) ==== 50235+0+0 (secure 0 0 0) 0x7fa4e4019660 con 0x7fa4f0108110 2024-05-03T21:11:28.278 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:27.457+0000 7fa4edffb700 1 --2- 172.21.15.67:0/796014281 >> [v2:172.21.15.67:6800/2576113038,v1:172.21.15.67:6801/2576113038] conn(0x7fa4e0040710 0x7fa4e0042bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:11:28.278 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:27.457+0000 7fa4edffb700 1 -- 172.21.15.67:0/796014281 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7fa4e40538a0 con 0x7fa4f0108110 2024-05-03T21:11:28.278 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:27.457+0000 7fa4effff700 1 -- 172.21.15.67:0/796014281 >> [v2:172.21.15.67:6800/2576113038,v1:172.21.15.67:6801/2576113038] conn(0x7fa4e0040710 msgr2=0x7fa4e0042bd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.67:6800/2576113038 2024-05-03T21:11:28.278 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:27.457+0000 7fa4effff700 1 --2- 172.21.15.67:0/796014281 >> [v2:172.21.15.67:6800/2576113038,v1:172.21.15.67:6801/2576113038] conn(0x7fa4e0040710 0x7fa4e0042bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-03T21:11:28.278 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:27.466+0000 7fa4edffb700 1 -- 172.21.15.67:0/796014281 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191839 (secure 0 0 0) 0x7fa4e4003b70 con 0x7fa4f0108110 2024-05-03T21:11:28.278 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:27.658+0000 7fa4effff700 1 -- 172.21.15.67:0/796014281 >> [v2:172.21.15.67:6800/2576113038,v1:172.21.15.67:6801/2576113038] conn(0x7fa4e0040710 msgr2=0x7fa4e0042bd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.67:6800/2576113038 2024-05-03T21:11:28.278 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:27.658+0000 7fa4effff700 1 --2- 172.21.15.67:0/796014281 >> [v2:172.21.15.67:6800/2576113038,v1:172.21.15.67:6801/2576113038] conn(0x7fa4e0040710 0x7fa4e0042bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-05-03T21:11:28.278 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:27.784+0000 7fa4f7103700 1 -- 172.21.15.67:0/796014281 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) -- 0x7fa4f01945d0 con 0x7fa4f0108110 2024-05-03T21:11:28.278 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:27.784+0000 7fa4edffb700 1 -- 172.21.15.67:0/796014281 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v9) ==== 56+0+88 (secure 0 0 0) 0x7fa4f01945d0 con 0x7fa4f0108110 2024-05-03T21:11:28.278 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:27.786+0000 7fa4f7103700 1 -- 172.21.15.67:0/796014281 >> [v2:172.21.15.67:6800/2576113038,v1:172.21.15.67:6801/2576113038] conn(0x7fa4e0040710 msgr2=0x7fa4e0042bd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-03T21:11:28.278 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:27.786+0000 7fa4f7103700 1 --2- 172.21.15.67:0/796014281 >> [v2:172.21.15.67:6800/2576113038,v1:172.21.15.67:6801/2576113038] conn(0x7fa4e0040710 0x7fa4e0042bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:11:28.279 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:27.786+0000 7fa4f7103700 1 -- 172.21.15.67:0/796014281 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa4f0108110 msgr2=0x7fa4f019b070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:11:28.279 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:27.786+0000 7fa4f7103700 1 --2- 172.21.15.67:0/796014281 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa4f0108110 0x7fa4f019b070 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fa4e40118e0 tx=0x7fa4e4011ca0 comp rx=0 tx=0).stop 2024-05-03T21:11:28.279 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:27.786+0000 7fa4f7103700 1 -- 172.21.15.67:0/796014281 shutdown_connections 2024-05-03T21:11:28.279 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:27.786+0000 7fa4f7103700 1 --2- 172.21.15.67:0/796014281 >> [v2:172.21.15.67:6800/2576113038,v1:172.21.15.67:6801/2576113038] conn(0x7fa4e0040710 0x7fa4e0042bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:11:28.279 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:27.786+0000 7fa4f7103700 1 --2- 172.21.15.67:0/796014281 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa4f0108110 0x7fa4f019b070 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:11:28.279 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:27.786+0000 7fa4f7103700 1 -- 172.21.15.67:0/796014281 >> 172.21.15.67:0/796014281 conn(0x7fa4f01037b0 msgr2=0x7fa4f0191690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T21:11:28.279 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:27.786+0000 7fa4f7103700 1 -- 172.21.15.67:0/796014281 shutdown_connections 2024-05-03T21:11:28.279 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:27.786+0000 7fa4f7103700 1 -- 172.21.15.67:0/796014281 wait complete. 2024-05-03T21:11:28.279 INFO:teuthology.orchestra.run.smithi067.stdout:Waiting for the mgr to restart... 2024-05-03T21:11:28.279 INFO:teuthology.orchestra.run.smithi067.stdout:Waiting for mgr epoch 9... 2024-05-03T21:11:28.397 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:11:28 smithi067 ceph-mon[29730]: from='client.? 172.21.15.67:0/796014281' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-05-03T21:11:28.685 INFO:journalctl@ceph.mgr.x.smithi067.stdout:May 03 21:11:28 smithi067 ceph-713bc57a-0991-11ef-bc97-c7b262605968-mgr-x[29964]: 2024-05-03T21:11:28.498+0000 7f14295b8200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-05-03T21:11:28.685 INFO:journalctl@ceph.mgr.x.smithi067.stdout:May 03 21:11:28 smithi067 ceph-713bc57a-0991-11ef-bc97-c7b262605968-mgr-x[29964]: 2024-05-03T21:11:28.682+0000 7f14295b8200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-05-03T21:11:28.971 INFO:journalctl@ceph.mgr.x.smithi067.stdout:May 03 21:11:28 smithi067 ceph-713bc57a-0991-11ef-bc97-c7b262605968-mgr-x[29964]: 2024-05-03T21:11:28.862+0000 7f14295b8200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-05-03T21:11:29.471 INFO:journalctl@ceph.mgr.x.smithi067.stdout:May 03 21:11:29 smithi067 ceph-713bc57a-0991-11ef-bc97-c7b262605968-mgr-x[29964]: 2024-05-03T21:11:29.165+0000 7f14295b8200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-05-03T21:11:29.471 INFO:journalctl@ceph.mgr.x.smithi067.stdout:May 03 21:11:29 smithi067 ceph-713bc57a-0991-11ef-bc97-c7b262605968-mgr-x[29964]: 2024-05-03T21:11:29.262+0000 7f14295b8200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-05-03T21:11:29.971 INFO:journalctl@ceph.mgr.x.smithi067.stdout:May 03 21:11:29 smithi067 ceph-713bc57a-0991-11ef-bc97-c7b262605968-mgr-x[29964]: 2024-05-03T21:11:29.473+0000 7f14295b8200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-05-03T21:11:29.971 INFO:journalctl@ceph.mgr.x.smithi067.stdout:May 03 21:11:29 smithi067 ceph-713bc57a-0991-11ef-bc97-c7b262605968-mgr-x[29964]: 2024-05-03T21:11:29.674+0000 7f14295b8200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-05-03T21:11:30.470 INFO:journalctl@ceph.mgr.x.smithi067.stdout:May 03 21:11:30 smithi067 ceph-713bc57a-0991-11ef-bc97-c7b262605968-mgr-x[29964]: 2024-05-03T21:11:30.304+0000 7f14295b8200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-05-03T21:11:30.970 INFO:journalctl@ceph.mgr.x.smithi067.stdout:May 03 21:11:30 smithi067 ceph-713bc57a-0991-11ef-bc97-c7b262605968-mgr-x[29964]: 2024-05-03T21:11:30.536+0000 7f14295b8200 -1 mgr[py] Module smb has missing NOTIFY_TYPES member 2024-05-03T21:11:31.465 INFO:journalctl@ceph.mgr.x.smithi067.stdout:May 03 21:11:30 smithi067 ceph-713bc57a-0991-11ef-bc97-c7b262605968-mgr-x[29964]: 2024-05-03T21:11:30.979+0000 7f14295b8200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-05-03T21:11:31.465 INFO:journalctl@ceph.mgr.x.smithi067.stdout:May 03 21:11:31 smithi067 ceph-713bc57a-0991-11ef-bc97-c7b262605968-mgr-x[29964]: 2024-05-03T21:11:31.079+0000 7f14295b8200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-05-03T21:11:31.465 INFO:journalctl@ceph.mgr.x.smithi067.stdout:May 03 21:11:31 smithi067 ceph-713bc57a-0991-11ef-bc97-c7b262605968-mgr-x[29964]: 2024-05-03T21:11:31.463+0000 7f14295b8200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-05-03T21:11:32.191 INFO:journalctl@ceph.mgr.x.smithi067.stdout:May 03 21:11:31 smithi067 ceph-713bc57a-0991-11ef-bc97-c7b262605968-mgr-x[29964]: 2024-05-03T21:11:31.838+0000 7f14295b8200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-05-03T21:11:32.191 INFO:journalctl@ceph.mgr.x.smithi067.stdout:May 03 21:11:31 smithi067 ceph-713bc57a-0991-11ef-bc97-c7b262605968-mgr-x[29964]: 2024-05-03T21:11:31.934+0000 7f14295b8200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-05-03T21:11:32.471 INFO:journalctl@ceph.mgr.x.smithi067.stdout:May 03 21:11:32 smithi067 ceph-713bc57a-0991-11ef-bc97-c7b262605968-mgr-x[29964]: 2024-05-03T21:11:32.189+0000 7f14295b8200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-05-03T21:11:32.471 INFO:journalctl@ceph.mgr.x.smithi067.stdout:May 03 21:11:32 smithi067 ceph-713bc57a-0991-11ef-bc97-c7b262605968-mgr-x[29964]: 2024-05-03T21:11:32.330+0000 7f14295b8200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-05-03T21:11:32.971 INFO:journalctl@ceph.mgr.x.smithi067.stdout:May 03 21:11:32 smithi067 ceph-713bc57a-0991-11ef-bc97-c7b262605968-mgr-x[29964]: 2024-05-03T21:11:32.645+0000 7f14295b8200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-05-03T21:11:32.971 INFO:journalctl@ceph.mgr.x.smithi067.stdout:May 03 21:11:32 smithi067 ceph-713bc57a-0991-11ef-bc97-c7b262605968-mgr-x[29964]: 2024-05-03T21:11:32.764+0000 7f14295b8200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-05-03T21:11:32.971 INFO:journalctl@ceph.mgr.x.smithi067.stdout:May 03 21:11:32 smithi067 ceph-713bc57a-0991-11ef-bc97-c7b262605968-mgr-x[29964]: 2024-05-03T21:11:32.860+0000 7f14295b8200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-05-03T21:11:33.774 INFO:journalctl@ceph.mgr.x.smithi067.stdout:May 03 21:11:33 smithi067 ceph-713bc57a-0991-11ef-bc97-c7b262605968-mgr-x[29964]: 2024-05-03T21:11:33.492+0000 7f14295b8200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-05-03T21:11:33.774 INFO:journalctl@ceph.mgr.x.smithi067.stdout:May 03 21:11:33 smithi067 ceph-713bc57a-0991-11ef-bc97-c7b262605968-mgr-x[29964]: 2024-05-03T21:11:33.719+0000 7f14295b8200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-05-03T21:11:34.221 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:11:33 smithi067 ceph-mon[29730]: Active manager daemon x restarted 2024-05-03T21:11:34.221 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:11:33 smithi067 ceph-mon[29730]: Activating manager daemon x 2024-05-03T21:11:34.221 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:11:33 smithi067 ceph-mon[29730]: osdmap e3: 0 total, 0 up, 0 in 2024-05-03T21:11:34.221 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:11:33 smithi067 ceph-mon[29730]: mgrmap e10: x(active, starting, since 0.00407295s) 2024-05-03T21:11:34.221 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:11:33 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-03T21:11:34.221 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:11:33 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-05-03T21:11:34.221 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:11:33 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-05-03T21:11:34.221 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:11:33 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-05-03T21:11:34.221 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:11:33 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-05-03T21:11:34.221 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:11:33 smithi067 ceph-mon[29730]: Manager daemon x is now available 2024-05-03T21:11:34.221 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:11:33 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T21:11:34.947 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:11:34 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-05-03T21:11:34.947 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:11:34 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-05-03T21:11:34.947 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:11:34 smithi067 ceph-mon[29730]: [03/May/2024:21:11:34] ENGINE Bus STARTING 2024-05-03T21:11:34.947 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:11:34 smithi067 ceph-mon[29730]: mgrmap e11: x(active, since 1.00689s) 2024-05-03T21:11:35.570 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout { 2024-05-03T21:11:35.570 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 11, 2024-05-03T21:11:35.570 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "initialized": true 2024-05-03T21:11:35.570 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout } 2024-05-03T21:11:35.570 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:28.722+0000 7f9fef367700 1 Processor -- start 2024-05-03T21:11:35.570 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:28.722+0000 7f9fef367700 1 -- start start 2024-05-03T21:11:35.570 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:28.722+0000 7f9fef367700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9fe81092d0 0x7f9fe81096f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:11:35.570 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:28.722+0000 7f9fef367700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f9fe8109c30 con 0x7f9fe81092d0 2024-05-03T21:11:35.570 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:28.723+0000 7f9fed103700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9fe81092d0 0x7f9fe81096f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:11:35.571 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:28.723+0000 7f9fed103700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9fe81092d0 0x7f9fe81096f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:33766/0 (socket says 172.21.15.67:33766) 2024-05-03T21:11:35.571 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:28.723+0000 7f9fed103700 1 -- 172.21.15.67:0/2630983698 learned_addr learned my addr 172.21.15.67:0/2630983698 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-03T21:11:35.571 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:28.723+0000 7f9fed103700 1 -- 172.21.15.67:0/2630983698 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9fe8109db0 con 0x7f9fe81092d0 2024-05-03T21:11:35.571 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:28.723+0000 7f9fed103700 1 --2- 172.21.15.67:0/2630983698 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9fe81092d0 0x7f9fe81096f0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f9fd80098a0 tx=0x7f9fd8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5a3baa2c1d555985 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T21:11:35.571 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:28.723+0000 7f9fdffff700 1 -- 172.21.15.67:0/2630983698 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9fd8004160 con 0x7f9fe81092d0 2024-05-03T21:11:35.571 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:28.723+0000 7f9fdffff700 1 -- 172.21.15.67:0/2630983698 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(28 keys) ==== 1170+0+0 (secure 0 0 0) 0x7f9fd8004300 con 0x7f9fe81092d0 2024-05-03T21:11:35.571 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:28.723+0000 7f9fdffff700 1 -- 172.21.15.67:0/2630983698 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9fd8011c20 con 0x7f9fe81092d0 2024-05-03T21:11:35.571 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:28.724+0000 7f9fef367700 1 -- 172.21.15.67:0/2630983698 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9fe81092d0 msgr2=0x7f9fe81096f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:11:35.571 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:28.724+0000 7f9fef367700 1 --2- 172.21.15.67:0/2630983698 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9fe81092d0 0x7f9fe81096f0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f9fd80098a0 tx=0x7f9fd8009bb0 comp rx=0 tx=0).stop 2024-05-03T21:11:35.571 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:28.724+0000 7f9fef367700 1 -- 172.21.15.67:0/2630983698 shutdown_connections 2024-05-03T21:11:35.571 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:28.724+0000 7f9fef367700 1 --2- 172.21.15.67:0/2630983698 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9fe81092d0 0x7f9fe81096f0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:11:35.571 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:28.724+0000 7f9fef367700 1 -- 172.21.15.67:0/2630983698 >> 172.21.15.67:0/2630983698 conn(0x7f9fe8104970 msgr2=0x7f9fe8106d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T21:11:35.571 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:28.724+0000 7f9fef367700 1 -- 172.21.15.67:0/2630983698 shutdown_connections 2024-05-03T21:11:35.571 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:28.724+0000 7f9fef367700 1 -- 172.21.15.67:0/2630983698 wait complete. 2024-05-03T21:11:35.572 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:28.725+0000 7f9fef367700 1 Processor -- start 2024-05-03T21:11:35.572 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:28.725+0000 7f9fef367700 1 -- start start 2024-05-03T21:11:35.572 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:28.725+0000 7f9fef367700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9fe81092d0 0x7f9fe819d290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:11:35.572 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:28.725+0000 7f9fef367700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f9fe810d6f0 con 0x7f9fe81092d0 2024-05-03T21:11:35.572 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:28.726+0000 7f9fed103700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9fe81092d0 0x7f9fe819d290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:11:35.572 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:28.726+0000 7f9fed103700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9fe81092d0 0x7f9fe819d290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:33770/0 (socket says 172.21.15.67:33770) 2024-05-03T21:11:35.572 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:28.726+0000 7f9fed103700 1 -- 172.21.15.67:0/3911617192 learned_addr learned my addr 172.21.15.67:0/3911617192 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-03T21:11:35.572 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:28.726+0000 7f9fed103700 1 -- 172.21.15.67:0/3911617192 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9fe819d7d0 con 0x7f9fe81092d0 2024-05-03T21:11:35.572 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:28.726+0000 7f9fed103700 1 --2- 172.21.15.67:0/3911617192 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9fe81092d0 0x7f9fe819d290 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f9fd8011bf0 tx=0x7f9fd8011cd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T21:11:35.572 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:28.726+0000 7f9fde7fc700 1 -- 172.21.15.67:0/3911617192 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9fd80038b0 con 0x7f9fe81092d0 2024-05-03T21:11:35.572 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:28.726+0000 7f9fde7fc700 1 -- 172.21.15.67:0/3911617192 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(28 keys) ==== 1170+0+0 (secure 0 0 0) 0x7f9fd8003a50 con 0x7f9fe81092d0 2024-05-03T21:11:35.572 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:28.726+0000 7f9fef367700 1 -- 172.21.15.67:0/3911617192 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9fe819da60 con 0x7f9fe81092d0 2024-05-03T21:11:35.572 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:28.726+0000 7f9fde7fc700 1 -- 172.21.15.67:0/3911617192 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9fd80243f0 con 0x7f9fe81092d0 2024-05-03T21:11:35.572 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:28.726+0000 7f9fef367700 1 -- 172.21.15.67:0/3911617192 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9fe81a0750 con 0x7f9fe81092d0 2024-05-03T21:11:35.573 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:28.727+0000 7f9fde7fc700 1 -- 172.21.15.67:0/3911617192 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 9) ==== 50235+0+0 (secure 0 0 0) 0x7f9fd8024710 con 0x7f9fe81092d0 2024-05-03T21:11:35.573 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:28.727+0000 7f9fde7fc700 1 --2- 172.21.15.67:0/3911617192 >> [v2:172.21.15.67:6800/2576113038,v1:172.21.15.67:6801/2576113038] conn(0x7f9fd403e4c0 0x7f9fd4040980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:11:35.573 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:28.727+0000 7f9fde7fc700 1 -- 172.21.15.67:0/3911617192 --> [v2:172.21.15.67:6800/2576113038,v1:172.21.15.67:6801/2576113038] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f9fd4041090 con 0x7f9fd403e4c0 2024-05-03T21:11:35.573 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:28.728+0000 7f9fde7fc700 1 -- 172.21.15.67:0/3911617192 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f9fd8053600 con 0x7f9fe81092d0 2024-05-03T21:11:35.573 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:28.728+0000 7f9fec902700 1 -- 172.21.15.67:0/3911617192 >> [v2:172.21.15.67:6800/2576113038,v1:172.21.15.67:6801/2576113038] conn(0x7f9fd403e4c0 msgr2=0x7f9fd4040980 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.67:6800/2576113038 2024-05-03T21:11:35.573 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:28.728+0000 7f9fec902700 1 --2- 172.21.15.67:0/3911617192 >> [v2:172.21.15.67:6800/2576113038,v1:172.21.15.67:6801/2576113038] conn(0x7f9fd403e4c0 0x7f9fd4040980 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-03T21:11:35.573 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:28.929+0000 7f9fec902700 1 -- 172.21.15.67:0/3911617192 >> [v2:172.21.15.67:6800/2576113038,v1:172.21.15.67:6801/2576113038] conn(0x7f9fd403e4c0 msgr2=0x7f9fd4040980 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.67:6800/2576113038 2024-05-03T21:11:35.573 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:28.929+0000 7f9fec902700 1 --2- 172.21.15.67:0/3911617192 >> [v2:172.21.15.67:6800/2576113038,v1:172.21.15.67:6801/2576113038] conn(0x7f9fd403e4c0 0x7f9fd4040980 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-05-03T21:11:35.573 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:29.330+0000 7f9fec902700 1 -- 172.21.15.67:0/3911617192 >> [v2:172.21.15.67:6800/2576113038,v1:172.21.15.67:6801/2576113038] conn(0x7f9fd403e4c0 msgr2=0x7f9fd4040980 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.67:6800/2576113038 2024-05-03T21:11:35.573 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:29.330+0000 7f9fec902700 1 --2- 172.21.15.67:0/3911617192 >> [v2:172.21.15.67:6800/2576113038,v1:172.21.15.67:6801/2576113038] conn(0x7f9fd403e4c0 0x7f9fd4040980 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-05-03T21:11:35.573 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:30.131+0000 7f9fec902700 1 -- 172.21.15.67:0/3911617192 >> [v2:172.21.15.67:6800/2576113038,v1:172.21.15.67:6801/2576113038] conn(0x7f9fd403e4c0 msgr2=0x7f9fd4040980 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.67:6800/2576113038 2024-05-03T21:11:35.573 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:30.131+0000 7f9fec902700 1 --2- 172.21.15.67:0/3911617192 >> [v2:172.21.15.67:6800/2576113038,v1:172.21.15.67:6801/2576113038] conn(0x7f9fd403e4c0 0x7f9fd4040980 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-05-03T21:11:35.573 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:31.732+0000 7f9fec902700 1 -- 172.21.15.67:0/3911617192 >> [v2:172.21.15.67:6800/2576113038,v1:172.21.15.67:6801/2576113038] conn(0x7f9fd403e4c0 msgr2=0x7f9fd4040980 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.67:6800/2576113038 2024-05-03T21:11:35.573 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:31.732+0000 7f9fec902700 1 --2- 172.21.15.67:0/3911617192 >> [v2:172.21.15.67:6800/2576113038,v1:172.21.15.67:6801/2576113038] conn(0x7f9fd403e4c0 0x7f9fd4040980 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-05-03T21:11:35.574 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:33.723+0000 7f9fde7fc700 1 -- 172.21.15.67:0/3911617192 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mgrmap(e 10) ==== 50002+0+0 (secure 0 0 0) 0x7f9fd8003c00 con 0x7f9fe81092d0 2024-05-03T21:11:35.574 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:33.723+0000 7f9fde7fc700 1 -- 172.21.15.67:0/3911617192 >> [v2:172.21.15.67:6800/2576113038,v1:172.21.15.67:6801/2576113038] conn(0x7f9fd403e4c0 msgr2=0x7f9fd4040980 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-03T21:11:35.574 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:33.723+0000 7f9fde7fc700 1 --2- 172.21.15.67:0/3911617192 >> [v2:172.21.15.67:6800/2576113038,v1:172.21.15.67:6801/2576113038] conn(0x7f9fd403e4c0 0x7f9fd4040980 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:11:35.574 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:34.724+0000 7f9fde7fc700 1 -- 172.21.15.67:0/3911617192 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mgrmap(e 11) ==== 50129+0+0 (secure 0 0 0) 0x7f9fd8023d60 con 0x7f9fe81092d0 2024-05-03T21:11:35.574 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:34.724+0000 7f9fde7fc700 1 --2- 172.21.15.67:0/3911617192 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f9fd4041f90 0x7f9fd4044380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:11:35.574 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:34.724+0000 7f9fde7fc700 1 -- 172.21.15.67:0/3911617192 --> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f9fd4041090 con 0x7f9fd4041f90 2024-05-03T21:11:35.574 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:34.725+0000 7f9fec902700 1 --2- 172.21.15.67:0/3911617192 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f9fd4041f90 0x7f9fd4044380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:11:35.574 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:34.725+0000 7f9fec902700 1 --2- 172.21.15.67:0/3911617192 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f9fd4041f90 0x7f9fd4044380 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f9fe4003a10 tx=0x7f9fe40092b0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T21:11:35.574 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:34.726+0000 7f9fde7fc700 1 -- 172.21.15.67:0/3911617192 <== mgr.14150 v2:172.21.15.67:6800/604451592 1 ==== command_reply(tid 0: 0 ) ==== 8+0+9564 (secure 0 0 0) 0x7f9fd4041090 con 0x7f9fd4041f90 2024-05-03T21:11:35.574 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:34.735+0000 7f9fef367700 1 -- 172.21.15.67:0/3911617192 --> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] -- command(tid 1: {"prefix": "mgr_status"}) -- 0x7f9fe810d870 con 0x7f9fd4041f90 2024-05-03T21:11:35.575 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:34.735+0000 7f9fde7fc700 1 -- 172.21.15.67:0/3911617192 <== mgr.14150 v2:172.21.15.67:6800/604451592 2 ==== command_reply(tid 1: 0 ) ==== 8+0+52 (secure 0 0 0) 0x7f9fe810d870 con 0x7f9fd4041f90 2024-05-03T21:11:35.575 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:34.735+0000 7f9fef367700 1 -- 172.21.15.67:0/3911617192 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f9fd4041f90 msgr2=0x7f9fd4044380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:11:35.575 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:34.735+0000 7f9fef367700 1 --2- 172.21.15.67:0/3911617192 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f9fd4041f90 0x7f9fd4044380 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f9fe4003a10 tx=0x7f9fe40092b0 comp rx=0 tx=0).stop 2024-05-03T21:11:35.575 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:34.735+0000 7f9fef367700 1 -- 172.21.15.67:0/3911617192 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9fe81092d0 msgr2=0x7f9fe819d290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:11:35.575 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:34.735+0000 7f9fef367700 1 --2- 172.21.15.67:0/3911617192 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9fe81092d0 0x7f9fe819d290 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f9fd8011bf0 tx=0x7f9fd8011cd0 comp rx=0 tx=0).stop 2024-05-03T21:11:35.575 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:34.735+0000 7f9fef367700 1 -- 172.21.15.67:0/3911617192 shutdown_connections 2024-05-03T21:11:35.575 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:34.735+0000 7f9fef367700 1 --2- 172.21.15.67:0/3911617192 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f9fd4041f90 0x7f9fd4044380 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:11:35.575 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:34.735+0000 7f9fef367700 1 --2- 172.21.15.67:0/3911617192 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9fe81092d0 0x7f9fe819d290 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:11:35.575 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:34.735+0000 7f9fef367700 1 --2- 172.21.15.67:0/3911617192 >> [v2:172.21.15.67:6800/2576113038,v1:172.21.15.67:6801/2576113038] conn(0x7f9fd403e4c0 0x7f9fd4040980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:11:35.575 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:34.735+0000 7f9fef367700 1 -- 172.21.15.67:0/3911617192 >> 172.21.15.67:0/3911617192 conn(0x7f9fe8104970 msgr2=0x7f9fe8105480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T21:11:35.575 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:34.735+0000 7f9fef367700 1 -- 172.21.15.67:0/3911617192 shutdown_connections 2024-05-03T21:11:35.575 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:34.735+0000 7f9fef367700 1 -- 172.21.15.67:0/3911617192 wait complete. 2024-05-03T21:11:35.575 INFO:teuthology.orchestra.run.smithi067.stdout:mgr epoch 9 is available 2024-05-03T21:11:35.575 INFO:teuthology.orchestra.run.smithi067.stdout:Generating a dashboard self-signed certificate... 2024-05-03T21:11:35.814 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:11:35 smithi067 ceph-mon[29730]: [03/May/2024:21:11:34] ENGINE Serving on http://172.21.15.67:8765 2024-05-03T21:11:35.814 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:11:35 smithi067 ceph-mon[29730]: [03/May/2024:21:11:34] ENGINE Serving on https://172.21.15.67:7150 2024-05-03T21:11:35.814 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:11:35 smithi067 ceph-mon[29730]: [03/May/2024:21:11:34] ENGINE Bus STARTED 2024-05-03T21:11:35.814 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:11:35 smithi067 ceph-mon[29730]: from='client.14154 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-05-03T21:11:35.814 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:11:35 smithi067 ceph-mon[29730]: from='client.14154 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-05-03T21:11:35.814 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:11:35 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:11:36.957 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout Self-signed certificate created 2024-05-03T21:11:36.957 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:36.014+0000 7f4b1b529700 1 Processor -- start 2024-05-03T21:11:36.957 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:36.015+0000 7f4b1b529700 1 -- start start 2024-05-03T21:11:36.957 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:36.015+0000 7f4b1b529700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4b14108120 0x7f4b14108540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:11:36.957 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:36.015+0000 7f4b1b529700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f4b14108b10 con 0x7f4b14108120 2024-05-03T21:11:36.957 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:36.015+0000 7f4b192c5700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4b14108120 0x7f4b14108540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:11:36.957 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:36.015+0000 7f4b192c5700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4b14108120 0x7f4b14108540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:55290/0 (socket says 172.21.15.67:55290) 2024-05-03T21:11:36.957 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:36.015+0000 7f4b192c5700 1 -- 172.21.15.67:0/1601374105 learned_addr learned my addr 172.21.15.67:0/1601374105 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-03T21:11:36.957 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:36.015+0000 7f4b192c5700 1 -- 172.21.15.67:0/1601374105 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4b14109320 con 0x7f4b14108120 2024-05-03T21:11:36.957 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:36.016+0000 7f4b192c5700 1 --2- 172.21.15.67:0/1601374105 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4b14108120 0x7f4b14108540 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f4b040098a0 tx=0x7f4b04009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fd9a62114ed323d4 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T21:11:36.957 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:36.016+0000 7f4b0bfff700 1 -- 172.21.15.67:0/1601374105 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4b04004160 con 0x7f4b14108120 2024-05-03T21:11:36.957 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:36.016+0000 7f4b0bfff700 1 -- 172.21.15.67:0/1601374105 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(28 keys) ==== 1170+0+0 (secure 0 0 0) 0x7f4b04004300 con 0x7f4b14108120 2024-05-03T21:11:36.957 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:36.016+0000 7f4b0bfff700 1 -- 172.21.15.67:0/1601374105 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4b04011c20 con 0x7f4b14108120 2024-05-03T21:11:36.958 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:36.016+0000 7f4b1b529700 1 -- 172.21.15.67:0/1601374105 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4b14108120 msgr2=0x7f4b14108540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:11:36.958 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:36.016+0000 7f4b1b529700 1 --2- 172.21.15.67:0/1601374105 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4b14108120 0x7f4b14108540 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f4b040098a0 tx=0x7f4b04009bb0 comp rx=0 tx=0).stop 2024-05-03T21:11:36.958 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:36.017+0000 7f4b1b529700 1 -- 172.21.15.67:0/1601374105 shutdown_connections 2024-05-03T21:11:36.958 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:36.017+0000 7f4b1b529700 1 --2- 172.21.15.67:0/1601374105 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4b14108120 0x7f4b14108540 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:11:36.958 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:36.017+0000 7f4b1b529700 1 -- 172.21.15.67:0/1601374105 >> 172.21.15.67:0/1601374105 conn(0x7f4b14103780 msgr2=0x7f4b14105ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T21:11:36.958 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:36.017+0000 7f4b1b529700 1 -- 172.21.15.67:0/1601374105 shutdown_connections 2024-05-03T21:11:36.959 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:36.017+0000 7f4b1b529700 1 -- 172.21.15.67:0/1601374105 wait complete. 2024-05-03T21:11:36.959 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:36.017+0000 7f4b1b529700 1 Processor -- start 2024-05-03T21:11:36.959 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:36.017+0000 7f4b1b529700 1 -- start start 2024-05-03T21:11:36.959 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:36.018+0000 7f4b1b529700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4b14108120 0x7f4b14193360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:11:36.959 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:36.018+0000 7f4b1b529700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f4b1410c6d0 con 0x7f4b14108120 2024-05-03T21:11:36.959 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:36.018+0000 7f4b192c5700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4b14108120 0x7f4b14193360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:11:36.959 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:36.018+0000 7f4b192c5700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4b14108120 0x7f4b14193360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:55292/0 (socket says 172.21.15.67:55292) 2024-05-03T21:11:36.959 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:36.018+0000 7f4b192c5700 1 -- 172.21.15.67:0/2211212452 learned_addr learned my addr 172.21.15.67:0/2211212452 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-03T21:11:36.959 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:36.018+0000 7f4b192c5700 1 -- 172.21.15.67:0/2211212452 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4b141938a0 con 0x7f4b14108120 2024-05-03T21:11:36.959 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:36.018+0000 7f4b192c5700 1 --2- 172.21.15.67:0/2211212452 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4b14108120 0x7f4b14193360 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f4b04011670 tx=0x7f4b04011de0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T21:11:36.959 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:36.018+0000 7f4b0a7fc700 1 -- 172.21.15.67:0/2211212452 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4b040039e0 con 0x7f4b14108120 2024-05-03T21:11:36.959 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:36.018+0000 7f4b0a7fc700 1 -- 172.21.15.67:0/2211212452 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(28 keys) ==== 1170+0+0 (secure 0 0 0) 0x7f4b04004080 con 0x7f4b14108120 2024-05-03T21:11:36.959 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:36.018+0000 7f4b1b529700 1 -- 172.21.15.67:0/2211212452 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4b14191a90 con 0x7f4b14108120 2024-05-03T21:11:36.960 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:36.018+0000 7f4b0a7fc700 1 -- 172.21.15.67:0/2211212452 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4b040193c0 con 0x7f4b14108120 2024-05-03T21:11:36.960 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:36.019+0000 7f4b1b529700 1 -- 172.21.15.67:0/2211212452 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4b14191fd0 con 0x7f4b14108120 2024-05-03T21:11:36.960 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:36.020+0000 7f4b0a7fc700 1 -- 172.21.15.67:0/2211212452 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 11) ==== 50129+0+0 (secure 0 0 0) 0x7f4b04019680 con 0x7f4b14108120 2024-05-03T21:11:36.960 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:36.020+0000 7f4b1b529700 1 -- 172.21.15.67:0/2211212452 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4b1410c850 con 0x7f4b14108120 2024-05-03T21:11:36.960 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:36.020+0000 7f4b0a7fc700 1 --2- 172.21.15.67:0/2211212452 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f4b0003e340 0x7f4b00040800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:11:36.960 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:36.020+0000 7f4b0a7fc700 1 -- 172.21.15.67:0/2211212452 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f4b04053670 con 0x7f4b14108120 2024-05-03T21:11:36.960 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:36.020+0000 7f4b18ac4700 1 --2- 172.21.15.67:0/2211212452 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f4b0003e340 0x7f4b00040800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:11:36.960 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:36.020+0000 7f4b18ac4700 1 --2- 172.21.15.67:0/2211212452 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f4b0003e340 0x7f4b00040800 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f4b10006fd0 tx=0x7f4b10006e40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T21:11:36.960 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:36.025+0000 7f4b0a7fc700 1 -- 172.21.15.67:0/2211212452 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191839 (secure 0 0 0) 0x7f4b04017e90 con 0x7f4b14108120 2024-05-03T21:11:36.960 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:36.253+0000 7f4b1b529700 1 -- 172.21.15.67:0/2211212452 --> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] -- mgr_command(tid 0: {"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}) -- 0x7f4b14197190 con 0x7f4b0003e340 2024-05-03T21:11:36.960 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:36.364+0000 7f4b0a7fc700 1 -- 172.21.15.67:0/2211212452 <== mgr.14150 v2:172.21.15.67:6800/604451592 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+32 (secure 0 0 0) 0x7f4b14197190 con 0x7f4b0003e340 2024-05-03T21:11:36.960 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:36.366+0000 7f4b1b529700 1 -- 172.21.15.67:0/2211212452 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f4b0003e340 msgr2=0x7f4b00040800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:11:36.960 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:36.366+0000 7f4b1b529700 1 --2- 172.21.15.67:0/2211212452 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f4b0003e340 0x7f4b00040800 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f4b10006fd0 tx=0x7f4b10006e40 comp rx=0 tx=0).stop 2024-05-03T21:11:36.960 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:36.366+0000 7f4b1b529700 1 -- 172.21.15.67:0/2211212452 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4b14108120 msgr2=0x7f4b14193360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:11:36.961 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:36.366+0000 7f4b1b529700 1 --2- 172.21.15.67:0/2211212452 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4b14108120 0x7f4b14193360 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f4b04011670 tx=0x7f4b04011de0 comp rx=0 tx=0).stop 2024-05-03T21:11:36.961 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:36.366+0000 7f4b1b529700 1 -- 172.21.15.67:0/2211212452 shutdown_connections 2024-05-03T21:11:36.961 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:36.366+0000 7f4b1b529700 1 --2- 172.21.15.67:0/2211212452 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f4b0003e340 0x7f4b00040800 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:11:36.961 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:36.366+0000 7f4b1b529700 1 --2- 172.21.15.67:0/2211212452 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4b14108120 0x7f4b14193360 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:11:36.961 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:36.366+0000 7f4b1b529700 1 -- 172.21.15.67:0/2211212452 >> 172.21.15.67:0/2211212452 conn(0x7f4b14103780 msgr2=0x7f4b14104690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T21:11:36.961 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:36.366+0000 7f4b1b529700 1 -- 172.21.15.67:0/2211212452 shutdown_connections 2024-05-03T21:11:36.961 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:36.366+0000 7f4b1b529700 1 -- 172.21.15.67:0/2211212452 wait complete. 2024-05-03T21:11:36.961 INFO:teuthology.orchestra.run.smithi067.stdout:Creating initial admin user... 2024-05-03T21:11:37.050 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:11:36 smithi067 ceph-mon[29730]: from='client.14162 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2024-05-03T21:11:37.050 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:11:36 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:11:37.050 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:11:36 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:11:37.050 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:11:36 smithi067 ceph-mon[29730]: mgrmap e12: x(active, since 2s) 2024-05-03T21:11:38.483 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:11:38 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:11:38.483 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:11:38 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:11:38.491 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout {"username": "admin", "password": "$2b$12$9ZW64FPTBEQhNtUGrkG1rusGv4/moduLMsnoidt65.1iosEBwwYg6", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1714770697, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2024-05-03T21:11:38.491 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:37.383+0000 7f85706d7700 1 Processor -- start 2024-05-03T21:11:38.491 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:37.384+0000 7f85706d7700 1 -- start start 2024-05-03T21:11:38.491 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:37.384+0000 7f85706d7700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8568109d40 0x7f856810c130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:11:38.492 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:37.384+0000 7f85706d7700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f8568076f30 con 0x7f8568109d40 2024-05-03T21:11:38.492 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:37.384+0000 7f856e473700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8568109d40 0x7f856810c130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:11:38.492 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:37.384+0000 7f856e473700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8568109d40 0x7f856810c130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:55298/0 (socket says 172.21.15.67:55298) 2024-05-03T21:11:38.492 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:37.384+0000 7f856e473700 1 -- 172.21.15.67:0/3935431821 learned_addr learned my addr 172.21.15.67:0/3935431821 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-03T21:11:38.492 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:37.384+0000 7f856e473700 1 -- 172.21.15.67:0/3935431821 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f856810c670 con 0x7f8568109d40 2024-05-03T21:11:38.492 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:37.385+0000 7f856e473700 1 --2- 172.21.15.67:0/3935431821 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8568109d40 0x7f856810c130 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f85640098a0 tx=0x7f8564009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c06bc4b544d2d12f server_cookie=0 in_seq=0 out_seq=0 2024-05-03T21:11:38.492 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:37.385+0000 7f856d471700 1 -- 172.21.15.67:0/3935431821 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8564004160 con 0x7f8568109d40 2024-05-03T21:11:38.492 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:37.385+0000 7f856d471700 1 -- 172.21.15.67:0/3935431821 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(28 keys) ==== 1170+0+0 (secure 0 0 0) 0x7f8564004300 con 0x7f8568109d40 2024-05-03T21:11:38.492 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:37.385+0000 7f856d471700 1 -- 172.21.15.67:0/3935431821 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8564011c20 con 0x7f8568109d40 2024-05-03T21:11:38.492 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:37.386+0000 7f85706d7700 1 -- 172.21.15.67:0/3935431821 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8568109d40 msgr2=0x7f856810c130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:11:38.492 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:37.386+0000 7f85706d7700 1 --2- 172.21.15.67:0/3935431821 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8568109d40 0x7f856810c130 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f85640098a0 tx=0x7f8564009bb0 comp rx=0 tx=0).stop 2024-05-03T21:11:38.492 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:37.386+0000 7f85706d7700 1 -- 172.21.15.67:0/3935431821 shutdown_connections 2024-05-03T21:11:38.492 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:37.386+0000 7f85706d7700 1 --2- 172.21.15.67:0/3935431821 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8568109d40 0x7f856810c130 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:11:38.492 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:37.386+0000 7f85706d7700 1 -- 172.21.15.67:0/3935431821 >> 172.21.15.67:0/3935431821 conn(0x7f85681037b0 msgr2=0x7f8568105bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T21:11:38.493 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:37.386+0000 7f85706d7700 1 -- 172.21.15.67:0/3935431821 shutdown_connections 2024-05-03T21:11:38.493 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:37.386+0000 7f85706d7700 1 -- 172.21.15.67:0/3935431821 wait complete. 2024-05-03T21:11:38.493 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:37.386+0000 7f85706d7700 1 Processor -- start 2024-05-03T21:11:38.493 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:37.386+0000 7f85706d7700 1 -- start start 2024-05-03T21:11:38.493 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:37.387+0000 7f85706d7700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8568109d40 0x7f85681095c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:11:38.493 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:37.387+0000 7f85706d7700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f856810d680 con 0x7f8568109d40 2024-05-03T21:11:38.493 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:37.387+0000 7f856e473700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8568109d40 0x7f85681095c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:11:38.493 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:37.387+0000 7f856e473700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8568109d40 0x7f85681095c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:55314/0 (socket says 172.21.15.67:55314) 2024-05-03T21:11:38.493 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:37.387+0000 7f856e473700 1 -- 172.21.15.67:0/3852701799 learned_addr learned my addr 172.21.15.67:0/3852701799 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-03T21:11:38.493 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:37.387+0000 7f856e473700 1 -- 172.21.15.67:0/3852701799 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8568109b00 con 0x7f8568109d40 2024-05-03T21:11:38.493 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:37.387+0000 7f856e473700 1 --2- 172.21.15.67:0/3852701799 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8568109d40 0x7f85681095c0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f8564011670 tx=0x7f8564011a70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T21:11:38.493 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:37.387+0000 7f855f7fe700 1 -- 172.21.15.67:0/3852701799 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f85640036a0 con 0x7f8568109d40 2024-05-03T21:11:38.493 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:37.387+0000 7f855f7fe700 1 -- 172.21.15.67:0/3852701799 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(28 keys) ==== 1170+0+0 (secure 0 0 0) 0x7f8564003d40 con 0x7f8568109d40 2024-05-03T21:11:38.493 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:37.387+0000 7f85706d7700 1 -- 172.21.15.67:0/3852701799 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8568107cf0 con 0x7f8568109d40 2024-05-03T21:11:38.493 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:37.387+0000 7f855f7fe700 1 -- 172.21.15.67:0/3852701799 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f85640193c0 con 0x7f8568109d40 2024-05-03T21:11:38.494 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:37.387+0000 7f85706d7700 1 -- 172.21.15.67:0/3852701799 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8568108230 con 0x7f8568109d40 2024-05-03T21:11:38.494 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:37.388+0000 7f855f7fe700 1 -- 172.21.15.67:0/3852701799 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 12) ==== 50235+0+0 (secure 0 0 0) 0x7f85640196c0 con 0x7f8568109d40 2024-05-03T21:11:38.494 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:37.389+0000 7f855f7fe700 1 --2- 172.21.15.67:0/3852701799 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f855403e470 0x7f8554040930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:11:38.494 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:37.389+0000 7f855f7fe700 1 -- 172.21.15.67:0/3852701799 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f8564053880 con 0x7f8568109d40 2024-05-03T21:11:38.494 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:37.389+0000 7f856dc72700 1 --2- 172.21.15.67:0/3852701799 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f855403e470 0x7f8554040930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:11:38.494 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:37.389+0000 7f85706d7700 1 -- 172.21.15.67:0/3852701799 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8568064b00 con 0x7f8568109d40 2024-05-03T21:11:38.494 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:37.389+0000 7f856dc72700 1 --2- 172.21.15.67:0/3852701799 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f855403e470 0x7f8554040930 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f8558006fd0 tx=0x7f8558006e40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T21:11:38.494 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:37.394+0000 7f855f7fe700 1 -- 172.21.15.67:0/3852701799 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191839 (secure 0 0 0) 0x7f8564017e90 con 0x7f8568109d40 2024-05-03T21:11:38.494 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:37.629+0000 7f85706d7700 1 -- 172.21.15.67:0/3852701799 --> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] -- mgr_command(tid 0: {"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}) -- 0x7f85681088b0 con 0x7f855403e470 2024-05-03T21:11:38.494 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:37.877+0000 7f855f7fe700 1 -- 172.21.15.67:0/3852701799 <== mgr.14150 v2:172.21.15.67:6800/604451592 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+252 (secure 0 0 0) 0x7f85681088b0 con 0x7f855403e470 2024-05-03T21:11:38.494 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:37.879+0000 7f85706d7700 1 -- 172.21.15.67:0/3852701799 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f855403e470 msgr2=0x7f8554040930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:11:38.494 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:37.879+0000 7f85706d7700 1 --2- 172.21.15.67:0/3852701799 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f855403e470 0x7f8554040930 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f8558006fd0 tx=0x7f8558006e40 comp rx=0 tx=0).stop 2024-05-03T21:11:38.494 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:37.879+0000 7f85706d7700 1 -- 172.21.15.67:0/3852701799 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8568109d40 msgr2=0x7f85681095c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:11:38.494 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:37.879+0000 7f85706d7700 1 --2- 172.21.15.67:0/3852701799 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8568109d40 0x7f85681095c0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f8564011670 tx=0x7f8564011a70 comp rx=0 tx=0).stop 2024-05-03T21:11:38.495 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:37.879+0000 7f85706d7700 1 -- 172.21.15.67:0/3852701799 shutdown_connections 2024-05-03T21:11:38.495 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:37.879+0000 7f85706d7700 1 --2- 172.21.15.67:0/3852701799 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f855403e470 0x7f8554040930 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:11:38.495 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:37.879+0000 7f85706d7700 1 --2- 172.21.15.67:0/3852701799 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8568109d40 0x7f85681095c0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:11:38.495 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:37.879+0000 7f85706d7700 1 -- 172.21.15.67:0/3852701799 >> 172.21.15.67:0/3852701799 conn(0x7f85681037b0 msgr2=0x7f85681042a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T21:11:38.495 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:37.879+0000 7f85706d7700 1 -- 172.21.15.67:0/3852701799 shutdown_connections 2024-05-03T21:11:38.495 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:37.879+0000 7f85706d7700 1 -- 172.21.15.67:0/3852701799 wait complete. 2024-05-03T21:11:38.495 INFO:teuthology.orchestra.run.smithi067.stdout:Fetching dashboard port number... 2024-05-03T21:11:39.471 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:11:39 smithi067 ceph-mon[29730]: from='client.14164 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2024-05-03T21:11:40.619 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:11:40 smithi067 ceph-mon[29730]: from='client.? 172.21.15.67:0/209602143' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2024-05-03T21:11:40.622 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout 8443 2024-05-03T21:11:40.628 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:39.791+0000 7f6c03f74700 1 Processor -- start 2024-05-03T21:11:40.628 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:39.791+0000 7f6c03f74700 1 -- start start 2024-05-03T21:11:40.628 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:39.791+0000 7f6c03f74700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6bfc109290 0x7f6bfc1096b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:11:40.628 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:39.791+0000 7f6c03f74700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f6bfc109bf0 con 0x7f6bfc109290 2024-05-03T21:11:40.628 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:39.791+0000 7f6c01d10700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6bfc109290 0x7f6bfc1096b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:11:40.628 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:39.791+0000 7f6c01d10700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6bfc109290 0x7f6bfc1096b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:55316/0 (socket says 172.21.15.67:55316) 2024-05-03T21:11:40.629 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:39.791+0000 7f6c01d10700 1 -- 172.21.15.67:0/3158489075 learned_addr learned my addr 172.21.15.67:0/3158489075 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-03T21:11:40.629 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:39.791+0000 7f6c01d10700 1 -- 172.21.15.67:0/3158489075 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6bfc109d70 con 0x7f6bfc109290 2024-05-03T21:11:40.629 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:39.791+0000 7f6c01d10700 1 --2- 172.21.15.67:0/3158489075 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6bfc109290 0x7f6bfc1096b0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f6bf00098a0 tx=0x7f6bf0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a926d95046717bd2 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T21:11:40.629 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:39.792+0000 7f6c00d0e700 1 -- 172.21.15.67:0/3158489075 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6bf0004160 con 0x7f6bfc109290 2024-05-03T21:11:40.629 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:39.792+0000 7f6c00d0e700 1 -- 172.21.15.67:0/3158489075 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(28 keys) ==== 1170+0+0 (secure 0 0 0) 0x7f6bf0004300 con 0x7f6bfc109290 2024-05-03T21:11:40.629 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:39.792+0000 7f6c00d0e700 1 -- 172.21.15.67:0/3158489075 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6bf0011c20 con 0x7f6bfc109290 2024-05-03T21:11:40.629 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:39.792+0000 7f6c03f74700 1 -- 172.21.15.67:0/3158489075 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6bfc109290 msgr2=0x7f6bfc1096b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:11:40.629 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:39.792+0000 7f6c03f74700 1 --2- 172.21.15.67:0/3158489075 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6bfc109290 0x7f6bfc1096b0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f6bf00098a0 tx=0x7f6bf0009bb0 comp rx=0 tx=0).stop 2024-05-03T21:11:40.629 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:39.792+0000 7f6c03f74700 1 -- 172.21.15.67:0/3158489075 shutdown_connections 2024-05-03T21:11:40.629 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:39.792+0000 7f6c03f74700 1 --2- 172.21.15.67:0/3158489075 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6bfc109290 0x7f6bfc1096b0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:11:40.629 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:39.792+0000 7f6c03f74700 1 -- 172.21.15.67:0/3158489075 >> 172.21.15.67:0/3158489075 conn(0x7f6bfc104950 msgr2=0x7f6bfc106d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T21:11:40.629 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:39.792+0000 7f6c03f74700 1 -- 172.21.15.67:0/3158489075 shutdown_connections 2024-05-03T21:11:40.629 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:39.792+0000 7f6c03f74700 1 -- 172.21.15.67:0/3158489075 wait complete. 2024-05-03T21:11:40.629 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:39.793+0000 7f6c03f74700 1 Processor -- start 2024-05-03T21:11:40.630 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:39.793+0000 7f6c03f74700 1 -- start start 2024-05-03T21:11:40.630 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:39.793+0000 7f6c03f74700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6bfc109290 0x7f6bfc19d2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:11:40.630 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:39.793+0000 7f6c03f74700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f6bfc10d700 con 0x7f6bfc109290 2024-05-03T21:11:40.630 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:39.793+0000 7f6c01d10700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6bfc109290 0x7f6bfc19d2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:11:40.630 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:39.793+0000 7f6c01d10700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6bfc109290 0x7f6bfc19d2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:55320/0 (socket says 172.21.15.67:55320) 2024-05-03T21:11:40.630 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:39.793+0000 7f6c01d10700 1 -- 172.21.15.67:0/209602143 learned_addr learned my addr 172.21.15.67:0/209602143 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-03T21:11:40.630 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:39.794+0000 7f6c01d10700 1 -- 172.21.15.67:0/209602143 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6bfc19d810 con 0x7f6bfc109290 2024-05-03T21:11:40.630 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:39.794+0000 7f6c01d10700 1 --2- 172.21.15.67:0/209602143 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6bfc109290 0x7f6bfc19d2d0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f6bf0009fb0 tx=0x7f6bf0011a70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T21:11:40.630 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:39.794+0000 7f6beeffd700 1 -- 172.21.15.67:0/209602143 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6bf00036a0 con 0x7f6bfc109290 2024-05-03T21:11:40.630 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:39.794+0000 7f6beeffd700 1 -- 172.21.15.67:0/209602143 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(28 keys) ==== 1170+0+0 (secure 0 0 0) 0x7f6bf0003d40 con 0x7f6bfc109290 2024-05-03T21:11:40.630 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:39.794+0000 7f6c03f74700 1 -- 172.21.15.67:0/209602143 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6bfc19daa0 con 0x7f6bfc109290 2024-05-03T21:11:40.630 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:39.794+0000 7f6beeffd700 1 -- 172.21.15.67:0/209602143 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6bf001a430 con 0x7f6bfc109290 2024-05-03T21:11:40.630 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:39.794+0000 7f6c03f74700 1 -- 172.21.15.67:0/209602143 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6bfc1a0790 con 0x7f6bfc109290 2024-05-03T21:11:40.630 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:39.795+0000 7f6beeffd700 1 -- 172.21.15.67:0/209602143 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 12) ==== 50235+0+0 (secure 0 0 0) 0x7f6bf001a6f0 con 0x7f6bfc109290 2024-05-03T21:11:40.631 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:39.795+0000 7f6c03f74700 1 -- 172.21.15.67:0/209602143 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6bfc10d880 con 0x7f6bfc109290 2024-05-03T21:11:40.631 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:39.795+0000 7f6beeffd700 1 --2- 172.21.15.67:0/209602143 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f6be803e470 0x7f6be8040930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:11:40.631 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:39.795+0000 7f6beeffd700 1 -- 172.21.15.67:0/209602143 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f6bf00538b0 con 0x7f6bfc109290 2024-05-03T21:11:40.631 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:39.795+0000 7f6c0150f700 1 --2- 172.21.15.67:0/209602143 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f6be803e470 0x7f6be8040930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:11:40.631 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:39.796+0000 7f6c0150f700 1 --2- 172.21.15.67:0/209602143 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f6be803e470 0x7f6be8040930 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f6bf8006fd0 tx=0x7f6bf8006e40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T21:11:40.631 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:39.801+0000 7f6beeffd700 1 -- 172.21.15.67:0/209602143 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191839 (secure 0 0 0) 0x7f6bf0018d00 con 0x7f6bfc109290 2024-05-03T21:11:40.631 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:40.013+0000 7f6c03f74700 1 -- 172.21.15.67:0/209602143 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"} v 0) -- 0x7f6bfc1a0960 con 0x7f6bfc109290 2024-05-03T21:11:40.631 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:40.013+0000 7f6beeffd700 1 -- 172.21.15.67:0/209602143 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]=0 v8) ==== 112+0+5 (secure 0 0 0) 0x7f6bf0019ca0 con 0x7f6bfc109290 2024-05-03T21:11:40.631 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:40.015+0000 7f6c03f74700 1 -- 172.21.15.67:0/209602143 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f6be803e470 msgr2=0x7f6be8040930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:11:40.631 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:40.015+0000 7f6c03f74700 1 --2- 172.21.15.67:0/209602143 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f6be803e470 0x7f6be8040930 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f6bf8006fd0 tx=0x7f6bf8006e40 comp rx=0 tx=0).stop 2024-05-03T21:11:40.631 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:40.015+0000 7f6c03f74700 1 -- 172.21.15.67:0/209602143 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6bfc109290 msgr2=0x7f6bfc19d2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:11:40.631 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:40.015+0000 7f6c03f74700 1 --2- 172.21.15.67:0/209602143 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6bfc109290 0x7f6bfc19d2d0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f6bf0009fb0 tx=0x7f6bf0011a70 comp rx=0 tx=0).stop 2024-05-03T21:11:40.631 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:40.016+0000 7f6c03f74700 1 -- 172.21.15.67:0/209602143 shutdown_connections 2024-05-03T21:11:40.631 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:40.016+0000 7f6c03f74700 1 --2- 172.21.15.67:0/209602143 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f6be803e470 0x7f6be8040930 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:11:40.632 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:40.016+0000 7f6c03f74700 1 --2- 172.21.15.67:0/209602143 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6bfc109290 0x7f6bfc19d2d0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:11:40.632 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:40.016+0000 7f6c03f74700 1 -- 172.21.15.67:0/209602143 >> 172.21.15.67:0/209602143 conn(0x7f6bfc104950 msgr2=0x7f6bfc105490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T21:11:40.632 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:40.016+0000 7f6c03f74700 1 -- 172.21.15.67:0/209602143 shutdown_connections 2024-05-03T21:11:40.632 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:40.016+0000 7f6c03f74700 1 -- 172.21.15.67:0/209602143 wait complete. 2024-05-03T21:11:40.636 INFO:teuthology.orchestra.run.smithi067.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-05-03T21:11:40.637 INFO:teuthology.orchestra.run.smithi067.stdout:systemctl: stdout disabled 2024-05-03T21:11:40.644 INFO:teuthology.orchestra.run.smithi067.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-05-03T21:11:40.644 INFO:teuthology.orchestra.run.smithi067.stdout:systemctl: stdout inactive 2024-05-03T21:11:40.644 INFO:teuthology.orchestra.run.smithi067.stdout:firewalld.service is not enabled 2024-05-03T21:11:40.645 INFO:teuthology.orchestra.run.smithi067.stdout:Not possible to open ports <[8443]>. firewalld.service is not available 2024-05-03T21:11:40.646 INFO:teuthology.orchestra.run.smithi067.stdout:Ceph Dashboard is now available at: 2024-05-03T21:11:40.646 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-05-03T21:11:40.646 INFO:teuthology.orchestra.run.smithi067.stdout: URL: https://smithi067:8443/ 2024-05-03T21:11:40.646 INFO:teuthology.orchestra.run.smithi067.stdout: User: admin 2024-05-03T21:11:40.646 INFO:teuthology.orchestra.run.smithi067.stdout: Password: 2reaiokv6o 2024-05-03T21:11:40.646 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-05-03T21:11:40.647 INFO:teuthology.orchestra.run.smithi067.stdout:Saving cluster configuration to /var/lib/ceph/713bc57a-0991-11ef-bc97-c7b262605968/config directory 2024-05-03T21:11:40.647 INFO:teuthology.orchestra.run.smithi067.stdout:Enabling autotune for osd_memory_target 2024-05-03T21:11:41.533 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:11:41 smithi067 ceph-mon[29730]: mgrmap e13: x(active, since 6s) 2024-05-03T21:11:41.859 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:41.055+0000 7f2108aa9700 1 Processor -- start 2024-05-03T21:11:41.859 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:41.055+0000 7f2108aa9700 1 -- start start 2024-05-03T21:11:41.859 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:41.055+0000 7f2108aa9700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2104109270 0x7f2104109690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:11:41.859 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:41.055+0000 7f2108aa9700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f2104109bd0 con 0x7f2104109270 2024-05-03T21:11:41.859 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:41.055+0000 7f210259c700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2104109270 0x7f2104109690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:11:41.859 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:41.055+0000 7f210259c700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2104109270 0x7f2104109690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:55332/0 (socket says 172.21.15.67:55332) 2024-05-03T21:11:41.859 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:41.055+0000 7f210259c700 1 -- 172.21.15.67:0/565086534 learned_addr learned my addr 172.21.15.67:0/565086534 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-03T21:11:41.859 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:41.056+0000 7f210259c700 1 -- 172.21.15.67:0/565086534 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2104109d50 con 0x7f2104109270 2024-05-03T21:11:41.859 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:41.056+0000 7f210259c700 1 --2- 172.21.15.67:0/565086534 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2104109270 0x7f2104109690 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f20f40098a0 tx=0x7f20f4009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=47e5c0beda0c82e9 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T21:11:41.859 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:41.056+0000 7f210159a700 1 -- 172.21.15.67:0/565086534 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f20f4004160 con 0x7f2104109270 2024-05-03T21:11:41.859 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:41.056+0000 7f210159a700 1 -- 172.21.15.67:0/565086534 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(28 keys) ==== 1170+0+0 (secure 0 0 0) 0x7f20f4004300 con 0x7f2104109270 2024-05-03T21:11:41.859 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:41.056+0000 7f210159a700 1 -- 172.21.15.67:0/565086534 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f20f4011c20 con 0x7f2104109270 2024-05-03T21:11:41.860 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:41.056+0000 7f2108aa9700 1 -- 172.21.15.67:0/565086534 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2104109270 msgr2=0x7f2104109690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:11:41.860 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:41.056+0000 7f2108aa9700 1 --2- 172.21.15.67:0/565086534 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2104109270 0x7f2104109690 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f20f40098a0 tx=0x7f20f4009bb0 comp rx=0 tx=0).stop 2024-05-03T21:11:41.860 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:41.057+0000 7f2108aa9700 1 -- 172.21.15.67:0/565086534 shutdown_connections 2024-05-03T21:11:41.860 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:41.057+0000 7f2108aa9700 1 --2- 172.21.15.67:0/565086534 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2104109270 0x7f2104109690 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:11:41.860 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:41.057+0000 7f2108aa9700 1 -- 172.21.15.67:0/565086534 >> 172.21.15.67:0/565086534 conn(0x7f2104104950 msgr2=0x7f2104106d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T21:11:41.860 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:41.057+0000 7f2108aa9700 1 -- 172.21.15.67:0/565086534 shutdown_connections 2024-05-03T21:11:41.860 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:41.057+0000 7f2108aa9700 1 -- 172.21.15.67:0/565086534 wait complete. 2024-05-03T21:11:41.860 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:41.057+0000 7f2108aa9700 1 Processor -- start 2024-05-03T21:11:41.860 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:41.058+0000 7f2108aa9700 1 -- start start 2024-05-03T21:11:41.860 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:41.058+0000 7f2108aa9700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2104109270 0x7f210407eb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:11:41.860 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:41.058+0000 7f2108aa9700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f210410d6e0 con 0x7f2104109270 2024-05-03T21:11:41.860 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:41.058+0000 7f210259c700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2104109270 0x7f210407eb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:11:41.860 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:41.058+0000 7f210259c700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2104109270 0x7f210407eb90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:55338/0 (socket says 172.21.15.67:55338) 2024-05-03T21:11:41.860 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:41.058+0000 7f210259c700 1 -- 172.21.15.67:0/2270991170 learned_addr learned my addr 172.21.15.67:0/2270991170 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-03T21:11:41.861 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:41.058+0000 7f210259c700 1 -- 172.21.15.67:0/2270991170 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f210407f0d0 con 0x7f2104109270 2024-05-03T21:11:41.861 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:41.058+0000 7f210259c700 1 --2- 172.21.15.67:0/2270991170 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2104109270 0x7f210407eb90 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f20f4011670 tx=0x7f20f4011a70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T21:11:41.861 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:41.058+0000 7f20f37fe700 1 -- 172.21.15.67:0/2270991170 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f20f40036a0 con 0x7f2104109270 2024-05-03T21:11:41.861 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:41.058+0000 7f20f37fe700 1 -- 172.21.15.67:0/2270991170 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(28 keys) ==== 1170+0+0 (secure 0 0 0) 0x7f20f4003d40 con 0x7f2104109270 2024-05-03T21:11:41.861 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:41.058+0000 7f2108aa9700 1 -- 172.21.15.67:0/2270991170 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f210407d2c0 con 0x7f2104109270 2024-05-03T21:11:41.861 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:41.058+0000 7f20f37fe700 1 -- 172.21.15.67:0/2270991170 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f20f40193c0 con 0x7f2104109270 2024-05-03T21:11:41.861 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:41.059+0000 7f2108aa9700 1 -- 172.21.15.67:0/2270991170 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f210407d800 con 0x7f2104109270 2024-05-03T21:11:41.861 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:41.059+0000 7f20f37fe700 1 -- 172.21.15.67:0/2270991170 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 13) ==== 50278+0+0 (secure 0 0 0) 0x7f20f4019670 con 0x7f2104109270 2024-05-03T21:11:41.861 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:41.059+0000 7f2108aa9700 1 -- 172.21.15.67:0/2270991170 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2104064b00 con 0x7f2104109270 2024-05-03T21:11:41.861 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:41.060+0000 7f20f37fe700 1 --2- 172.21.15.67:0/2270991170 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f20ec03e570 0x7f20ec040a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:11:41.861 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:41.060+0000 7f20f37fe700 1 -- 172.21.15.67:0/2270991170 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f20f40539b0 con 0x7f2104109270 2024-05-03T21:11:41.861 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:41.060+0000 7f2101d9b700 1 --2- 172.21.15.67:0/2270991170 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f20ec03e570 0x7f20ec040a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:11:41.861 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:41.060+0000 7f2101d9b700 1 --2- 172.21.15.67:0/2270991170 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f20ec03e570 0x7f20ec040a30 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f20f8006fd0 tx=0x7f20f8006e40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T21:11:41.862 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:41.065+0000 7f20f37fe700 1 -- 172.21.15.67:0/2270991170 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191839 (secure 0 0 0) 0x7f20f4023c90 con 0x7f2104109270 2024-05-03T21:11:41.862 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:41.278+0000 7f2108aa9700 1 -- 172.21.15.67:0/2270991170 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command([{prefix=config set, name=osd_memory_target_autotune}] v 0) -- 0x7f210407da80 con 0x7f2104109270 2024-05-03T21:11:41.862 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:41.279+0000 7f20f37fe700 1 -- 172.21.15.67:0/2270991170 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{prefix=config set, name=osd_memory_target_autotune}]=0 v8)=0 v8) ==== 127+0+0 (secure 0 0 0) 0x7f20f4018350 con 0x7f2104109270 2024-05-03T21:11:41.862 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:41.281+0000 7f2108aa9700 1 -- 172.21.15.67:0/2270991170 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f20ec03e570 msgr2=0x7f20ec040a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:11:41.862 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:41.281+0000 7f2108aa9700 1 --2- 172.21.15.67:0/2270991170 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f20ec03e570 0x7f20ec040a30 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f20f8006fd0 tx=0x7f20f8006e40 comp rx=0 tx=0).stop 2024-05-03T21:11:41.862 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:41.281+0000 7f2108aa9700 1 -- 172.21.15.67:0/2270991170 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2104109270 msgr2=0x7f210407eb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:11:41.862 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:41.281+0000 7f2108aa9700 1 --2- 172.21.15.67:0/2270991170 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2104109270 0x7f210407eb90 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f20f4011670 tx=0x7f20f4011a70 comp rx=0 tx=0).stop 2024-05-03T21:11:41.862 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:41.281+0000 7f2108aa9700 1 -- 172.21.15.67:0/2270991170 shutdown_connections 2024-05-03T21:11:41.862 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:41.281+0000 7f2108aa9700 1 --2- 172.21.15.67:0/2270991170 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f20ec03e570 0x7f20ec040a30 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:11:41.862 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:41.281+0000 7f2108aa9700 1 --2- 172.21.15.67:0/2270991170 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2104109270 0x7f210407eb90 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:11:41.862 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:41.281+0000 7f2108aa9700 1 -- 172.21.15.67:0/2270991170 >> 172.21.15.67:0/2270991170 conn(0x7f2104104950 msgr2=0x7f21041054b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T21:11:41.862 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:41.281+0000 7f2108aa9700 1 -- 172.21.15.67:0/2270991170 shutdown_connections 2024-05-03T21:11:41.862 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:41.281+0000 7f2108aa9700 1 -- 172.21.15.67:0/2270991170 wait complete. 2024-05-03T21:11:43.115 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:42.274+0000 7f7d7e8eb700 1 Processor -- start 2024-05-03T21:11:43.115 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:42.274+0000 7f7d7e8eb700 1 -- start start 2024-05-03T21:11:43.115 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:42.274+0000 7f7d7e8eb700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7d78107b60 0x7f7d78107f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:11:43.116 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:42.274+0000 7f7d7e8eb700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f7d7807ce50 con 0x7f7d78107b60 2024-05-03T21:11:43.116 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:42.274+0000 7f7d77fff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7d78107b60 0x7f7d78107f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:11:43.116 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:42.274+0000 7f7d77fff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7d78107b60 0x7f7d78107f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:55348/0 (socket says 172.21.15.67:55348) 2024-05-03T21:11:43.116 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:42.274+0000 7f7d77fff700 1 -- 172.21.15.67:0/288180611 learned_addr learned my addr 172.21.15.67:0/288180611 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-03T21:11:43.116 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:42.274+0000 7f7d77fff700 1 -- 172.21.15.67:0/288180611 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7d781084c0 con 0x7f7d78107b60 2024-05-03T21:11:43.116 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:42.275+0000 7f7d77fff700 1 --2- 172.21.15.67:0/288180611 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7d78107b60 0x7f7d78107f80 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f7d680098a0 tx=0x7f7d68009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f83a1665fa447c3d server_cookie=0 in_seq=0 out_seq=0 2024-05-03T21:11:43.116 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:42.275+0000 7f7d76ffd700 1 -- 172.21.15.67:0/288180611 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7d68004160 con 0x7f7d78107b60 2024-05-03T21:11:43.116 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:42.275+0000 7f7d76ffd700 1 -- 172.21.15.67:0/288180611 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(28 keys) ==== 1170+0+0 (secure 0 0 0) 0x7f7d68004300 con 0x7f7d78107b60 2024-05-03T21:11:43.116 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:42.275+0000 7f7d76ffd700 1 -- 172.21.15.67:0/288180611 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7d68011c20 con 0x7f7d78107b60 2024-05-03T21:11:43.116 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:42.275+0000 7f7d7e8eb700 1 -- 172.21.15.67:0/288180611 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7d78107b60 msgr2=0x7f7d78107f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:11:43.116 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:42.275+0000 7f7d7e8eb700 1 --2- 172.21.15.67:0/288180611 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7d78107b60 0x7f7d78107f80 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f7d680098a0 tx=0x7f7d68009bb0 comp rx=0 tx=0).stop 2024-05-03T21:11:43.116 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:42.275+0000 7f7d7e8eb700 1 -- 172.21.15.67:0/288180611 shutdown_connections 2024-05-03T21:11:43.116 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:42.275+0000 7f7d7e8eb700 1 --2- 172.21.15.67:0/288180611 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7d78107b60 0x7f7d78107f80 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:11:43.117 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:42.276+0000 7f7d7e8eb700 1 -- 172.21.15.67:0/288180611 >> 172.21.15.67:0/288180611 conn(0x7f7d78103760 msgr2=0x7f7d78105b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T21:11:43.117 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:42.276+0000 7f7d7e8eb700 1 -- 172.21.15.67:0/288180611 shutdown_connections 2024-05-03T21:11:43.117 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:42.276+0000 7f7d7e8eb700 1 -- 172.21.15.67:0/288180611 wait complete. 2024-05-03T21:11:43.117 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:42.276+0000 7f7d7e8eb700 1 Processor -- start 2024-05-03T21:11:43.117 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:42.276+0000 7f7d7e8eb700 1 -- start start 2024-05-03T21:11:43.117 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:42.276+0000 7f7d7e8eb700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7d78107b60 0x7f7d7819f4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:11:43.117 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:42.276+0000 7f7d7e8eb700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f7d781095f0 con 0x7f7d78107b60 2024-05-03T21:11:43.117 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:42.277+0000 7f7d77fff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7d78107b60 0x7f7d7819f4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:11:43.117 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:42.277+0000 7f7d77fff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7d78107b60 0x7f7d7819f4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:55360/0 (socket says 172.21.15.67:55360) 2024-05-03T21:11:43.117 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:42.277+0000 7f7d77fff700 1 -- 172.21.15.67:0/1389400288 learned_addr learned my addr 172.21.15.67:0/1389400288 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-03T21:11:43.117 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:42.277+0000 7f7d77fff700 1 -- 172.21.15.67:0/1389400288 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7d7819f9e0 con 0x7f7d78107b60 2024-05-03T21:11:43.117 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:42.277+0000 7f7d77fff700 1 --2- 172.21.15.67:0/1389400288 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7d78107b60 0x7f7d7819f4a0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f7d68011bf0 tx=0x7f7d68011cd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T21:11:43.117 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:42.277+0000 7f7d757fa700 1 -- 172.21.15.67:0/1389400288 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7d680038b0 con 0x7f7d78107b60 2024-05-03T21:11:43.117 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:42.277+0000 7f7d757fa700 1 -- 172.21.15.67:0/1389400288 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(28 keys) ==== 1170+0+0 (secure 0 0 0) 0x7f7d68003a50 con 0x7f7d78107b60 2024-05-03T21:11:43.117 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:42.277+0000 7f7d7e8eb700 1 -- 172.21.15.67:0/1389400288 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7d7819fc70 con 0x7f7d78107b60 2024-05-03T21:11:43.118 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:42.277+0000 7f7d757fa700 1 -- 172.21.15.67:0/1389400288 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7d680243f0 con 0x7f7d78107b60 2024-05-03T21:11:43.118 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:42.277+0000 7f7d7e8eb700 1 -- 172.21.15.67:0/1389400288 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7d781a2960 con 0x7f7d78107b60 2024-05-03T21:11:43.118 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:42.278+0000 7f7d757fa700 1 -- 172.21.15.67:0/1389400288 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 13) ==== 50278+0+0 (secure 0 0 0) 0x7f7d680246f0 con 0x7f7d78107b60 2024-05-03T21:11:43.118 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:42.278+0000 7f7d7e8eb700 1 -- 172.21.15.67:0/1389400288 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7d78064b00 con 0x7f7d78107b60 2024-05-03T21:11:43.118 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:42.278+0000 7f7d757fa700 1 --2- 172.21.15.67:0/1389400288 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f7d6003e560 0x7f7d60040a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:11:43.118 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:42.278+0000 7f7d757fa700 1 -- 172.21.15.67:0/1389400288 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f7d68053910 con 0x7f7d78107b60 2024-05-03T21:11:43.118 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:42.279+0000 7f7d777fe700 1 --2- 172.21.15.67:0/1389400288 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f7d6003e560 0x7f7d60040a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:11:43.118 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:42.279+0000 7f7d777fe700 1 --2- 172.21.15.67:0/1389400288 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f7d6003e560 0x7f7d60040a20 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f7d6c006fd0 tx=0x7f7d6c006e40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T21:11:43.118 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:42.283+0000 7f7d757fa700 1 -- 172.21.15.67:0/1389400288 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191839 (secure 0 0 0) 0x7f7d6800a0d0 con 0x7f7d78107b60 2024-05-03T21:11:43.118 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:42.616+0000 7f7d7e8eb700 1 -- 172.21.15.67:0/1389400288 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command([{prefix=config-key set, key=mgr/dashboard/cluster/status}] v 0) -- 0x7f7d7802e760 con 0x7f7d78107b60 2024-05-03T21:11:43.118 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:42.618+0000 7f7d757fa700 1 -- 172.21.15.67:0/1389400288 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{prefix=config-key set, key=mgr/dashboard/cluster/status}]=0 set mgr/dashboard/cluster/status v27)=0 set mgr/dashboard/cluster/status v27) ==== 153+0+0 (secure 0 0 0) 0x7f7d6802c350 con 0x7f7d78107b60 2024-05-03T21:11:43.118 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:42.620+0000 7f7d7e8eb700 1 -- 172.21.15.67:0/1389400288 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f7d6003e560 msgr2=0x7f7d60040a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:11:43.118 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:42.620+0000 7f7d7e8eb700 1 --2- 172.21.15.67:0/1389400288 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f7d6003e560 0x7f7d60040a20 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f7d6c006fd0 tx=0x7f7d6c006e40 comp rx=0 tx=0).stop 2024-05-03T21:11:43.119 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:42.620+0000 7f7d7e8eb700 1 -- 172.21.15.67:0/1389400288 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7d78107b60 msgr2=0x7f7d7819f4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:11:43.119 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:42.620+0000 7f7d7e8eb700 1 --2- 172.21.15.67:0/1389400288 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7d78107b60 0x7f7d7819f4a0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f7d68011bf0 tx=0x7f7d68011cd0 comp rx=0 tx=0).stop 2024-05-03T21:11:43.119 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:42.621+0000 7f7d7e8eb700 1 -- 172.21.15.67:0/1389400288 shutdown_connections 2024-05-03T21:11:43.119 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:42.621+0000 7f7d7e8eb700 1 --2- 172.21.15.67:0/1389400288 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f7d6003e560 0x7f7d60040a20 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:11:43.119 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:42.621+0000 7f7d7e8eb700 1 --2- 172.21.15.67:0/1389400288 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7d78107b60 0x7f7d7819f4a0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:11:43.119 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:42.621+0000 7f7d7e8eb700 1 -- 172.21.15.67:0/1389400288 >> 172.21.15.67:0/1389400288 conn(0x7f7d78103760 msgr2=0x7f7d78196110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T21:11:43.119 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:42.621+0000 7f7d7e8eb700 1 -- 172.21.15.67:0/1389400288 shutdown_connections 2024-05-03T21:11:43.119 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T21:11:42.621+0000 7f7d7e8eb700 1 -- 172.21.15.67:0/1389400288 wait complete. 2024-05-03T21:11:43.119 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr set mgr/dashboard/cluster/status 2024-05-03T21:11:43.119 INFO:teuthology.orchestra.run.smithi067.stdout:You can access the Ceph CLI as following in case of multi-cluster or non-default config: 2024-05-03T21:11:43.119 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-05-03T21:11:43.119 INFO:teuthology.orchestra.run.smithi067.stdout: sudo /home/ubuntu/cephtest/cephadm shell --fsid 713bc57a-0991-11ef-bc97-c7b262605968 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2024-05-03T21:11:43.119 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-05-03T21:11:43.119 INFO:teuthology.orchestra.run.smithi067.stdout:Or, if you are only running a single cluster on this host: 2024-05-03T21:11:43.119 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-05-03T21:11:43.120 INFO:teuthology.orchestra.run.smithi067.stdout: sudo /home/ubuntu/cephtest/cephadm shell 2024-05-03T21:11:43.120 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-05-03T21:11:43.120 INFO:teuthology.orchestra.run.smithi067.stdout:Please consider enabling telemetry to help improve Ceph: 2024-05-03T21:11:43.120 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-05-03T21:11:43.120 INFO:teuthology.orchestra.run.smithi067.stdout: ceph telemetry on 2024-05-03T21:11:43.120 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-05-03T21:11:43.120 INFO:teuthology.orchestra.run.smithi067.stdout:For more information see: 2024-05-03T21:11:43.120 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-05-03T21:11:43.120 INFO:teuthology.orchestra.run.smithi067.stdout: https://docs.ceph.com/en/latest/mgr/telemetry/ 2024-05-03T21:11:43.120 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-05-03T21:11:43.120 INFO:teuthology.orchestra.run.smithi067.stdout:Bootstrap complete. 2024-05-03T21:11:43.151 INFO:tasks.cephadm:Fetching config... 2024-05-03T21:11:43.152 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-05-03T21:11:43.152 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2024-05-03T21:11:43.168 INFO:tasks.cephadm:Fetching client.admin keyring... 2024-05-03T21:11:43.168 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-05-03T21:11:43.168 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2024-05-03T21:11:43.225 INFO:tasks.cephadm:Fetching mon keyring... 2024-05-03T21:11:43.225 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-05-03T21:11:43.226 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd if=/var/lib/ceph/713bc57a-0991-11ef-bc97-c7b262605968/mon.a/keyring of=/dev/stdout 2024-05-03T21:11:43.291 INFO:tasks.cephadm:Fetching pub ssh key... 2024-05-03T21:11:43.291 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-05-03T21:11:43.291 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2024-05-03T21:11:43.345 INFO:tasks.cephadm:Installing pub ssh key for root users... 2024-05-03T21:11:43.346 DEBUG:teuthology.orchestra.run.smithi067:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC/J7oMsbQ7wmnz8fHnpQAQlJo7xVjtRYwju3LjA335iqv29dC8Yq4lVSe5WbJdwxhrNYDPzkAbdX8JqGo43dfJKXeZ/ThoEzTVypm0JJmLN/xTJ6fckSUSmuMXVZGoyUlVZLczOc81cal2gw64UrsR8/gmF9l11lonPXKDwqkFFyN63BgLtXeImIEwZI9jBdHr7LECXjEbJTgwjeb/CDP3SIOARk/Vx4s+GIxbMLNxZd0e+7r1+uIlkBNk5FZb10B0RI+KVPOUp85Wgr/BX7gJAItsQCbH/Io04CtlytXCDpBr8iXJynjU69xUTcQN2kNYPhmCulb2VCV0gcB/cfufcoj4oqTBcP0846E/DdNrQ1cVKM1kedZdFK37T3thRUx1LlSzrdqCxX4NJklj9p5yhEs4N2SO3gmS3i/vdNHcIJ61LZuFsKqOmINIyRqq0wkoRWhdLxJRp5l/pwMR9rW7Mukcqw+fM3KDga0wLgvTSGdBw5EkxZzJQnCD0FGuJYc= ceph-713bc57a-0991-11ef-bc97-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-05-03T21:11:43.419 INFO:teuthology.orchestra.run.smithi067.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC/J7oMsbQ7wmnz8fHnpQAQlJo7xVjtRYwju3LjA335iqv29dC8Yq4lVSe5WbJdwxhrNYDPzkAbdX8JqGo43dfJKXeZ/ThoEzTVypm0JJmLN/xTJ6fckSUSmuMXVZGoyUlVZLczOc81cal2gw64UrsR8/gmF9l11lonPXKDwqkFFyN63BgLtXeImIEwZI9jBdHr7LECXjEbJTgwjeb/CDP3SIOARk/Vx4s+GIxbMLNxZd0e+7r1+uIlkBNk5FZb10B0RI+KVPOUp85Wgr/BX7gJAItsQCbH/Io04CtlytXCDpBr8iXJynjU69xUTcQN2kNYPhmCulb2VCV0gcB/cfufcoj4oqTBcP0846E/DdNrQ1cVKM1kedZdFK37T3thRUx1LlSzrdqCxX4NJklj9p5yhEs4N2SO3gmS3i/vdNHcIJ61LZuFsKqOmINIyRqq0wkoRWhdLxJRp5l/pwMR9rW7Mukcqw+fM3KDga0wLgvTSGdBw5EkxZzJQnCD0FGuJYc= ceph-713bc57a-0991-11ef-bc97-c7b262605968 2024-05-03T21:11:43.446 DEBUG:teuthology.orchestra.run.smithi082:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC/J7oMsbQ7wmnz8fHnpQAQlJo7xVjtRYwju3LjA335iqv29dC8Yq4lVSe5WbJdwxhrNYDPzkAbdX8JqGo43dfJKXeZ/ThoEzTVypm0JJmLN/xTJ6fckSUSmuMXVZGoyUlVZLczOc81cal2gw64UrsR8/gmF9l11lonPXKDwqkFFyN63BgLtXeImIEwZI9jBdHr7LECXjEbJTgwjeb/CDP3SIOARk/Vx4s+GIxbMLNxZd0e+7r1+uIlkBNk5FZb10B0RI+KVPOUp85Wgr/BX7gJAItsQCbH/Io04CtlytXCDpBr8iXJynjU69xUTcQN2kNYPhmCulb2VCV0gcB/cfufcoj4oqTBcP0846E/DdNrQ1cVKM1kedZdFK37T3thRUx1LlSzrdqCxX4NJklj9p5yhEs4N2SO3gmS3i/vdNHcIJ61LZuFsKqOmINIyRqq0wkoRWhdLxJRp5l/pwMR9rW7Mukcqw+fM3KDga0wLgvTSGdBw5EkxZzJQnCD0FGuJYc= ceph-713bc57a-0991-11ef-bc97-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-05-03T21:11:43.485 INFO:teuthology.orchestra.run.smithi082.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC/J7oMsbQ7wmnz8fHnpQAQlJo7xVjtRYwju3LjA335iqv29dC8Yq4lVSe5WbJdwxhrNYDPzkAbdX8JqGo43dfJKXeZ/ThoEzTVypm0JJmLN/xTJ6fckSUSmuMXVZGoyUlVZLczOc81cal2gw64UrsR8/gmF9l11lonPXKDwqkFFyN63BgLtXeImIEwZI9jBdHr7LECXjEbJTgwjeb/CDP3SIOARk/Vx4s+GIxbMLNxZd0e+7r1+uIlkBNk5FZb10B0RI+KVPOUp85Wgr/BX7gJAItsQCbH/Io04CtlytXCDpBr8iXJynjU69xUTcQN2kNYPhmCulb2VCV0gcB/cfufcoj4oqTBcP0846E/DdNrQ1cVKM1kedZdFK37T3thRUx1LlSzrdqCxX4NJklj9p5yhEs4N2SO3gmS3i/vdNHcIJ61LZuFsKqOmINIyRqq0wkoRWhdLxJRp5l/pwMR9rW7Mukcqw+fM3KDga0wLgvTSGdBw5EkxZzJQnCD0FGuJYc= ceph-713bc57a-0991-11ef-bc97-c7b262605968 2024-05-03T21:11:43.497 DEBUG:teuthology.orchestra.run.smithi138:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC/J7oMsbQ7wmnz8fHnpQAQlJo7xVjtRYwju3LjA335iqv29dC8Yq4lVSe5WbJdwxhrNYDPzkAbdX8JqGo43dfJKXeZ/ThoEzTVypm0JJmLN/xTJ6fckSUSmuMXVZGoyUlVZLczOc81cal2gw64UrsR8/gmF9l11lonPXKDwqkFFyN63BgLtXeImIEwZI9jBdHr7LECXjEbJTgwjeb/CDP3SIOARk/Vx4s+GIxbMLNxZd0e+7r1+uIlkBNk5FZb10B0RI+KVPOUp85Wgr/BX7gJAItsQCbH/Io04CtlytXCDpBr8iXJynjU69xUTcQN2kNYPhmCulb2VCV0gcB/cfufcoj4oqTBcP0846E/DdNrQ1cVKM1kedZdFK37T3thRUx1LlSzrdqCxX4NJklj9p5yhEs4N2SO3gmS3i/vdNHcIJ61LZuFsKqOmINIyRqq0wkoRWhdLxJRp5l/pwMR9rW7Mukcqw+fM3KDga0wLgvTSGdBw5EkxZzJQnCD0FGuJYc= ceph-713bc57a-0991-11ef-bc97-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-05-03T21:11:43.536 INFO:teuthology.orchestra.run.smithi138.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC/J7oMsbQ7wmnz8fHnpQAQlJo7xVjtRYwju3LjA335iqv29dC8Yq4lVSe5WbJdwxhrNYDPzkAbdX8JqGo43dfJKXeZ/ThoEzTVypm0JJmLN/xTJ6fckSUSmuMXVZGoyUlVZLczOc81cal2gw64UrsR8/gmF9l11lonPXKDwqkFFyN63BgLtXeImIEwZI9jBdHr7LECXjEbJTgwjeb/CDP3SIOARk/Vx4s+GIxbMLNxZd0e+7r1+uIlkBNk5FZb10B0RI+KVPOUp85Wgr/BX7gJAItsQCbH/Io04CtlytXCDpBr8iXJynjU69xUTcQN2kNYPhmCulb2VCV0gcB/cfufcoj4oqTBcP0846E/DdNrQ1cVKM1kedZdFK37T3thRUx1LlSzrdqCxX4NJklj9p5yhEs4N2SO3gmS3i/vdNHcIJ61LZuFsKqOmINIyRqq0wkoRWhdLxJRp5l/pwMR9rW7Mukcqw+fM3KDga0wLgvTSGdBw5EkxZzJQnCD0FGuJYc= ceph-713bc57a-0991-11ef-bc97-c7b262605968 2024-05-03T21:11:43.548 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9a047fe7599a549042d06a3fb5b6813a203eabea shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 713bc57a-0991-11ef-bc97-c7b262605968 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2024-05-03T21:11:43.794 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/713bc57a-0991-11ef-bc97-c7b262605968/mon.a/config 2024-05-03T21:11:43.917 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:11:43 smithi067 ceph-mon[29730]: from='client.? 172.21.15.67:0/1389400288' entity='client.admin' 2024-05-03T21:11:46.075 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:11:45.938+0000 7fbc9da0e700 1 -- 172.21.15.67:0/1328859408 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbc981039e0 msgr2=0x7fbc98105dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:11:46.075 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:11:45.938+0000 7fbc9da0e700 1 --2- 172.21.15.67:0/1328859408 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbc981039e0 0x7fbc98105dd0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fbc88009910 tx=0x7fbc88009c20 comp rx=0 tx=0).stop 2024-05-03T21:11:46.075 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:11:45.938+0000 7fbc9da0e700 1 -- 172.21.15.67:0/1328859408 shutdown_connections 2024-05-03T21:11:46.075 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:11:45.938+0000 7fbc9da0e700 1 --2- 172.21.15.67:0/1328859408 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbc981039e0 0x7fbc98105dd0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:11:46.075 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:11:45.938+0000 7fbc9da0e700 1 -- 172.21.15.67:0/1328859408 >> 172.21.15.67:0/1328859408 conn(0x7fbc980fd3b0 msgr2=0x7fbc980ff830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T21:11:46.075 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:11:45.938+0000 7fbc9da0e700 1 -- 172.21.15.67:0/1328859408 shutdown_connections 2024-05-03T21:11:46.075 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:11:45.938+0000 7fbc9da0e700 1 -- 172.21.15.67:0/1328859408 wait complete. 2024-05-03T21:11:46.075 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:11:45.938+0000 7fbc9da0e700 1 Processor -- start 2024-05-03T21:11:46.075 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:11:45.939+0000 7fbc9da0e700 1 -- start start 2024-05-03T21:11:46.076 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:11:45.939+0000 7fbc9da0e700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbc981039e0 0x7fbc98103260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:11:46.076 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:11:45.939+0000 7fbc9da0e700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fbc981090c0 con 0x7fbc981039e0 2024-05-03T21:11:46.076 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:11:45.939+0000 7fbc96ffd700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbc981039e0 0x7fbc98103260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:11:46.076 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:11:45.939+0000 7fbc96ffd700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbc981039e0 0x7fbc98103260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:41358/0 (socket says 172.21.15.67:41358) 2024-05-03T21:11:46.076 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:11:45.939+0000 7fbc96ffd700 1 -- 172.21.15.67:0/1336874012 learned_addr learned my addr 172.21.15.67:0/1336874012 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-03T21:11:46.076 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:11:45.939+0000 7fbc96ffd700 1 -- 172.21.15.67:0/1336874012 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbc981037a0 con 0x7fbc981039e0 2024-05-03T21:11:46.076 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:11:45.939+0000 7fbc96ffd700 1 --2- 172.21.15.67:0/1336874012 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbc981039e0 0x7fbc98103260 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fbc88004800 tx=0x7fbc880050b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T21:11:46.076 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:11:45.939+0000 7fbc9ca0c700 1 -- 172.21.15.67:0/1336874012 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fbc88027070 con 0x7fbc981039e0 2024-05-03T21:11:46.076 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:11:45.939+0000 7fbc9ca0c700 1 -- 172.21.15.67:0/1336874012 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(28 keys) ==== 1170+0+0 (secure 0 0 0) 0x7fbc88022470 con 0x7fbc981039e0 2024-05-03T21:11:46.076 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:11:45.939+0000 7fbc9da0e700 1 -- 172.21.15.67:0/1336874012 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbc98101990 con 0x7fbc981039e0 2024-05-03T21:11:46.076 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:11:45.940+0000 7fbc9ca0c700 1 -- 172.21.15.67:0/1336874012 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fbc8800f5f0 con 0x7fbc981039e0 2024-05-03T21:11:46.076 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:11:45.940+0000 7fbc9da0e700 1 -- 172.21.15.67:0/1336874012 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbc98101ed0 con 0x7fbc981039e0 2024-05-03T21:11:46.076 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:11:45.941+0000 7fbc9da0e700 1 -- 172.21.15.67:0/1336874012 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbc98109840 con 0x7fbc981039e0 2024-05-03T21:11:46.077 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:11:45.941+0000 7fbc9ca0c700 1 -- 172.21.15.67:0/1336874012 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 13) ==== 50278+0+0 (secure 0 0 0) 0x7fbc8802a070 con 0x7fbc981039e0 2024-05-03T21:11:46.077 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:11:45.941+0000 7fbc9ca0c700 1 --2- 172.21.15.67:0/1336874012 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7fbc8403e510 0x7fbc840409d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:11:46.077 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:11:45.941+0000 7fbc9ca0c700 1 -- 172.21.15.67:0/1336874012 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7fbc8801e030 con 0x7fbc981039e0 2024-05-03T21:11:46.077 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:11:45.941+0000 7fbc967fc700 1 --2- 172.21.15.67:0/1336874012 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7fbc8403e510 0x7fbc840409d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:11:46.077 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:11:45.941+0000 7fbc967fc700 1 --2- 172.21.15.67:0/1336874012 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7fbc8403e510 0x7fbc840409d0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fbc80006fd0 tx=0x7fbc80006e40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T21:11:46.077 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:11:45.946+0000 7fbc9ca0c700 1 -- 172.21.15.67:0/1336874012 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191839 (secure 0 0 0) 0x7fbc88013070 con 0x7fbc981039e0 2024-05-03T21:11:46.160 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:11:46.158+0000 7fbc9da0e700 1 -- 172.21.15.67:0/1336874012 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/allow_ptrace}] v 0) -- 0x7fbc98191d80 con 0x7fbc981039e0 2024-05-03T21:11:46.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:11:46.163+0000 7fbc9ca0c700 1 -- 172.21.15.67:0/1336874012 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/allow_ptrace}]=0 v9)=0 v9) ==== 125+0+0 (secure 0 0 0) 0x7fbc88031070 con 0x7fbc981039e0 2024-05-03T21:11:46.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:11:46.166+0000 7fbc9da0e700 1 -- 172.21.15.67:0/1336874012 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7fbc8403e510 msgr2=0x7fbc840409d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:11:46.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:11:46.166+0000 7fbc9da0e700 1 --2- 172.21.15.67:0/1336874012 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7fbc8403e510 0x7fbc840409d0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fbc80006fd0 tx=0x7fbc80006e40 comp rx=0 tx=0).stop 2024-05-03T21:11:46.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:11:46.166+0000 7fbc9da0e700 1 -- 172.21.15.67:0/1336874012 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbc981039e0 msgr2=0x7fbc98103260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:11:46.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:11:46.166+0000 7fbc9da0e700 1 --2- 172.21.15.67:0/1336874012 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbc981039e0 0x7fbc98103260 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fbc88004800 tx=0x7fbc880050b0 comp rx=0 tx=0).stop 2024-05-03T21:11:46.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:11:46.166+0000 7fbc9da0e700 1 -- 172.21.15.67:0/1336874012 shutdown_connections 2024-05-03T21:11:46.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:11:46.166+0000 7fbc9da0e700 1 --2- 172.21.15.67:0/1336874012 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7fbc8403e510 0x7fbc840409d0 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:11:46.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:11:46.166+0000 7fbc9da0e700 1 --2- 172.21.15.67:0/1336874012 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbc981039e0 0x7fbc98103260 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:11:46.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:11:46.166+0000 7fbc9da0e700 1 -- 172.21.15.67:0/1336874012 >> 172.21.15.67:0/1336874012 conn(0x7fbc980fd3b0 msgr2=0x7fbc980fdda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T21:11:46.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:11:46.166+0000 7fbc9da0e700 1 -- 172.21.15.67:0/1336874012 shutdown_connections 2024-05-03T21:11:46.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:11:46.166+0000 7fbc9da0e700 1 -- 172.21.15.67:0/1336874012 wait complete. 2024-05-03T21:11:46.220 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:11:46 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:11:46.221 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:11:46 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:11:46.221 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:11:46 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi067", "name": "osd_memory_target"}]: dispatch 2024-05-03T21:11:46.221 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:11:46 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:11:46.221 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:11:46 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T21:11:46.221 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:11:46 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:11:46.759 INFO:tasks.cephadm:Distributing conf and client.admin keyring to all hosts + 0755 2024-05-03T21:11:46.759 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9a047fe7599a549042d06a3fb5b6813a203eabea shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 713bc57a-0991-11ef-bc97-c7b262605968 -- ceph orch client-keyring set client.admin '*' --mode 0755 2024-05-03T21:11:47.003 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/713bc57a-0991-11ef-bc97-c7b262605968/mon.a/config 2024-05-03T21:11:47.414 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:11:47 smithi067 ceph-mon[29730]: from='client.? 172.21.15.67:0/1336874012' entity='client.admin' 2024-05-03T21:11:47.415 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:11:47 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T21:11:47.415 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:11:47 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:11:48.473 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:11:48.470+0000 7fb57c587700 1 -- 172.21.15.67:0/2037152510 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb574104570 msgr2=0x7fb574104990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:11:48.473 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:11:48.470+0000 7fb57c587700 1 --2- 172.21.15.67:0/2037152510 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb574104570 0x7fb574104990 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fb570009910 tx=0x7fb570009c20 comp rx=0 tx=0).stop 2024-05-03T21:11:48.473 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:11:48.470+0000 7fb57c587700 1 -- 172.21.15.67:0/2037152510 shutdown_connections 2024-05-03T21:11:48.473 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:11:48.470+0000 7fb57c587700 1 --2- 172.21.15.67:0/2037152510 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb574104570 0x7fb574104990 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:11:48.474 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:11:48.470+0000 7fb57c587700 1 -- 172.21.15.67:0/2037152510 >> 172.21.15.67:0/2037152510 conn(0x7fb5740fe4f0 msgr2=0x7fb574100910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T21:11:48.474 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:11:48.470+0000 7fb57c587700 1 -- 172.21.15.67:0/2037152510 shutdown_connections 2024-05-03T21:11:48.474 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:11:48.470+0000 7fb57c587700 1 -- 172.21.15.67:0/2037152510 wait complete. 2024-05-03T21:11:48.474 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:11:48.471+0000 7fb57c587700 1 Processor -- start 2024-05-03T21:11:48.474 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:11:48.471+0000 7fb57c587700 1 -- start start 2024-05-03T21:11:48.474 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:11:48.471+0000 7fb57c587700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb574104570 0x7fb574077160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:11:48.474 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:11:48.471+0000 7fb57c587700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fb574109180 con 0x7fb574104570 2024-05-03T21:11:48.474 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:11:48.471+0000 7fb57a323700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb574104570 0x7fb574077160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:11:48.474 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:11:48.471+0000 7fb57a323700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb574104570 0x7fb574077160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:41376/0 (socket says 172.21.15.67:41376) 2024-05-03T21:11:48.474 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:11:48.471+0000 7fb57a323700 1 -- 172.21.15.67:0/2363297475 learned_addr learned my addr 172.21.15.67:0/2363297475 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-03T21:11:48.474 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:11:48.471+0000 7fb57a323700 1 -- 172.21.15.67:0/2363297475 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb5740776a0 con 0x7fb574104570 2024-05-03T21:11:48.474 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:11:48.472+0000 7fb57a323700 1 --2- 172.21.15.67:0/2363297475 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb574104570 0x7fb574077160 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fb570004800 tx=0x7fb5700050b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T21:11:48.476 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:11:48.472+0000 7fb56b7fe700 1 -- 172.21.15.67:0/2363297475 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb570027070 con 0x7fb574104570 2024-05-03T21:11:48.476 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:11:48.472+0000 7fb56b7fe700 1 -- 172.21.15.67:0/2363297475 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(28 keys) ==== 1170+0+0 (secure 0 0 0) 0x7fb570022470 con 0x7fb574104570 2024-05-03T21:11:48.476 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:11:48.472+0000 7fb57c587700 1 -- 172.21.15.67:0/2363297475 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb574075890 con 0x7fb574104570 2024-05-03T21:11:48.476 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:11:48.472+0000 7fb56b7fe700 1 -- 172.21.15.67:0/2363297475 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb57000f5f0 con 0x7fb574104570 2024-05-03T21:11:48.476 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:11:48.472+0000 7fb57c587700 1 -- 172.21.15.67:0/2363297475 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb574075dd0 con 0x7fb574104570 2024-05-03T21:11:48.476 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:11:48.473+0000 7fb56b7fe700 1 -- 172.21.15.67:0/2363297475 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 13) ==== 50278+0+0 (secure 0 0 0) 0x7fb57002a070 con 0x7fb574104570 2024-05-03T21:11:48.476 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:11:48.473+0000 7fb57c587700 1 -- 172.21.15.67:0/2363297475 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb574109900 con 0x7fb574104570 2024-05-03T21:11:48.476 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:11:48.473+0000 7fb56b7fe700 1 --2- 172.21.15.67:0/2363297475 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7fb56003e570 0x7fb560040a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:11:48.476 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:11:48.473+0000 7fb56b7fe700 1 -- 172.21.15.67:0/2363297475 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7fb57001e030 con 0x7fb574104570 2024-05-03T21:11:48.476 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:11:48.473+0000 7fb579b22700 1 --2- 172.21.15.67:0/2363297475 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7fb56003e570 0x7fb560040a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:11:48.476 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:11:48.474+0000 7fb579b22700 1 --2- 172.21.15.67:0/2363297475 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7fb56003e570 0x7fb560040a30 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fb564006fd0 tx=0x7fb564006e40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T21:11:48.480 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:11:48.478+0000 7fb56b7fe700 1 -- 172.21.15.67:0/2363297475 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191839 (secure 0 0 0) 0x7fb570013070 con 0x7fb574104570 2024-05-03T21:11:48.697 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:11:48.695+0000 7fb57c587700 1 -- 172.21.15.67:0/2363297475 --> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] -- mgr_command(tid 0: {"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}) -- 0x7fb574191af0 con 0x7fb56003e570 2024-05-03T21:11:48.700 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:11:48.699+0000 7fb56b7fe700 1 -- 172.21.15.67:0/2363297475 <== mgr.14150 v2:172.21.15.67:6800/604451592 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7fb574191af0 con 0x7fb56003e570 2024-05-03T21:11:48.703 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:11:48.701+0000 7fb57c587700 1 -- 172.21.15.67:0/2363297475 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7fb56003e570 msgr2=0x7fb560040a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:11:48.703 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:11:48.701+0000 7fb57c587700 1 --2- 172.21.15.67:0/2363297475 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7fb56003e570 0x7fb560040a30 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fb564006fd0 tx=0x7fb564006e40 comp rx=0 tx=0).stop 2024-05-03T21:11:48.703 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:11:48.701+0000 7fb57c587700 1 -- 172.21.15.67:0/2363297475 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb574104570 msgr2=0x7fb574077160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:11:48.703 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:11:48.701+0000 7fb57c587700 1 --2- 172.21.15.67:0/2363297475 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb574104570 0x7fb574077160 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fb570004800 tx=0x7fb5700050b0 comp rx=0 tx=0).stop 2024-05-03T21:11:48.703 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:11:48.701+0000 7fb57c587700 1 -- 172.21.15.67:0/2363297475 shutdown_connections 2024-05-03T21:11:48.703 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:11:48.701+0000 7fb57c587700 1 --2- 172.21.15.67:0/2363297475 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7fb56003e570 0x7fb560040a30 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:11:48.703 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:11:48.701+0000 7fb57c587700 1 --2- 172.21.15.67:0/2363297475 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb574104570 0x7fb574077160 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:11:48.703 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:11:48.701+0000 7fb57c587700 1 -- 172.21.15.67:0/2363297475 >> 172.21.15.67:0/2363297475 conn(0x7fb5740fe4f0 msgr2=0x7fb5740fedf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T21:11:48.703 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:11:48.701+0000 7fb57c587700 1 -- 172.21.15.67:0/2363297475 shutdown_connections 2024-05-03T21:11:48.703 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:11:48.701+0000 7fb57c587700 1 -- 172.21.15.67:0/2363297475 wait complete. 2024-05-03T21:11:49.491 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi082 2024-05-03T21:11:49.491 DEBUG:teuthology.orchestra.run.smithi082:> set -ex 2024-05-03T21:11:49.491 DEBUG:teuthology.orchestra.run.smithi082:> dd of=/etc/ceph/ceph.conf 2024-05-03T21:11:49.511 DEBUG:teuthology.orchestra.run.smithi082:> set -ex 2024-05-03T21:11:49.511 DEBUG:teuthology.orchestra.run.smithi082:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-05-03T21:11:49.569 INFO:tasks.cephadm:Adding host smithi082 to orchestrator... 2024-05-03T21:11:49.569 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9a047fe7599a549042d06a3fb5b6813a203eabea shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 713bc57a-0991-11ef-bc97-c7b262605968 -- ceph orch host add smithi082 2024-05-03T21:11:49.721 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:11:49 smithi067 ceph-mon[29730]: from='client.14174 -' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2024-05-03T21:11:49.721 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:11:49 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:11:49.721 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:11:49 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T21:11:49.721 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:11:49 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T21:11:49.721 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:11:49 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T21:11:49.721 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:11:49 smithi067 ceph-mon[29730]: Updating smithi067:/etc/ceph/ceph.conf 2024-05-03T21:11:49.721 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:11:49 smithi067 ceph-mon[29730]: Updating smithi067:/var/lib/ceph/713bc57a-0991-11ef-bc97-c7b262605968/config/ceph.conf 2024-05-03T21:11:49.721 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:11:49 smithi067 ceph-mon[29730]: Updating smithi067:/etc/ceph/ceph.client.admin.keyring 2024-05-03T21:11:49.721 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:11:49 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:11:49.722 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:11:49 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:11:49.722 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:11:49 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:11:49.812 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/713bc57a-0991-11ef-bc97-c7b262605968/mon.a/config 2024-05-03T21:11:50.876 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:11:50 smithi067 ceph-mon[29730]: Updating smithi067:/var/lib/ceph/713bc57a-0991-11ef-bc97-c7b262605968/config/ceph.client.admin.keyring 2024-05-03T21:11:51.352 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:11:51.350+0000 7f5b5fd8c700 1 -- 172.21.15.67:0/3113759263 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5b58109030 msgr2=0x7f5b58109410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:11:51.353 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:11:51.350+0000 7f5b5fd8c700 1 --2- 172.21.15.67:0/3113759263 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5b58109030 0x7f5b58109410 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f5b54009910 tx=0x7f5b54009c20 comp rx=0 tx=0).stop 2024-05-03T21:11:51.353 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:11:51.350+0000 7f5b5fd8c700 1 -- 172.21.15.67:0/3113759263 shutdown_connections 2024-05-03T21:11:51.353 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:11:51.350+0000 7f5b5fd8c700 1 --2- 172.21.15.67:0/3113759263 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5b58109030 0x7f5b58109410 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:11:51.353 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:11:51.350+0000 7f5b5fd8c700 1 -- 172.21.15.67:0/3113759263 >> 172.21.15.67:0/3113759263 conn(0x7f5b58078520 msgr2=0x7f5b58078930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T21:11:51.353 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:11:51.350+0000 7f5b5fd8c700 1 -- 172.21.15.67:0/3113759263 shutdown_connections 2024-05-03T21:11:51.353 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:11:51.350+0000 7f5b5fd8c700 1 -- 172.21.15.67:0/3113759263 wait complete. 2024-05-03T21:11:51.354 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:11:51.351+0000 7f5b5fd8c700 1 Processor -- start 2024-05-03T21:11:51.354 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:11:51.351+0000 7f5b5fd8c700 1 -- start start 2024-05-03T21:11:51.355 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:11:51.351+0000 7f5b5fd8c700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5b58109030 0x7f5b5819e7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:11:51.355 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:11:51.351+0000 7f5b5fd8c700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f5b5810d770 con 0x7f5b58109030 2024-05-03T21:11:51.355 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:11:51.352+0000 7f5b5db28700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5b58109030 0x7f5b5819e7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:11:51.355 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:11:51.352+0000 7f5b5db28700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5b58109030 0x7f5b5819e7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:41390/0 (socket says 172.21.15.67:41390) 2024-05-03T21:11:51.355 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:11:51.352+0000 7f5b5db28700 1 -- 172.21.15.67:0/2831368360 learned_addr learned my addr 172.21.15.67:0/2831368360 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-03T21:11:51.355 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:11:51.352+0000 7f5b5db28700 1 -- 172.21.15.67:0/2831368360 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5b5819ed20 con 0x7f5b58109030 2024-05-03T21:11:51.355 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:11:51.352+0000 7f5b5db28700 1 --2- 172.21.15.67:0/2831368360 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5b58109030 0x7f5b5819e7e0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f5b540052a0 tx=0x7f5b54004cc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T21:11:51.355 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:11:51.352+0000 7f5b4effd700 1 -- 172.21.15.67:0/2831368360 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5b54026070 con 0x7f5b58109030 2024-05-03T21:11:51.355 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:11:51.352+0000 7f5b4effd700 1 -- 172.21.15.67:0/2831368360 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(28 keys) ==== 1170+0+0 (secure 0 0 0) 0x7f5b5400bc50 con 0x7f5b58109030 2024-05-03T21:11:51.355 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:11:51.352+0000 7f5b5fd8c700 1 -- 172.21.15.67:0/2831368360 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5b5819efb0 con 0x7f5b58109030 2024-05-03T21:11:51.355 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:11:51.352+0000 7f5b4effd700 1 -- 172.21.15.67:0/2831368360 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5b54003a20 con 0x7f5b58109030 2024-05-03T21:11:51.355 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:11:51.352+0000 7f5b5fd8c700 1 -- 172.21.15.67:0/2831368360 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5b581989a0 con 0x7f5b58109030 2024-05-03T21:11:51.356 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:11:51.353+0000 7f5b5fd8c700 1 -- 172.21.15.67:0/2831368360 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5b58104750 con 0x7f5b58109030 2024-05-03T21:11:51.356 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:11:51.353+0000 7f5b4effd700 1 -- 172.21.15.67:0/2831368360 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 13) ==== 50278+0+0 (secure 0 0 0) 0x7f5b54029070 con 0x7f5b58109030 2024-05-03T21:11:51.356 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:11:51.353+0000 7f5b4effd700 1 --2- 172.21.15.67:0/2831368360 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f5b4403e570 0x7f5b44040a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:11:51.356 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:11:51.353+0000 7f5b4effd700 1 -- 172.21.15.67:0/2831368360 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f5b5401e070 con 0x7f5b58109030 2024-05-03T21:11:51.356 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:11:51.354+0000 7f5b5d327700 1 --2- 172.21.15.67:0/2831368360 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f5b4403e570 0x7f5b44040a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:11:51.356 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:11:51.354+0000 7f5b5d327700 1 --2- 172.21.15.67:0/2831368360 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f5b4403e570 0x7f5b44040a30 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f5b48006fd0 tx=0x7f5b48006e40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T21:11:51.360 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:11:51.359+0000 7f5b4effd700 1 -- 172.21.15.67:0/2831368360 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191839 (secure 0 0 0) 0x7f5b54013070 con 0x7f5b58109030 2024-05-03T21:11:51.577 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:11:51.575+0000 7f5b5fd8c700 1 -- 172.21.15.67:0/2831368360 --> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi082", "target": ["mon-mgr", ""]}) -- 0x7f5b58199ed0 con 0x7f5b4403e570 2024-05-03T21:11:52.721 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:11:52 smithi067 ceph-mon[29730]: from='client.14176 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi082", "target": ["mon-mgr", ""]}]: dispatch 2024-05-03T21:11:52.721 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:11:52 smithi067 ceph-mon[29730]: Deploying cephadm binary to smithi082 2024-05-03T21:11:55.221 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:11:54 smithi067 ceph-mon[29730]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-03T21:11:55.749 INFO:teuthology.orchestra.run.smithi067.stdout:Added host 'smithi082' with addr '172.21.15.82' 2024-05-03T21:11:55.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:11:55.746+0000 7f5b4effd700 1 -- 172.21.15.67:0/2831368360 <== mgr.14150 v2:172.21.15.67:6800/604451592 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+48 (secure 0 0 0) 0x7f5b58199ed0 con 0x7f5b4403e570 2024-05-03T21:11:55.751 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:11:55.749+0000 7f5b5fd8c700 1 -- 172.21.15.67:0/2831368360 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f5b4403e570 msgr2=0x7f5b44040a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:11:55.751 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:11:55.749+0000 7f5b5fd8c700 1 --2- 172.21.15.67:0/2831368360 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f5b4403e570 0x7f5b44040a30 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f5b48006fd0 tx=0x7f5b48006e40 comp rx=0 tx=0).stop 2024-05-03T21:11:55.751 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:11:55.749+0000 7f5b5fd8c700 1 -- 172.21.15.67:0/2831368360 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5b58109030 msgr2=0x7f5b5819e7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:11:55.751 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:11:55.749+0000 7f5b5fd8c700 1 --2- 172.21.15.67:0/2831368360 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5b58109030 0x7f5b5819e7e0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f5b540052a0 tx=0x7f5b54004cc0 comp rx=0 tx=0).stop 2024-05-03T21:11:55.751 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:11:55.749+0000 7f5b5fd8c700 1 -- 172.21.15.67:0/2831368360 shutdown_connections 2024-05-03T21:11:55.751 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:11:55.749+0000 7f5b5fd8c700 1 --2- 172.21.15.67:0/2831368360 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f5b4403e570 0x7f5b44040a30 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:11:55.752 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:11:55.749+0000 7f5b5fd8c700 1 --2- 172.21.15.67:0/2831368360 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5b58109030 0x7f5b5819e7e0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:11:55.752 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:11:55.749+0000 7f5b5fd8c700 1 -- 172.21.15.67:0/2831368360 >> 172.21.15.67:0/2831368360 conn(0x7f5b58078520 msgr2=0x7f5b58104ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T21:11:55.752 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:11:55.749+0000 7f5b5fd8c700 1 -- 172.21.15.67:0/2831368360 shutdown_connections 2024-05-03T21:11:55.752 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:11:55.749+0000 7f5b5fd8c700 1 -- 172.21.15.67:0/2831368360 wait complete. 2024-05-03T21:11:56.384 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9a047fe7599a549042d06a3fb5b6813a203eabea shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 713bc57a-0991-11ef-bc97-c7b262605968 -- ceph orch host ls --format=json 2024-05-03T21:11:56.629 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/713bc57a-0991-11ef-bc97-c7b262605968/mon.a/config 2024-05-03T21:11:56.970 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:11:56 smithi067 ceph-mon[29730]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-03T21:11:56.971 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:11:56 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:11:56.971 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:11:56 smithi067 ceph-mon[29730]: Added host smithi082 2024-05-03T21:11:56.971 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:11:56 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T21:11:56.971 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:11:56 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:11:57.988 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:11:57 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:11:58.213 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:11:58.211+0000 7fa732648700 1 -- 172.21.15.67:0/2117016468 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa72c102010 msgr2=0x7fa72c1023f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:11:58.213 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:11:58.211+0000 7fa732648700 1 --2- 172.21.15.67:0/2117016468 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa72c102010 0x7fa72c1023f0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fa71c009910 tx=0x7fa71c009c20 comp rx=0 tx=0).stop 2024-05-03T21:11:58.213 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:11:58.211+0000 7fa732648700 1 -- 172.21.15.67:0/2117016468 shutdown_connections 2024-05-03T21:11:58.213 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:11:58.211+0000 7fa732648700 1 --2- 172.21.15.67:0/2117016468 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa72c102010 0x7fa72c1023f0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:11:58.213 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:11:58.211+0000 7fa732648700 1 -- 172.21.15.67:0/2117016468 >> 172.21.15.67:0/2117016468 conn(0x7fa72c0fd8c0 msgr2=0x7fa72c0ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T21:11:58.214 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:11:58.211+0000 7fa732648700 1 -- 172.21.15.67:0/2117016468 shutdown_connections 2024-05-03T21:11:58.214 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:11:58.211+0000 7fa732648700 1 -- 172.21.15.67:0/2117016468 wait complete. 2024-05-03T21:11:58.214 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:11:58.211+0000 7fa732648700 1 Processor -- start 2024-05-03T21:11:58.214 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:11:58.211+0000 7fa732648700 1 -- start start 2024-05-03T21:11:58.216 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:11:58.212+0000 7fa732648700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa72c102010 0x7fa72c10ad70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:11:58.216 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:11:58.212+0000 7fa732648700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fa72c10d640 con 0x7fa72c102010 2024-05-03T21:11:58.216 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:11:58.212+0000 7fa72bfff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa72c102010 0x7fa72c10ad70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:11:58.216 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:11:58.212+0000 7fa72bfff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa72c102010 0x7fa72c10ad70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:41716/0 (socket says 172.21.15.67:41716) 2024-05-03T21:11:58.216 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:11:58.212+0000 7fa72bfff700 1 -- 172.21.15.67:0/227835128 learned_addr learned my addr 172.21.15.67:0/227835128 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-03T21:11:58.216 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:11:58.212+0000 7fa72bfff700 1 -- 172.21.15.67:0/227835128 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa72c10cc60 con 0x7fa72c102010 2024-05-03T21:11:58.216 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:11:58.212+0000 7fa72bfff700 1 --2- 172.21.15.67:0/227835128 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa72c102010 0x7fa72c10ad70 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fa71c0052a0 tx=0x7fa71c004cc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T21:11:58.216 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:11:58.212+0000 7fa7297fa700 1 -- 172.21.15.67:0/227835128 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa71c026070 con 0x7fa72c102010 2024-05-03T21:11:58.216 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:11:58.212+0000 7fa7297fa700 1 -- 172.21.15.67:0/227835128 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(28 keys) ==== 1170+0+0 (secure 0 0 0) 0x7fa71c00bc50 con 0x7fa72c102010 2024-05-03T21:11:58.217 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:11:58.212+0000 7fa732648700 1 -- 172.21.15.67:0/227835128 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa72c10b310 con 0x7fa72c102010 2024-05-03T21:11:58.217 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:11:58.212+0000 7fa7297fa700 1 -- 172.21.15.67:0/227835128 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa71c003a20 con 0x7fa72c102010 2024-05-03T21:11:58.217 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:11:58.212+0000 7fa732648700 1 -- 172.21.15.67:0/227835128 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa72c10b7d0 con 0x7fa72c102010 2024-05-03T21:11:58.217 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:11:58.213+0000 7fa7297fa700 1 -- 172.21.15.67:0/227835128 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 13) ==== 50278+0+0 (secure 0 0 0) 0x7fa71c029070 con 0x7fa72c102010 2024-05-03T21:11:58.217 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:11:58.213+0000 7fa732648700 1 -- 172.21.15.67:0/227835128 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa72c1034f0 con 0x7fa72c102010 2024-05-03T21:11:58.217 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:11:58.214+0000 7fa7297fa700 1 --2- 172.21.15.67:0/227835128 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7fa71403e510 0x7fa7140409d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:11:58.217 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:11:58.214+0000 7fa7297fa700 1 -- 172.21.15.67:0/227835128 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7fa71c01e070 con 0x7fa72c102010 2024-05-03T21:11:58.217 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:11:58.214+0000 7fa72b7fe700 1 --2- 172.21.15.67:0/227835128 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7fa71403e510 0x7fa7140409d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:11:58.217 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:11:58.214+0000 7fa72b7fe700 1 --2- 172.21.15.67:0/227835128 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7fa71403e510 0x7fa7140409d0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fa720006fd0 tx=0x7fa720006e40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T21:11:58.221 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:11:58.219+0000 7fa7297fa700 1 -- 172.21.15.67:0/227835128 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191839 (secure 0 0 0) 0x7fa71c013070 con 0x7fa72c102010 2024-05-03T21:11:58.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:11:58.453+0000 7fa732648700 1 -- 172.21.15.67:0/227835128 --> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fa72c10c2a0 con 0x7fa71403e510 2024-05-03T21:11:58.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:11:58.455+0000 7fa7297fa700 1 -- 172.21.15.67:0/227835128 <== mgr.14150 v2:172.21.15.67:6800/604451592 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+159 (secure 0 0 0) 0x7fa72c10c2a0 con 0x7fa71403e510 2024-05-03T21:11:58.456 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-05-03T21:11:58.457 INFO:teuthology.orchestra.run.smithi067.stdout:[{"addr": "172.21.15.67", "hostname": "smithi067", "labels": [], "status": ""}, {"addr": "172.21.15.82", "hostname": "smithi082", "labels": [], "status": ""}] 2024-05-03T21:11:58.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:11:58.457+0000 7fa732648700 1 -- 172.21.15.67:0/227835128 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7fa71403e510 msgr2=0x7fa7140409d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:11:58.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:11:58.457+0000 7fa732648700 1 --2- 172.21.15.67:0/227835128 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7fa71403e510 0x7fa7140409d0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fa720006fd0 tx=0x7fa720006e40 comp rx=0 tx=0).stop 2024-05-03T21:11:58.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:11:58.457+0000 7fa732648700 1 -- 172.21.15.67:0/227835128 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa72c102010 msgr2=0x7fa72c10ad70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:11:58.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:11:58.457+0000 7fa732648700 1 --2- 172.21.15.67:0/227835128 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa72c102010 0x7fa72c10ad70 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fa71c0052a0 tx=0x7fa71c004cc0 comp rx=0 tx=0).stop 2024-05-03T21:11:58.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:11:58.457+0000 7fa732648700 1 -- 172.21.15.67:0/227835128 shutdown_connections 2024-05-03T21:11:58.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:11:58.457+0000 7fa732648700 1 --2- 172.21.15.67:0/227835128 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7fa71403e510 0x7fa7140409d0 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:11:58.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:11:58.457+0000 7fa732648700 1 --2- 172.21.15.67:0/227835128 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa72c102010 0x7fa72c10ad70 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:11:58.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:11:58.457+0000 7fa732648700 1 -- 172.21.15.67:0/227835128 >> 172.21.15.67:0/227835128 conn(0x7fa72c0fd8c0 msgr2=0x7fa72c1039e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T21:11:58.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:11:58.458+0000 7fa732648700 1 -- 172.21.15.67:0/227835128 shutdown_connections 2024-05-03T21:11:58.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:11:58.458+0000 7fa732648700 1 -- 172.21.15.67:0/227835128 wait complete. 2024-05-03T21:11:59.019 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:11:58 smithi067 ceph-mon[29730]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-03T21:11:59.020 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:11:58 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:11:59.048 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi138 2024-05-03T21:11:59.048 DEBUG:teuthology.orchestra.run.smithi138:> set -ex 2024-05-03T21:11:59.048 DEBUG:teuthology.orchestra.run.smithi138:> dd of=/etc/ceph/ceph.conf 2024-05-03T21:11:59.068 DEBUG:teuthology.orchestra.run.smithi138:> set -ex 2024-05-03T21:11:59.068 DEBUG:teuthology.orchestra.run.smithi138:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-05-03T21:11:59.126 INFO:tasks.cephadm:Adding host smithi138 to orchestrator... 2024-05-03T21:11:59.126 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9a047fe7599a549042d06a3fb5b6813a203eabea shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 713bc57a-0991-11ef-bc97-c7b262605968 -- ceph orch host add smithi138 2024-05-03T21:11:59.370 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/713bc57a-0991-11ef-bc97-c7b262605968/mon.a/config 2024-05-03T21:12:00.221 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:11:59 smithi067 ceph-mon[29730]: from='client.14178 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-03T21:12:01.005 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:12:01.002+0000 7f85c97dd700 1 -- 172.21.15.67:0/3830900208 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f85c4104de0 msgr2=0x7f85c41051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:12:01.005 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:12:01.002+0000 7f85c97dd700 1 --2- 172.21.15.67:0/3830900208 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f85c4104de0 0x7f85c41051c0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f85b4009910 tx=0x7f85b4009c20 comp rx=0 tx=0).stop 2024-05-03T21:12:01.005 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:12:01.002+0000 7f85c97dd700 1 -- 172.21.15.67:0/3830900208 shutdown_connections 2024-05-03T21:12:01.005 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:12:01.002+0000 7f85c97dd700 1 --2- 172.21.15.67:0/3830900208 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f85c4104de0 0x7f85c41051c0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:12:01.005 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:12:01.002+0000 7f85c97dd700 1 -- 172.21.15.67:0/3830900208 >> 172.21.15.67:0/3830900208 conn(0x7f85c40fd010 msgr2=0x7f85c40ff430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T21:12:01.005 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:12:01.002+0000 7f85c97dd700 1 -- 172.21.15.67:0/3830900208 shutdown_connections 2024-05-03T21:12:01.005 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:12:01.002+0000 7f85c97dd700 1 -- 172.21.15.67:0/3830900208 wait complete. 2024-05-03T21:12:01.005 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:12:01.003+0000 7f85c97dd700 1 Processor -- start 2024-05-03T21:12:01.007 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:12:01.003+0000 7f85c97dd700 1 -- start start 2024-05-03T21:12:01.007 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:12:01.003+0000 7f85c97dd700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f85c4104de0 0x7f85c419e7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:12:01.007 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:12:01.003+0000 7f85c97dd700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f85c410d750 con 0x7f85c4104de0 2024-05-03T21:12:01.007 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:12:01.003+0000 7f85c2ffd700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f85c4104de0 0x7f85c419e7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:12:01.007 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:12:01.004+0000 7f85c2ffd700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f85c4104de0 0x7f85c419e7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:41730/0 (socket says 172.21.15.67:41730) 2024-05-03T21:12:01.007 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:12:01.004+0000 7f85c2ffd700 1 -- 172.21.15.67:0/2868363479 learned_addr learned my addr 172.21.15.67:0/2868363479 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-03T21:12:01.007 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:12:01.004+0000 7f85c2ffd700 1 -- 172.21.15.67:0/2868363479 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f85c419ed00 con 0x7f85c4104de0 2024-05-03T21:12:01.007 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:12:01.004+0000 7f85c2ffd700 1 --2- 172.21.15.67:0/2868363479 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f85c4104de0 0x7f85c419e7c0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f85b40052a0 tx=0x7f85b4004cc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T21:12:01.007 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:12:01.004+0000 7f85abfff700 1 -- 172.21.15.67:0/2868363479 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f85b4026070 con 0x7f85c4104de0 2024-05-03T21:12:01.007 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:12:01.004+0000 7f85abfff700 1 -- 172.21.15.67:0/2868363479 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(28 keys) ==== 1170+0+0 (secure 0 0 0) 0x7f85b400bbf0 con 0x7f85c4104de0 2024-05-03T21:12:01.007 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:12:01.004+0000 7f85c97dd700 1 -- 172.21.15.67:0/2868363479 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f85c419eff0 con 0x7f85c4104de0 2024-05-03T21:12:01.007 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:12:01.004+0000 7f85abfff700 1 -- 172.21.15.67:0/2868363479 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f85b4003a30 con 0x7f85c4104de0 2024-05-03T21:12:01.008 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:12:01.004+0000 7f85c97dd700 1 -- 172.21.15.67:0/2868363479 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f85c4198980 con 0x7f85c4104de0 2024-05-03T21:12:01.008 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:12:01.005+0000 7f85c97dd700 1 -- 172.21.15.67:0/2868363479 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f85c4104730 con 0x7f85c4104de0 2024-05-03T21:12:01.008 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:12:01.005+0000 7f85abfff700 1 -- 172.21.15.67:0/2868363479 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 13) ==== 50278+0+0 (secure 0 0 0) 0x7f85b4029070 con 0x7f85c4104de0 2024-05-03T21:12:01.008 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:12:01.005+0000 7f85abfff700 1 --2- 172.21.15.67:0/2868363479 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f85ac03e510 0x7f85ac0409d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:12:01.008 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:12:01.005+0000 7f85abfff700 1 -- 172.21.15.67:0/2868363479 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f85b401e070 con 0x7f85c4104de0 2024-05-03T21:12:01.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:12:01.005+0000 7f85c27fc700 1 --2- 172.21.15.67:0/2868363479 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f85ac03e510 0x7f85ac0409d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:12:01.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:12:01.006+0000 7f85c27fc700 1 --2- 172.21.15.67:0/2868363479 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f85ac03e510 0x7f85ac0409d0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f85b8006fd0 tx=0x7f85b8006e40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T21:12:01.012 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:12:01.010+0000 7f85abfff700 1 -- 172.21.15.67:0/2868363479 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191839 (secure 0 0 0) 0x7f85b4013070 con 0x7f85c4104de0 2024-05-03T21:12:01.012 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:12:00 smithi067 ceph-mon[29730]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-03T21:12:01.229 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:12:01.227+0000 7f85c97dd700 1 -- 172.21.15.67:0/2868363479 --> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi138", "target": ["mon-mgr", ""]}) -- 0x7f85c4199eb0 con 0x7f85ac03e510 2024-05-03T21:12:02.221 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:12:01 smithi067 ceph-mon[29730]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi138", "target": ["mon-mgr", ""]}]: dispatch 2024-05-03T21:12:02.221 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:12:01 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:12:02.221 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:12:01 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:12:02.221 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:12:01 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:12:02.221 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:12:01 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:12:02.221 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:12:01 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi082", "name": "osd_memory_target"}]: dispatch 2024-05-03T21:12:02.221 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:12:01 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T21:12:02.221 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:12:01 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T21:12:03.221 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:12:02 smithi067 ceph-mon[29730]: Updating smithi082:/etc/ceph/ceph.conf 2024-05-03T21:12:03.221 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:12:02 smithi067 ceph-mon[29730]: Updating smithi082:/var/lib/ceph/713bc57a-0991-11ef-bc97-c7b262605968/config/ceph.conf 2024-05-03T21:12:03.221 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:12:02 smithi067 ceph-mon[29730]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-03T21:12:03.221 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:12:02 smithi067 ceph-mon[29730]: Deploying cephadm binary to smithi138 2024-05-03T21:12:03.221 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:12:02 smithi067 ceph-mon[29730]: Updating smithi082:/etc/ceph/ceph.client.admin.keyring 2024-05-03T21:12:03.221 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:12:02 smithi067 ceph-mon[29730]: Updating smithi082:/var/lib/ceph/713bc57a-0991-11ef-bc97-c7b262605968/config/ceph.client.admin.keyring 2024-05-03T21:12:05.221 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:12:04 smithi067 ceph-mon[29730]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-03T21:12:05.221 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:12:04 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:12:05.221 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:12:04 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:12:05.221 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:12:04 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:12:05.469 INFO:teuthology.orchestra.run.smithi067.stdout:Added host 'smithi138' with addr '172.21.15.138' 2024-05-03T21:12:05.470 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:12:05.467+0000 7f85abfff700 1 -- 172.21.15.67:0/2868363479 <== mgr.14150 v2:172.21.15.67:6800/604451592 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+49 (secure 0 0 0) 0x7f85c4199eb0 con 0x7f85ac03e510 2024-05-03T21:12:05.472 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:12:05.470+0000 7f85c97dd700 1 -- 172.21.15.67:0/2868363479 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f85ac03e510 msgr2=0x7f85ac0409d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:12:05.472 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:12:05.470+0000 7f85c97dd700 1 --2- 172.21.15.67:0/2868363479 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f85ac03e510 0x7f85ac0409d0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f85b8006fd0 tx=0x7f85b8006e40 comp rx=0 tx=0).stop 2024-05-03T21:12:05.472 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:12:05.470+0000 7f85c97dd700 1 -- 172.21.15.67:0/2868363479 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f85c4104de0 msgr2=0x7f85c419e7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:12:05.472 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:12:05.470+0000 7f85c97dd700 1 --2- 172.21.15.67:0/2868363479 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f85c4104de0 0x7f85c419e7c0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f85b40052a0 tx=0x7f85b4004cc0 comp rx=0 tx=0).stop 2024-05-03T21:12:05.472 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:12:05.470+0000 7f85c97dd700 1 -- 172.21.15.67:0/2868363479 shutdown_connections 2024-05-03T21:12:05.473 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:12:05.470+0000 7f85c97dd700 1 --2- 172.21.15.67:0/2868363479 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f85ac03e510 0x7f85ac0409d0 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:12:05.473 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:12:05.470+0000 7f85c97dd700 1 --2- 172.21.15.67:0/2868363479 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f85c4104de0 0x7f85c419e7c0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:12:05.473 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:12:05.470+0000 7f85c97dd700 1 -- 172.21.15.67:0/2868363479 >> 172.21.15.67:0/2868363479 conn(0x7f85c40fd010 msgr2=0x7f85c40fe8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T21:12:05.473 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:12:05.470+0000 7f85c97dd700 1 -- 172.21.15.67:0/2868363479 shutdown_connections 2024-05-03T21:12:05.473 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:12:05.470+0000 7f85c97dd700 1 -- 172.21.15.67:0/2868363479 wait complete. 2024-05-03T21:12:06.131 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9a047fe7599a549042d06a3fb5b6813a203eabea shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 713bc57a-0991-11ef-bc97-c7b262605968 -- ceph orch host ls --format=json 2024-05-03T21:12:06.378 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/713bc57a-0991-11ef-bc97-c7b262605968/mon.a/config 2024-05-03T21:12:06.721 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:12:06 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:12:06.721 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:12:06 smithi067 ceph-mon[29730]: Added host smithi138 2024-05-03T21:12:06.721 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:12:06 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T21:12:06.721 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:12:06 smithi067 ceph-mon[29730]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-03T21:12:06.721 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:12:06 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:12:07.900 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:12:07 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:12:08.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:12:08.002+0000 7fe5fc0fc700 1 -- 172.21.15.67:0/305497045 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe5f4104110 msgr2=0x7fe5f41044f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:12:08.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:12:08.002+0000 7fe5fc0fc700 1 --2- 172.21.15.67:0/305497045 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe5f4104110 0x7fe5f41044f0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fe5e8009910 tx=0x7fe5e8009c20 comp rx=0 tx=0).stop 2024-05-03T21:12:08.005 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:12:08.002+0000 7fe5fc0fc700 1 -- 172.21.15.67:0/305497045 shutdown_connections 2024-05-03T21:12:08.005 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:12:08.002+0000 7fe5fc0fc700 1 --2- 172.21.15.67:0/305497045 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe5f4104110 0x7fe5f41044f0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:12:08.005 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:12:08.002+0000 7fe5fc0fc700 1 -- 172.21.15.67:0/305497045 >> 172.21.15.67:0/305497045 conn(0x7fe5f40fd8a0 msgr2=0x7fe5f40ffcc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T21:12:08.005 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:12:08.002+0000 7fe5fc0fc700 1 -- 172.21.15.67:0/305497045 shutdown_connections 2024-05-03T21:12:08.005 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:12:08.002+0000 7fe5fc0fc700 1 -- 172.21.15.67:0/305497045 wait complete. 2024-05-03T21:12:08.006 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:12:08.003+0000 7fe5fc0fc700 1 Processor -- start 2024-05-03T21:12:08.006 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:12:08.003+0000 7fe5fc0fc700 1 -- start start 2024-05-03T21:12:08.006 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:12:08.003+0000 7fe5fc0fc700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe5f4104110 0x7fe5f419e6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:12:08.006 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:12:08.003+0000 7fe5fc0fc700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fe5f410d6a0 con 0x7fe5f4104110 2024-05-03T21:12:08.007 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:12:08.003+0000 7fe5f9e98700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe5f4104110 0x7fe5f419e6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:12:08.007 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:12:08.003+0000 7fe5f9e98700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe5f4104110 0x7fe5f419e6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:54898/0 (socket says 172.21.15.67:54898) 2024-05-03T21:12:08.007 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:12:08.003+0000 7fe5f9e98700 1 -- 172.21.15.67:0/2150622236 learned_addr learned my addr 172.21.15.67:0/2150622236 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-03T21:12:08.007 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:12:08.004+0000 7fe5f9e98700 1 -- 172.21.15.67:0/2150622236 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe5f419ec20 con 0x7fe5f4104110 2024-05-03T21:12:08.007 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:12:08.004+0000 7fe5f9e98700 1 --2- 172.21.15.67:0/2150622236 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe5f4104110 0x7fe5f419e6e0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fe5e8004c10 tx=0x7fe5e8004cf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T21:12:08.007 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:12:08.004+0000 7fe5e6ffd700 1 -- 172.21.15.67:0/2150622236 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe5e8026070 con 0x7fe5f4104110 2024-05-03T21:12:08.007 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:12:08.004+0000 7fe5fc0fc700 1 -- 172.21.15.67:0/2150622236 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe5f419eeb0 con 0x7fe5f4104110 2024-05-03T21:12:08.007 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:12:08.004+0000 7fe5e6ffd700 1 -- 172.21.15.67:0/2150622236 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(28 keys) ==== 1170+0+0 (secure 0 0 0) 0x7fe5e8004370 con 0x7fe5f4104110 2024-05-03T21:12:08.007 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:12:08.004+0000 7fe5e6ffd700 1 -- 172.21.15.67:0/2150622236 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe5e8003c00 con 0x7fe5f4104110 2024-05-03T21:12:08.007 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:12:08.004+0000 7fe5fc0fc700 1 -- 172.21.15.67:0/2150622236 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe5f41988a0 con 0x7fe5f4104110 2024-05-03T21:12:08.008 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:12:08.005+0000 7fe5e6ffd700 1 -- 172.21.15.67:0/2150622236 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 13) ==== 50278+0+0 (secure 0 0 0) 0x7fe5e8029070 con 0x7fe5f4104110 2024-05-03T21:12:08.008 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:12:08.005+0000 7fe5fc0fc700 1 -- 172.21.15.67:0/2150622236 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe5f406bb50 con 0x7fe5f4104110 2024-05-03T21:12:08.008 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:12:08.005+0000 7fe5e6ffd700 1 --2- 172.21.15.67:0/2150622236 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7fe5e003e570 0x7fe5e0040a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:12:08.008 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:12:08.005+0000 7fe5e6ffd700 1 -- 172.21.15.67:0/2150622236 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7fe5e801e070 con 0x7fe5f4104110 2024-05-03T21:12:08.008 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:12:08.005+0000 7fe5f9697700 1 --2- 172.21.15.67:0/2150622236 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7fe5e003e570 0x7fe5e0040a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:12:08.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:12:08.006+0000 7fe5f9697700 1 --2- 172.21.15.67:0/2150622236 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7fe5e003e570 0x7fe5e0040a30 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7fe5f0006fd0 tx=0x7fe5f0006e40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T21:12:08.012 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:12:08.010+0000 7fe5e6ffd700 1 -- 172.21.15.67:0/2150622236 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191839 (secure 0 0 0) 0x7fe5e8013070 con 0x7fe5f4104110 2024-05-03T21:12:08.245 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:12:08.243+0000 7fe5fc0fc700 1 -- 172.21.15.67:0/2150622236 --> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fe5f4199dd0 con 0x7fe5e003e570 2024-05-03T21:12:08.245 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:12:08.244+0000 7fe5e6ffd700 1 -- 172.21.15.67:0/2150622236 <== mgr.14150 v2:172.21.15.67:6800/604451592 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+239 (secure 0 0 0) 0x7fe5f4199dd0 con 0x7fe5e003e570 2024-05-03T21:12:08.246 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-05-03T21:12:08.246 INFO:teuthology.orchestra.run.smithi067.stdout:[{"addr": "172.21.15.67", "hostname": "smithi067", "labels": [], "status": ""}, {"addr": "172.21.15.82", "hostname": "smithi082", "labels": [], "status": ""}, {"addr": "172.21.15.138", "hostname": "smithi138", "labels": [], "status": ""}] 2024-05-03T21:12:08.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:12:08.246+0000 7fe5fc0fc700 1 -- 172.21.15.67:0/2150622236 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7fe5e003e570 msgr2=0x7fe5e0040a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:12:08.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:12:08.246+0000 7fe5fc0fc700 1 --2- 172.21.15.67:0/2150622236 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7fe5e003e570 0x7fe5e0040a30 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7fe5f0006fd0 tx=0x7fe5f0006e40 comp rx=0 tx=0).stop 2024-05-03T21:12:08.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:12:08.246+0000 7fe5fc0fc700 1 -- 172.21.15.67:0/2150622236 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe5f4104110 msgr2=0x7fe5f419e6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:12:08.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:12:08.246+0000 7fe5fc0fc700 1 --2- 172.21.15.67:0/2150622236 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe5f4104110 0x7fe5f419e6e0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fe5e8004c10 tx=0x7fe5e8004cf0 comp rx=0 tx=0).stop 2024-05-03T21:12:08.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:12:08.246+0000 7fe5fc0fc700 1 -- 172.21.15.67:0/2150622236 shutdown_connections 2024-05-03T21:12:08.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:12:08.246+0000 7fe5fc0fc700 1 --2- 172.21.15.67:0/2150622236 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7fe5e003e570 0x7fe5e0040a30 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:12:08.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:12:08.246+0000 7fe5fc0fc700 1 --2- 172.21.15.67:0/2150622236 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe5f4104110 0x7fe5f419e6e0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:12:08.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:12:08.246+0000 7fe5fc0fc700 1 -- 172.21.15.67:0/2150622236 >> 172.21.15.67:0/2150622236 conn(0x7fe5f40fd8a0 msgr2=0x7fe5f4104ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T21:12:08.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:12:08.246+0000 7fe5fc0fc700 1 -- 172.21.15.67:0/2150622236 shutdown_connections 2024-05-03T21:12:08.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:12:08.246+0000 7fe5fc0fc700 1 -- 172.21.15.67:0/2150622236 wait complete. 2024-05-03T21:12:08.832 INFO:tasks.cephadm:Setting crush tunables to default 2024-05-03T21:12:08.832 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9a047fe7599a549042d06a3fb5b6813a203eabea shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 713bc57a-0991-11ef-bc97-c7b262605968 -- ceph osd crush tunables default 2024-05-03T21:12:09.072 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/713bc57a-0991-11ef-bc97-c7b262605968/mon.a/config 2024-05-03T21:12:09.412 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:12:09 smithi067 ceph-mon[29730]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-03T21:12:09.412 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:12:09 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:12:09.412 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:12:09 smithi067 ceph-mon[29730]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-03T21:12:10.621 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:12:10.619+0000 7f193efcb700 1 -- 172.21.15.67:0/1212162255 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1938109030 msgr2=0x7f1938109410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:12:10.621 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:12:10.619+0000 7f193efcb700 1 --2- 172.21.15.67:0/1212162255 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1938109030 0x7f1938109410 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f193000b0a0 tx=0x7f193000b3b0 comp rx=0 tx=0).stop 2024-05-03T21:12:10.621 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:12:10.619+0000 7f193efcb700 1 -- 172.21.15.67:0/1212162255 shutdown_connections 2024-05-03T21:12:10.622 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:12:10.619+0000 7f193efcb700 1 --2- 172.21.15.67:0/1212162255 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1938109030 0x7f1938109410 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:12:10.622 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:12:10.619+0000 7f193efcb700 1 -- 172.21.15.67:0/1212162255 >> 172.21.15.67:0/1212162255 conn(0x7f1938078520 msgr2=0x7f1938078930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T21:12:10.622 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:12:10.619+0000 7f193efcb700 1 -- 172.21.15.67:0/1212162255 shutdown_connections 2024-05-03T21:12:10.622 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:12:10.619+0000 7f193efcb700 1 -- 172.21.15.67:0/1212162255 wait complete. 2024-05-03T21:12:10.623 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:12:10.620+0000 7f193efcb700 1 Processor -- start 2024-05-03T21:12:10.623 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:12:10.620+0000 7f193efcb700 1 -- start start 2024-05-03T21:12:10.623 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:12:10.620+0000 7f193efcb700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1938109030 0x7f193819e7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:12:10.623 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:12:10.620+0000 7f193efcb700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f193810d770 con 0x7f1938109030 2024-05-03T21:12:10.624 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:12:10.620+0000 7f193cd67700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1938109030 0x7f193819e7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:12:10.624 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:12:10.620+0000 7f193cd67700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1938109030 0x7f193819e7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:54914/0 (socket says 172.21.15.67:54914) 2024-05-03T21:12:10.624 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:12:10.620+0000 7f193cd67700 1 -- 172.21.15.67:0/1569214377 learned_addr learned my addr 172.21.15.67:0/1569214377 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-03T21:12:10.624 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:12:10.620+0000 7f193cd67700 1 -- 172.21.15.67:0/1569214377 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f193819ed20 con 0x7f1938109030 2024-05-03T21:12:10.624 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:12:10.621+0000 7f193cd67700 1 --2- 172.21.15.67:0/1569214377 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1938109030 0x7f193819e7e0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f1930015040 tx=0x7f193000cd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T21:12:10.624 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:12:10.621+0000 7f1935ffb700 1 -- 172.21.15.67:0/1569214377 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1930017040 con 0x7f1938109030 2024-05-03T21:12:10.624 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:12:10.621+0000 7f1935ffb700 1 -- 172.21.15.67:0/1569214377 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(28 keys) ==== 1170+0+0 (secure 0 0 0) 0x7f1930022070 con 0x7f1938109030 2024-05-03T21:12:10.624 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:12:10.621+0000 7f193efcb700 1 -- 172.21.15.67:0/1569214377 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f193819efb0 con 0x7f1938109030 2024-05-03T21:12:10.624 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:12:10.621+0000 7f1935ffb700 1 -- 172.21.15.67:0/1569214377 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f193001d450 con 0x7f1938109030 2024-05-03T21:12:10.624 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:12:10.621+0000 7f193efcb700 1 -- 172.21.15.67:0/1569214377 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f19381989a0 con 0x7f1938109030 2024-05-03T21:12:10.625 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:12:10.622+0000 7f1935ffb700 1 -- 172.21.15.67:0/1569214377 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 13) ==== 50278+0+0 (secure 0 0 0) 0x7f1930004040 con 0x7f1938109030 2024-05-03T21:12:10.625 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:12:10.622+0000 7f193efcb700 1 -- 172.21.15.67:0/1569214377 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1938104750 con 0x7f1938109030 2024-05-03T21:12:10.625 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:12:10.622+0000 7f1935ffb700 1 --2- 172.21.15.67:0/1569214377 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f192003e570 0x7f1920040a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:12:10.625 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:12:10.622+0000 7f1935ffb700 1 -- 172.21.15.67:0/1569214377 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f193000f050 con 0x7f1938109030 2024-05-03T21:12:10.625 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:12:10.622+0000 7f1937fff700 1 --2- 172.21.15.67:0/1569214377 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f192003e570 0x7f1920040a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:12:10.625 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:12:10.623+0000 7f1937fff700 1 --2- 172.21.15.67:0/1569214377 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f192003e570 0x7f1920040a30 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f192c006fd0 tx=0x7f192c006e40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T21:12:10.629 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:12:10.627+0000 7f1935ffb700 1 -- 172.21.15.67:0/1569214377 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191839 (secure 0 0 0) 0x7f1930004480 con 0x7f1938109030 2024-05-03T21:12:10.848 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:12:10.846+0000 7f193efcb700 1 -- 172.21.15.67:0/1569214377 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) -- 0x7f1938199ed0 con 0x7f1938109030 2024-05-03T21:12:11.084 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:12:11.082+0000 7f1935ffb700 1 -- 172.21.15.67:0/1569214377 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v4) ==== 124+0+0 (secure 0 0 0) 0x7f19300102f0 con 0x7f1938109030 2024-05-03T21:12:11.087 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:12:11.085+0000 7f193efcb700 1 -- 172.21.15.67:0/1569214377 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f192003e570 msgr2=0x7f1920040a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:12:11.087 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:12:11.085+0000 7f193efcb700 1 --2- 172.21.15.67:0/1569214377 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f192003e570 0x7f1920040a30 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f192c006fd0 tx=0x7f192c006e40 comp rx=0 tx=0).stop 2024-05-03T21:12:11.087 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:12:11.085+0000 7f193efcb700 1 -- 172.21.15.67:0/1569214377 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1938109030 msgr2=0x7f193819e7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:12:11.087 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:12:11.085+0000 7f193efcb700 1 --2- 172.21.15.67:0/1569214377 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1938109030 0x7f193819e7e0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f1930015040 tx=0x7f193000cd50 comp rx=0 tx=0).stop 2024-05-03T21:12:11.087 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:12:11.085+0000 7f193efcb700 1 -- 172.21.15.67:0/1569214377 shutdown_connections 2024-05-03T21:12:11.087 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:12:11.085+0000 7f193efcb700 1 --2- 172.21.15.67:0/1569214377 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f192003e570 0x7f1920040a30 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:12:11.087 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:12:11.085+0000 7f193efcb700 1 --2- 172.21.15.67:0/1569214377 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1938109030 0x7f193819e7e0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:12:11.087 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:12:11.085+0000 7f193efcb700 1 -- 172.21.15.67:0/1569214377 >> 172.21.15.67:0/1569214377 conn(0x7f1938078520 msgr2=0x7f1938104ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T21:12:11.088 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:12:11.085+0000 7f193efcb700 1 -- 172.21.15.67:0/1569214377 shutdown_connections 2024-05-03T21:12:11.088 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:12:11.085+0000 7f193efcb700 1 -- 172.21.15.67:0/1569214377 wait complete. 2024-05-03T21:12:11.088 INFO:teuthology.orchestra.run.smithi067.stderr:adjusted tunables profile to default 2024-05-03T21:12:11.399 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:12:11 smithi067 ceph-mon[29730]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-03T21:12:11.399 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:12:11 smithi067 ceph-mon[29730]: from='client.? 172.21.15.67:0/1569214377' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-05-03T21:12:11.719 INFO:tasks.cephadm:Adding mon.a on smithi067 2024-05-03T21:12:11.719 INFO:tasks.cephadm:Adding mon.b on smithi082 2024-05-03T21:12:11.720 INFO:tasks.cephadm:Adding mon.c on smithi138 2024-05-03T21:12:11.720 DEBUG:teuthology.orchestra.run.smithi138:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9a047fe7599a549042d06a3fb5b6813a203eabea shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 713bc57a-0991-11ef-bc97-c7b262605968 -- ceph orch apply mon '3;smithi067:172.21.15.67=a;smithi082:172.21.15.82=b;smithi138:172.21.15.138=c' 2024-05-03T21:12:11.953 INFO:teuthology.orchestra.run.smithi138.stderr:Inferring config /var/lib/ceph/713bc57a-0991-11ef-bc97-c7b262605968/config/ceph.conf 2024-05-03T21:12:12.471 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:12:12 smithi067 ceph-mon[29730]: from='client.? 172.21.15.67:0/1569214377' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-05-03T21:12:12.471 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:12:12 smithi067 ceph-mon[29730]: osdmap e4: 0 total, 0 up, 0 in 2024-05-03T21:12:12.471 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:12:12 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:12:12.472 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:12:12 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:12:12.472 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:12:12 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:12:12.472 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:12:12 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:12:12.472 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:12:12 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi138", "name": "osd_memory_target"}]: dispatch 2024-05-03T21:12:12.472 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:12:12 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T21:12:12.472 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:12:12 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T21:12:12.472 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:12:12 smithi067 ceph-mon[29730]: Updating smithi138:/etc/ceph/ceph.conf 2024-05-03T21:12:12.473 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:12:12 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:12:12.473 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:12:12 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:12:12.473 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:12:12 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:12:13.471 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:12:13 smithi067 ceph-mon[29730]: Updating smithi138:/var/lib/ceph/713bc57a-0991-11ef-bc97-c7b262605968/config/ceph.conf 2024-05-03T21:12:13.471 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:12:13 smithi067 ceph-mon[29730]: Updating smithi138:/etc/ceph/ceph.client.admin.keyring 2024-05-03T21:12:13.471 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:12:13 smithi067 ceph-mon[29730]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-03T21:12:13.471 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:12:13 smithi067 ceph-mon[29730]: Updating smithi138:/var/lib/ceph/713bc57a-0991-11ef-bc97-c7b262605968/config/ceph.client.admin.keyring 2024-05-03T21:12:13.518 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:12:13.516+0000 7ffb574a7700 1 -- 172.21.15.138:0/1229935549 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ffb50102010 msgr2=0x7ffb501023f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:12:13.518 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:12:13.516+0000 7ffb574a7700 1 --2- 172.21.15.138:0/1229935549 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ffb50102010 0x7ffb501023f0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7ffb4c009910 tx=0x7ffb4c009c20 comp rx=0 tx=0).stop 2024-05-03T21:12:13.518 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:12:13.516+0000 7ffb574a7700 1 -- 172.21.15.138:0/1229935549 shutdown_connections 2024-05-03T21:12:13.518 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:12:13.516+0000 7ffb574a7700 1 --2- 172.21.15.138:0/1229935549 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ffb50102010 0x7ffb501023f0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:12:13.518 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:12:13.516+0000 7ffb574a7700 1 -- 172.21.15.138:0/1229935549 >> 172.21.15.138:0/1229935549 conn(0x7ffb500fd8a0 msgr2=0x7ffb500ffcc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T21:12:13.518 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:12:13.516+0000 7ffb574a7700 1 -- 172.21.15.138:0/1229935549 shutdown_connections 2024-05-03T21:12:13.519 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:12:13.516+0000 7ffb574a7700 1 -- 172.21.15.138:0/1229935549 wait complete. 2024-05-03T21:12:13.520 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:12:13.517+0000 7ffb574a7700 1 Processor -- start 2024-05-03T21:12:13.520 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:12:13.517+0000 7ffb574a7700 1 -- start start 2024-05-03T21:12:13.520 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:12:13.517+0000 7ffb574a7700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ffb50102010 0x7ffb501998b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:12:13.520 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:12:13.517+0000 7ffb574a7700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7ffb5010c520 con 0x7ffb50102010 2024-05-03T21:12:13.521 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:12:13.517+0000 7ffb55243700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ffb50102010 0x7ffb501998b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:12:13.521 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:12:13.517+0000 7ffb55243700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ffb50102010 0x7ffb501998b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.138:55268/0 (socket says 172.21.15.138:55268) 2024-05-03T21:12:13.521 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:12:13.517+0000 7ffb55243700 1 -- 172.21.15.138:0/3412331336 learned_addr learned my addr 172.21.15.138:0/3412331336 (peer_addr_for_me v2:172.21.15.138:0/0) 2024-05-03T21:12:13.521 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:12:13.518+0000 7ffb55243700 1 -- 172.21.15.138:0/3412331336 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffb50199df0 con 0x7ffb50102010 2024-05-03T21:12:13.521 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:12:13.518+0000 7ffb55243700 1 --2- 172.21.15.138:0/3412331336 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ffb50102010 0x7ffb501998b0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7ffb4c006010 tx=0x7ffb4c004cc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T21:12:13.521 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:12:13.518+0000 7ffb467fc700 1 -- 172.21.15.138:0/3412331336 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ffb4c026070 con 0x7ffb50102010 2024-05-03T21:12:13.521 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:12:13.518+0000 7ffb574a7700 1 -- 172.21.15.138:0/3412331336 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffb5019a0e0 con 0x7ffb50102010 2024-05-03T21:12:13.521 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:12:13.518+0000 7ffb467fc700 1 -- 172.21.15.138:0/3412331336 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(28 keys) ==== 1170+0+0 (secure 0 0 0) 0x7ffb4c0044a0 con 0x7ffb50102010 2024-05-03T21:12:13.521 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:12:13.518+0000 7ffb467fc700 1 -- 172.21.15.138:0/3412331336 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ffb4c003a70 con 0x7ffb50102010 2024-05-03T21:12:13.521 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:12:13.518+0000 7ffb574a7700 1 -- 172.21.15.138:0/3412331336 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffb5019a5a0 con 0x7ffb50102010 2024-05-03T21:12:13.522 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:12:13.519+0000 7ffb574a7700 1 -- 172.21.15.138:0/3412331336 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ffb501034f0 con 0x7ffb50102010 2024-05-03T21:12:13.522 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:12:13.519+0000 7ffb467fc700 1 -- 172.21.15.138:0/3412331336 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 13) ==== 50278+0+0 (secure 0 0 0) 0x7ffb4c029070 con 0x7ffb50102010 2024-05-03T21:12:13.522 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:12:13.519+0000 7ffb467fc700 1 --2- 172.21.15.138:0/3412331336 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7ffb3c03e510 0x7ffb3c0409d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:12:13.522 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:12:13.519+0000 7ffb467fc700 1 -- 172.21.15.138:0/3412331336 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7ffb4c01e070 con 0x7ffb50102010 2024-05-03T21:12:13.522 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:12:13.520+0000 7ffb54a42700 1 --2- 172.21.15.138:0/3412331336 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7ffb3c03e510 0x7ffb3c0409d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:12:13.523 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:12:13.520+0000 7ffb54a42700 1 --2- 172.21.15.138:0/3412331336 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7ffb3c03e510 0x7ffb3c0409d0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7ffb40006fd0 tx=0x7ffb40006e40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T21:12:13.526 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:12:13.525+0000 7ffb467fc700 1 -- 172.21.15.138:0/3412331336 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191839 (secure 0 0 0) 0x7ffb4c013070 con 0x7ffb50102010 2024-05-03T21:12:13.740 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:12:13.738+0000 7ffb574a7700 1 -- 172.21.15.138:0/3412331336 --> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi067:172.21.15.67=a;smithi082:172.21.15.82=b;smithi138:172.21.15.138=c", "target": ["mon-mgr", ""]}) -- 0x7ffb5006b8d0 con 0x7ffb3c03e510 2024-05-03T21:12:13.747 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:12:13.745+0000 7ffb467fc700 1 -- 172.21.15.138:0/3412331336 <== mgr.14150 v2:172.21.15.67:6800/604451592 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7ffb5006b8d0 con 0x7ffb3c03e510 2024-05-03T21:12:13.747 INFO:teuthology.orchestra.run.smithi138.stdout:Scheduled mon update... 2024-05-03T21:12:13.749 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:12:13.747+0000 7ffb574a7700 1 -- 172.21.15.138:0/3412331336 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7ffb3c03e510 msgr2=0x7ffb3c0409d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:12:13.749 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:12:13.747+0000 7ffb574a7700 1 --2- 172.21.15.138:0/3412331336 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7ffb3c03e510 0x7ffb3c0409d0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7ffb40006fd0 tx=0x7ffb40006e40 comp rx=0 tx=0).stop 2024-05-03T21:12:13.749 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:12:13.747+0000 7ffb574a7700 1 -- 172.21.15.138:0/3412331336 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ffb50102010 msgr2=0x7ffb501998b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:12:13.749 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:12:13.747+0000 7ffb574a7700 1 --2- 172.21.15.138:0/3412331336 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ffb50102010 0x7ffb501998b0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7ffb4c006010 tx=0x7ffb4c004cc0 comp rx=0 tx=0).stop 2024-05-03T21:12:13.749 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:12:13.747+0000 7ffb574a7700 1 -- 172.21.15.138:0/3412331336 shutdown_connections 2024-05-03T21:12:13.749 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:12:13.747+0000 7ffb574a7700 1 --2- 172.21.15.138:0/3412331336 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7ffb3c03e510 0x7ffb3c0409d0 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:12:13.749 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:12:13.747+0000 7ffb574a7700 1 --2- 172.21.15.138:0/3412331336 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ffb50102010 0x7ffb501998b0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:12:13.749 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:12:13.747+0000 7ffb574a7700 1 -- 172.21.15.138:0/3412331336 >> 172.21.15.138:0/3412331336 conn(0x7ffb500fd8a0 msgr2=0x7ffb501039e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T21:12:13.750 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:12:13.747+0000 7ffb574a7700 1 -- 172.21.15.138:0/3412331336 shutdown_connections 2024-05-03T21:12:13.750 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:12:13.747+0000 7ffb574a7700 1 -- 172.21.15.138:0/3412331336 wait complete. 2024-05-03T21:12:14.328 DEBUG:teuthology.orchestra.run.smithi082:mon.b> sudo journalctl -f -n 0 -u ceph-713bc57a-0991-11ef-bc97-c7b262605968@mon.b.service 2024-05-03T21:12:14.332 DEBUG:teuthology.orchestra.run.smithi138:mon.c> sudo journalctl -f -n 0 -u ceph-713bc57a-0991-11ef-bc97-c7b262605968@mon.c.service 2024-05-03T21:12:14.335 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-05-03T21:12:14.336 DEBUG:teuthology.orchestra.run.smithi138:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9a047fe7599a549042d06a3fb5b6813a203eabea shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 713bc57a-0991-11ef-bc97-c7b262605968 -- ceph mon dump -f json 2024-05-03T21:12:15.221 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:12:14 smithi067 ceph-mon[29730]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-03T21:12:15.221 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:12:14 smithi067 ceph-mon[29730]: from='client.14186 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi067:172.21.15.67=a;smithi082:172.21.15.82=b;smithi138:172.21.15.138=c", "target": ["mon-mgr", ""]}]: dispatch 2024-05-03T21:12:15.221 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:12:14 smithi067 ceph-mon[29730]: Saving service mon spec with placement smithi067:172.21.15.67=a;smithi082:172.21.15.82=b;smithi138:172.21.15.138=c;count:3 2024-05-03T21:12:15.221 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:12:14 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:12:15.221 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:12:14 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T21:12:15.221 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:12:14 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T21:12:15.222 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:12:14 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T21:12:15.222 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:12:14 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:12:15.222 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:12:14 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-05-03T21:12:15.222 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:12:14 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T21:12:15.222 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:12:14 smithi067 ceph-mon[29730]: Deploying daemon mon.c on smithi138 2024-05-03T21:12:15.405 INFO:teuthology.orchestra.run.smithi138.stderr:Inferring config /var/lib/ceph/713bc57a-0991-11ef-bc97-c7b262605968/config/ceph.conf 2024-05-03T21:12:17.221 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:12:16 smithi067 ceph-mon[29730]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-03T21:12:18.011 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:12:18.007+0000 7f1c731e4700 1 -- 172.21.15.138:0/3296933406 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1c6c101e60 msgr2=0x7f1c6c102240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:12:18.011 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:12:18.007+0000 7f1c731e4700 1 --2- 172.21.15.138:0/3296933406 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1c6c101e60 0x7f1c6c102240 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f1c60009910 tx=0x7f1c60009c20 comp rx=0 tx=0).stop 2024-05-03T21:12:18.011 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:12:18.007+0000 7f1c731e4700 1 -- 172.21.15.138:0/3296933406 shutdown_connections 2024-05-03T21:12:18.011 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:12:18.007+0000 7f1c731e4700 1 --2- 172.21.15.138:0/3296933406 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1c6c101e60 0x7f1c6c102240 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:12:18.011 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:12:18.007+0000 7f1c731e4700 1 -- 172.21.15.138:0/3296933406 >> 172.21.15.138:0/3296933406 conn(0x7f1c6c0fd8a0 msgr2=0x7f1c6c0ffcc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T21:12:18.011 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:12:18.007+0000 7f1c731e4700 1 -- 172.21.15.138:0/3296933406 shutdown_connections 2024-05-03T21:12:18.011 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:12:18.007+0000 7f1c731e4700 1 -- 172.21.15.138:0/3296933406 wait complete. 2024-05-03T21:12:18.011 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:12:18.007+0000 7f1c731e4700 1 Processor -- start 2024-05-03T21:12:18.011 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:12:18.008+0000 7f1c731e4700 1 -- start start 2024-05-03T21:12:18.011 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:12:18.008+0000 7f1c731e4700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1c6c101e60 0x7f1c6c19f0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:12:18.011 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:12:18.008+0000 7f1c731e4700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f1c6c113ca0 con 0x7f1c6c101e60 2024-05-03T21:12:18.012 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:12:18.008+0000 7f1c70f80700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1c6c101e60 0x7f1c6c19f0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:12:18.012 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:12:18.008+0000 7f1c70f80700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1c6c101e60 0x7f1c6c19f0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.138:44330/0 (socket says 172.21.15.138:44330) 2024-05-03T21:12:18.012 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:12:18.008+0000 7f1c70f80700 1 -- 172.21.15.138:0/492594802 learned_addr learned my addr 172.21.15.138:0/492594802 (peer_addr_for_me v2:172.21.15.138:0/0) 2024-05-03T21:12:18.012 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:12:18.008+0000 7f1c70f80700 1 -- 172.21.15.138:0/492594802 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1c6c19f5e0 con 0x7f1c6c101e60 2024-05-03T21:12:18.012 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:12:18.009+0000 7f1c70f80700 1 --2- 172.21.15.138:0/492594802 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1c6c101e60 0x7f1c6c19f0a0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f1c60004c10 tx=0x7f1c60004cf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T21:12:18.012 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:12:18.009+0000 7f1c69ffb700 1 -- 172.21.15.138:0/492594802 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1c60026070 con 0x7f1c6c101e60 2024-05-03T21:12:18.012 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:12:18.009+0000 7f1c731e4700 1 -- 172.21.15.138:0/492594802 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1c6c19f870 con 0x7f1c6c101e60 2024-05-03T21:12:18.012 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:12:18.009+0000 7f1c731e4700 1 -- 172.21.15.138:0/492594802 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1c6c19fcd0 con 0x7f1c6c101e60 2024-05-03T21:12:18.012 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:12:18.009+0000 7f1c69ffb700 1 -- 172.21.15.138:0/492594802 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(28 keys) ==== 1170+0+0 (secure 0 0 0) 0x7f1c60004370 con 0x7f1c6c101e60 2024-05-03T21:12:18.012 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:12:18.009+0000 7f1c69ffb700 1 -- 172.21.15.138:0/492594802 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1c60003b80 con 0x7f1c6c101e60 2024-05-03T21:12:18.013 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:12:18.010+0000 7f1c731e4700 1 -- 172.21.15.138:0/492594802 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1c50005330 con 0x7f1c6c101e60 2024-05-03T21:12:18.013 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:12:18.010+0000 7f1c69ffb700 1 -- 172.21.15.138:0/492594802 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 13) ==== 50278+0+0 (secure 0 0 0) 0x7f1c60029070 con 0x7f1c6c101e60 2024-05-03T21:12:18.014 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:12:18.010+0000 7f1c69ffb700 1 --2- 172.21.15.138:0/492594802 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f1c5c03e510 0x7f1c5c0409d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:12:18.014 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:12:18.010+0000 7f1c69ffb700 1 -- 172.21.15.138:0/492594802 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f1c6001e070 con 0x7f1c6c101e60 2024-05-03T21:12:18.014 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:12:18.010+0000 7f1c6bfff700 1 --2- 172.21.15.138:0/492594802 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f1c5c03e510 0x7f1c5c0409d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:12:18.014 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:12:18.011+0000 7f1c6bfff700 1 --2- 172.21.15.138:0/492594802 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f1c5c03e510 0x7f1c5c0409d0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f1c58006fd0 tx=0x7f1c58006e40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T21:12:18.017 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:12:18.016+0000 7f1c69ffb700 1 -- 172.21.15.138:0/492594802 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191839 (secure 0 0 0) 0x7f1c60013070 con 0x7f1c6c101e60 2024-05-03T21:12:18.331 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:12:18.329+0000 7f1c731e4700 1 -- 172.21.15.138:0/492594802 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) -- 0x7f1c50005620 con 0x7f1c6c101e60 2024-05-03T21:12:18.332 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:12:18.330+0000 7f1c69ffb700 1 -- 172.21.15.138:0/492594802 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) ==== 95+0+745 (secure 0 0 0) 0x7f1c6001f310 con 0x7f1c6c101e60 2024-05-03T21:12:18.332 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-05-03T21:12:18.332 INFO:teuthology.orchestra.run.smithi138.stdout:{"epoch":1,"fsid":"713bc57a-0991-11ef-bc97-c7b262605968","modified":"2024-05-03T21:10:31.243292Z","created":"2024-05-03T21:10:31.243292Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:3300","nonce":0},{"type":"v1","addr":"172.21.15.67:6789","nonce":0}]},"addr":"172.21.15.67:6789/0","public_addr":"172.21.15.67:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-05-03T21:12:18.333 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:12:18.332+0000 7f1c731e4700 1 -- 172.21.15.138:0/492594802 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f1c5c03e510 msgr2=0x7f1c5c0409d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:12:18.334 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:12:18.332+0000 7f1c731e4700 1 --2- 172.21.15.138:0/492594802 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f1c5c03e510 0x7f1c5c0409d0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f1c58006fd0 tx=0x7f1c58006e40 comp rx=0 tx=0).stop 2024-05-03T21:12:18.334 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:12:18.332+0000 7f1c731e4700 1 -- 172.21.15.138:0/492594802 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1c6c101e60 msgr2=0x7f1c6c19f0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:12:18.334 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:12:18.332+0000 7f1c731e4700 1 --2- 172.21.15.138:0/492594802 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1c6c101e60 0x7f1c6c19f0a0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f1c60004c10 tx=0x7f1c60004cf0 comp rx=0 tx=0).stop 2024-05-03T21:12:18.334 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:12:18.332+0000 7f1c731e4700 1 -- 172.21.15.138:0/492594802 shutdown_connections 2024-05-03T21:12:18.334 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:12:18.332+0000 7f1c731e4700 1 --2- 172.21.15.138:0/492594802 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f1c5c03e510 0x7f1c5c0409d0 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:12:18.334 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:12:18.332+0000 7f1c731e4700 1 --2- 172.21.15.138:0/492594802 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1c6c101e60 0x7f1c6c19f0a0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:12:18.334 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:12:18.332+0000 7f1c731e4700 1 -- 172.21.15.138:0/492594802 >> 172.21.15.138:0/492594802 conn(0x7f1c6c0fd8a0 msgr2=0x7f1c6c0ff000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T21:12:18.334 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:12:18.332+0000 7f1c731e4700 1 -- 172.21.15.138:0/492594802 shutdown_connections 2024-05-03T21:12:18.334 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:12:18.332+0000 7f1c731e4700 1 -- 172.21.15.138:0/492594802 wait complete. 2024-05-03T21:12:18.335 INFO:teuthology.orchestra.run.smithi138.stderr:dumped monmap epoch 1 2024-05-03T21:12:18.574 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:18 smithi138 systemd[1]: Starting Ceph mon.c for 713bc57a-0991-11ef-bc97-c7b262605968... 2024-05-03T21:12:19.221 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:12:18 smithi067 ceph-mon[29730]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-03T21:12:19.221 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:12:18 smithi067 ceph-mon[29730]: from='client.? 172.21.15.138:0/492594802' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-05-03T21:12:19.284 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:19 smithi138 podman[33090]: 2024-05-03 21:12:18.934913202 +0000 UTC m=+0.019381180 image pull ee4d846202496ba6096593e8e8becc0700a5eb7ebd95536031187f5f337d4012 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:68e15ae152ef5c805c733e76da8fc1d3dd4ee669d5ba025f457e05725baa24dd 2024-05-03T21:12:19.284 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:19 smithi138 podman[33090]: 2024-05-03 21:12:19.034866446 +0000 UTC m=+0.119334396 container create b833bba2b5cf358958269b1691b789d1894d99c30b42d1dbccfc1a54ea56a1fb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:68e15ae152ef5c805c733e76da8fc1d3dd4ee669d5ba025f457e05725baa24dd, name=ceph-713bc57a-0991-11ef-bc97-c7b262605968-mon-c, org.label-schema.build-date=20240430, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, ceph=True, GIT_CLEAN=True, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.vendor=CentOS, RELEASE=wip-rishabh-testing-20240501.193033-9a047fe) 2024-05-03T21:12:19.284 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:19 smithi138 podman[33090]: 2024-05-03 21:12:19.221440324 +0000 UTC m=+0.305908285 container init b833bba2b5cf358958269b1691b789d1894d99c30b42d1dbccfc1a54ea56a1fb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:68e15ae152ef5c805c733e76da8fc1d3dd4ee669d5ba025f457e05725baa24dd, name=ceph-713bc57a-0991-11ef-bc97-c7b262605968-mon-c, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240430, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.schema-version=1.0, ceph=True, RELEASE=wip-rishabh-testing-20240501.193033-9a047fe, maintainer=Guillaume Abrioux ) 2024-05-03T21:12:19.284 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:19 smithi138 podman[33090]: 2024-05-03 21:12:19.224631089 +0000 UTC m=+0.309099039 container start b833bba2b5cf358958269b1691b789d1894d99c30b42d1dbccfc1a54ea56a1fb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:68e15ae152ef5c805c733e76da8fc1d3dd4ee669d5ba025f457e05725baa24dd, name=ceph-713bc57a-0991-11ef-bc97-c7b262605968-mon-c, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240430, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, RELEASE=wip-rishabh-testing-20240501.193033-9a047fe) 2024-05-03T21:12:19.284 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:19 smithi138 ceph-mon[33110]: set uid:gid to 167:167 (ceph:ceph) 2024-05-03T21:12:19.284 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:19 smithi138 ceph-mon[33110]: ceph version 19.0.0-3450-g9a047fe7 (9a047fe7599a549042d06a3fb5b6813a203eabea) squid (dev), process ceph-mon, pid 2 2024-05-03T21:12:19.285 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:19 smithi138 ceph-mon[33110]: pidfile_write: ignore empty --pid-file 2024-05-03T21:12:19.285 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:19 smithi138 ceph-mon[33110]: load: jerasure load: lrc 2024-05-03T21:12:19.285 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:19 smithi138 ceph-mon[33110]: rocksdb: RocksDB version: 7.9.2 2024-05-03T21:12:19.285 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:19 smithi138 ceph-mon[33110]: rocksdb: Git sha 0 2024-05-03T21:12:19.285 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:19 smithi138 ceph-mon[33110]: rocksdb: Compile date 2024-05-01 21:12:03 2024-05-03T21:12:19.285 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:19 smithi138 ceph-mon[33110]: rocksdb: DB SUMMARY 2024-05-03T21:12:19.285 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:19 smithi138 ceph-mon[33110]: rocksdb: DB Session ID: 0HNDXC0MG2ZP1SQYM610 2024-05-03T21:12:19.285 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:19 smithi138 ceph-mon[33110]: rocksdb: CURRENT file: CURRENT 2024-05-03T21:12:19.285 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:19 smithi138 ceph-mon[33110]: rocksdb: IDENTITY file: IDENTITY 2024-05-03T21:12:19.285 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:19 smithi138 ceph-mon[33110]: rocksdb: MANIFEST file: MANIFEST-000005 size: 59 Bytes 2024-05-03T21:12:19.285 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:19 smithi138 ceph-mon[33110]: rocksdb: SST files in /var/lib/ceph/mon/ceph-c/store.db dir, Total Num: 0, files: 2024-05-03T21:12:19.286 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:19 smithi138 ceph-mon[33110]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-c/store.db: 000004.log size: 511 ; 2024-05-03T21:12:19.286 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:19 smithi138 ceph-mon[33110]: rocksdb: Options.error_if_exists: 0 2024-05-03T21:12:19.286 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:19 smithi138 ceph-mon[33110]: rocksdb: Options.create_if_missing: 0 2024-05-03T21:12:19.287 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:19 smithi138 ceph-mon[33110]: rocksdb: Options.paranoid_checks: 1 2024-05-03T21:12:19.287 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:19 smithi138 ceph-mon[33110]: rocksdb: Options.flush_verify_memtable_count: 1 2024-05-03T21:12:19.287 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:19 smithi138 ceph-mon[33110]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-05-03T21:12:19.287 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:19 smithi138 ceph-mon[33110]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-05-03T21:12:19.287 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:19 smithi138 ceph-mon[33110]: rocksdb: Options.env: 0x564afa1a5cc0 2024-05-03T21:12:19.287 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:19 smithi138 ceph-mon[33110]: rocksdb: Options.fs: PosixFileSystem 2024-05-03T21:12:19.287 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:19 smithi138 ceph-mon[33110]: rocksdb: Options.info_log: 0x564afb5c9da0 2024-05-03T21:12:19.287 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:19 smithi138 ceph-mon[33110]: rocksdb: Options.max_file_opening_threads: 16 2024-05-03T21:12:19.287 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:19 smithi138 ceph-mon[33110]: rocksdb: Options.statistics: (nil) 2024-05-03T21:12:19.287 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:19 smithi138 ceph-mon[33110]: rocksdb: Options.use_fsync: 0 2024-05-03T21:12:19.287 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:19 smithi138 ceph-mon[33110]: rocksdb: Options.max_log_file_size: 0 2024-05-03T21:12:19.287 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:19 smithi138 ceph-mon[33110]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-05-03T21:12:19.288 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:19 smithi138 ceph-mon[33110]: rocksdb: Options.log_file_time_to_roll: 0 2024-05-03T21:12:19.288 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:19 smithi138 ceph-mon[33110]: rocksdb: Options.keep_log_file_num: 1000 2024-05-03T21:12:19.288 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:19 smithi138 ceph-mon[33110]: rocksdb: Options.recycle_log_file_num: 0 2024-05-03T21:12:19.288 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:19 smithi138 ceph-mon[33110]: rocksdb: Options.allow_fallocate: 1 2024-05-03T21:12:19.288 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:19 smithi138 ceph-mon[33110]: rocksdb: Options.allow_mmap_reads: 0 2024-05-03T21:12:19.288 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:19 smithi138 ceph-mon[33110]: rocksdb: Options.allow_mmap_writes: 0 2024-05-03T21:12:19.288 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:19 smithi138 ceph-mon[33110]: rocksdb: Options.use_direct_reads: 0 2024-05-03T21:12:19.288 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:19 smithi138 ceph-mon[33110]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-05-03T21:12:19.288 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:19 smithi138 ceph-mon[33110]: rocksdb: Options.create_missing_column_families: 0 2024-05-03T21:12:19.288 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:19 smithi138 ceph-mon[33110]: rocksdb: Options.db_log_dir: 2024-05-03T21:12:19.288 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:19 smithi138 ceph-mon[33110]: rocksdb: Options.wal_dir: 2024-05-03T21:12:19.288 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:19 smithi138 ceph-mon[33110]: rocksdb: Options.table_cache_numshardbits: 6 2024-05-03T21:12:19.288 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:19 smithi138 ceph-mon[33110]: rocksdb: Options.WAL_ttl_seconds: 0 2024-05-03T21:12:19.288 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:19 smithi138 ceph-mon[33110]: rocksdb: Options.WAL_size_limit_MB: 0 2024-05-03T21:12:19.289 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:19 smithi138 ceph-mon[33110]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-05-03T21:12:19.289 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:19 smithi138 ceph-mon[33110]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-05-03T21:12:19.289 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:19 smithi138 ceph-mon[33110]: rocksdb: Options.is_fd_close_on_exec: 1 2024-05-03T21:12:19.289 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:19 smithi138 ceph-mon[33110]: rocksdb: Options.advise_random_on_open: 1 2024-05-03T21:12:19.289 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:19 smithi138 ceph-mon[33110]: rocksdb: Options.db_write_buffer_size: 0 2024-05-03T21:12:19.289 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:19 smithi138 ceph-mon[33110]: rocksdb: Options.write_buffer_manager: 0x564afb59b0e0 2024-05-03T21:12:19.289 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:19 smithi138 ceph-mon[33110]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-05-03T21:12:19.289 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:19 smithi138 ceph-mon[33110]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-05-03T21:12:19.289 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:19 smithi138 ceph-mon[33110]: rocksdb: Options.use_adaptive_mutex: 0 2024-05-03T21:12:19.289 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:19 smithi138 ceph-mon[33110]: rocksdb: Options.rate_limiter: (nil) 2024-05-03T21:12:19.289 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:19 smithi138 ceph-mon[33110]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-05-03T21:12:19.289 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:19 smithi138 ceph-mon[33110]: rocksdb: Options.wal_recovery_mode: 2 2024-05-03T21:12:19.290 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:19 smithi138 ceph-mon[33110]: rocksdb: Options.enable_thread_tracking: 0 2024-05-03T21:12:19.290 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:19 smithi138 ceph-mon[33110]: rocksdb: Options.enable_pipelined_write: 0 2024-05-03T21:12:19.290 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:19 smithi138 ceph-mon[33110]: rocksdb: Options.unordered_write: 0 2024-05-03T21:12:19.290 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:19 smithi138 ceph-mon[33110]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-05-03T21:12:19.290 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:19 smithi138 ceph-mon[33110]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-05-03T21:12:19.290 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:19 smithi138 ceph-mon[33110]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-05-03T21:12:19.290 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:19 smithi138 ceph-mon[33110]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-05-03T21:12:19.290 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:19 smithi138 ceph-mon[33110]: rocksdb: Options.row_cache: None 2024-05-03T21:12:19.290 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:19 smithi138 ceph-mon[33110]: rocksdb: Options.wal_filter: None 2024-05-03T21:12:19.290 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:19 smithi138 ceph-mon[33110]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-05-03T21:12:19.290 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:19 smithi138 ceph-mon[33110]: rocksdb: Options.allow_ingest_behind: 0 2024-05-03T21:12:19.290 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:19 smithi138 ceph-mon[33110]: rocksdb: Options.two_write_queues: 0 2024-05-03T21:12:19.290 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:19 smithi138 ceph-mon[33110]: rocksdb: Options.manual_wal_flush: 0 2024-05-03T21:12:19.290 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:19 smithi138 ceph-mon[33110]: rocksdb: Options.wal_compression: 0 2024-05-03T21:12:19.291 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:19 smithi138 ceph-mon[33110]: rocksdb: Options.atomic_flush: 0 2024-05-03T21:12:19.291 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:19 smithi138 ceph-mon[33110]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-05-03T21:12:19.291 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:19 smithi138 ceph-mon[33110]: rocksdb: Options.persist_stats_to_disk: 0 2024-05-03T21:12:19.291 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:19 smithi138 ceph-mon[33110]: rocksdb: Options.write_dbid_to_manifest: 0 2024-05-03T21:12:19.291 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:19 smithi138 ceph-mon[33110]: rocksdb: Options.log_readahead_size: 0 2024-05-03T21:12:19.291 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:19 smithi138 ceph-mon[33110]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-05-03T21:12:19.291 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:19 smithi138 ceph-mon[33110]: rocksdb: Options.best_efforts_recovery: 0 2024-05-03T21:12:19.291 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:19 smithi138 ceph-mon[33110]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-05-03T21:12:19.291 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:19 smithi138 ceph-mon[33110]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-05-03T21:12:19.291 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:19 smithi138 ceph-mon[33110]: rocksdb: Options.allow_data_in_errors: 0 2024-05-03T21:12:19.291 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:19 smithi138 ceph-mon[33110]: rocksdb: Options.db_host_id: __hostname__ 2024-05-03T21:12:19.291 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:19 smithi138 ceph-mon[33110]: rocksdb: Options.enforce_single_del_contracts: true 2024-05-03T21:12:19.291 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:19 smithi138 ceph-mon[33110]: rocksdb: Options.max_background_jobs: 2 2024-05-03T21:12:19.291 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:19 smithi138 ceph-mon[33110]: rocksdb: Options.max_background_compactions: -1 2024-05-03T21:12:19.292 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:19 smithi138 ceph-mon[33110]: rocksdb: Options.max_subcompactions: 1 2024-05-03T21:12:19.292 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:19 smithi138 ceph-mon[33110]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-05-03T21:12:19.292 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:19 smithi138 ceph-mon[33110]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-05-03T21:12:19.292 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:19 smithi138 ceph-mon[33110]: rocksdb: Options.delayed_write_rate : 16777216 2024-05-03T21:12:19.292 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:19 smithi138 ceph-mon[33110]: rocksdb: Options.max_total_wal_size: 0 2024-05-03T21:12:19.292 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:19 smithi138 ceph-mon[33110]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-05-03T21:12:19.292 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:19 smithi138 ceph-mon[33110]: rocksdb: Options.stats_dump_period_sec: 600 2024-05-03T21:12:19.292 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:19 smithi138 ceph-mon[33110]: rocksdb: Options.stats_persist_period_sec: 600 2024-05-03T21:12:19.292 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:19 smithi138 ceph-mon[33110]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-05-03T21:12:19.292 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:19 smithi138 ceph-mon[33110]: rocksdb: Options.max_open_files: -1 2024-05-03T21:12:19.292 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:19 smithi138 ceph-mon[33110]: rocksdb: Options.bytes_per_sync: 0 2024-05-03T21:12:19.292 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:19 smithi138 ceph-mon[33110]: rocksdb: Options.wal_bytes_per_sync: 0 2024-05-03T21:12:19.292 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:19 smithi138 ceph-mon[33110]: rocksdb: Options.strict_bytes_per_sync: 0 2024-05-03T21:12:19.293 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:19 smithi138 ceph-mon[33110]: rocksdb: Options.compaction_readahead_size: 0 2024-05-03T21:12:19.293 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:19 smithi138 ceph-mon[33110]: rocksdb: Options.max_background_flushes: -1 2024-05-03T21:12:19.293 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:19 smithi138 ceph-mon[33110]: rocksdb: Compression algorithms supported: 2024-05-03T21:12:19.293 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:19 smithi138 ceph-mon[33110]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-05-03T21:12:19.293 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:19 smithi138 ceph-mon[33110]: rocksdb: kZSTD supported: 0 2024-05-03T21:12:19.293 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:19 smithi138 ceph-mon[33110]: rocksdb: kXpressCompression supported: 0 2024-05-03T21:12:19.293 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:19 smithi138 ceph-mon[33110]: rocksdb: kLZ4HCCompression supported: 1 2024-05-03T21:12:19.293 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:19 smithi138 ceph-mon[33110]: rocksdb: kZlibCompression supported: 1 2024-05-03T21:12:19.293 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:19 smithi138 ceph-mon[33110]: rocksdb: kSnappyCompression supported: 1 2024-05-03T21:12:19.293 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:19 smithi138 ceph-mon[33110]: rocksdb: kLZ4Compression supported: 1 2024-05-03T21:12:19.293 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:19 smithi138 ceph-mon[33110]: rocksdb: kBZip2Compression supported: 0 2024-05-03T21:12:19.293 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:19 smithi138 ceph-mon[33110]: rocksdb: Fast CRC32 supported: Supported on x86 2024-05-03T21:12:19.293 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:19 smithi138 ceph-mon[33110]: rocksdb: DMutex implementation: pthread_mutex_t 2024-05-03T21:12:19.293 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:19 smithi138 ceph-mon[33110]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000005 2024-05-03T21:12:19.294 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:19 smithi138 ceph-mon[33110]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-05-03T21:12:19.294 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:19 smithi138 ceph-mon[33110]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-05-03T21:12:19.294 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:19 smithi138 ceph-mon[33110]: rocksdb: Options.merge_operator: 2024-05-03T21:12:19.294 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:19 smithi138 ceph-mon[33110]: rocksdb: Options.compaction_filter: None 2024-05-03T21:12:19.294 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:19 smithi138 ceph-mon[33110]: rocksdb: Options.compaction_filter_factory: None 2024-05-03T21:12:19.294 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:19 smithi138 ceph-mon[33110]: rocksdb: Options.sst_partitioner_factory: None 2024-05-03T21:12:19.294 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:19 smithi138 ceph-mon[33110]: rocksdb: Options.memtable_factory: SkipListFactory 2024-05-03T21:12:19.294 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:19 smithi138 ceph-mon[33110]: rocksdb: Options.table_factory: BlockBasedTable 2024-05-03T21:12:19.294 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:19 smithi138 ceph-mon[33110]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x564afb5c9f60) 2024-05-03T21:12:19.294 INFO:journalctl@ceph.mon.c.smithi138.stdout: cache_index_and_filter_blocks: 1 2024-05-03T21:12:19.294 INFO:journalctl@ceph.mon.c.smithi138.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-05-03T21:12:19.294 INFO:journalctl@ceph.mon.c.smithi138.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-05-03T21:12:19.294 INFO:journalctl@ceph.mon.c.smithi138.stdout: pin_top_level_index_and_filter: 1 2024-05-03T21:12:19.295 INFO:journalctl@ceph.mon.c.smithi138.stdout: index_type: 0 2024-05-03T21:12:19.295 INFO:journalctl@ceph.mon.c.smithi138.stdout: data_block_index_type: 0 2024-05-03T21:12:19.295 INFO:journalctl@ceph.mon.c.smithi138.stdout: index_shortening: 1 2024-05-03T21:12:19.295 INFO:journalctl@ceph.mon.c.smithi138.stdout: data_block_hash_table_util_ratio: 0.750000 2024-05-03T21:12:19.295 INFO:journalctl@ceph.mon.c.smithi138.stdout: checksum: 4 2024-05-03T21:12:19.296 INFO:journalctl@ceph.mon.c.smithi138.stdout: no_block_cache: 0 2024-05-03T21:12:19.296 INFO:journalctl@ceph.mon.c.smithi138.stdout: block_cache: 0x564afb5cd1f0 2024-05-03T21:12:19.296 INFO:journalctl@ceph.mon.c.smithi138.stdout: block_cache_name: BinnedLRUCache 2024-05-03T21:12:19.296 INFO:journalctl@ceph.mon.c.smithi138.stdout: block_cache_options: 2024-05-03T21:12:19.296 INFO:journalctl@ceph.mon.c.smithi138.stdout: capacity : 536870912 2024-05-03T21:12:19.296 INFO:journalctl@ceph.mon.c.smithi138.stdout: num_shard_bits : 4 2024-05-03T21:12:19.296 INFO:journalctl@ceph.mon.c.smithi138.stdout: strict_capacity_limit : 0 2024-05-03T21:12:19.296 INFO:journalctl@ceph.mon.c.smithi138.stdout: high_pri_pool_ratio: 0.000 2024-05-03T21:12:19.296 INFO:journalctl@ceph.mon.c.smithi138.stdout: block_cache_compressed: (nil) 2024-05-03T21:12:19.296 INFO:journalctl@ceph.mon.c.smithi138.stdout: persistent_cache: (nil) 2024-05-03T21:12:19.296 INFO:journalctl@ceph.mon.c.smithi138.stdout: block_size: 4096 2024-05-03T21:12:19.298 INFO:journalctl@ceph.mon.c.smithi138.stdout: block_size_deviation: 10 2024-05-03T21:12:19.298 INFO:journalctl@ceph.mon.c.smithi138.stdout: block_restart_interval: 16 2024-05-03T21:12:19.298 INFO:journalctl@ceph.mon.c.smithi138.stdout: index_block_restart_interval: 1 2024-05-03T21:12:19.298 INFO:journalctl@ceph.mon.c.smithi138.stdout: metadata_block_size: 4096 2024-05-03T21:12:19.298 INFO:journalctl@ceph.mon.c.smithi138.stdout: partition_filters: 0 2024-05-03T21:12:19.298 INFO:journalctl@ceph.mon.c.smithi138.stdout: use_delta_encoding: 1 2024-05-03T21:12:19.298 INFO:journalctl@ceph.mon.c.smithi138.stdout: filter_policy: bloomfilter 2024-05-03T21:12:19.298 INFO:journalctl@ceph.mon.c.smithi138.stdout: whole_key_filtering: 1 2024-05-03T21:12:19.299 INFO:journalctl@ceph.mon.c.smithi138.stdout: verify_compression: 0 2024-05-03T21:12:19.299 INFO:journalctl@ceph.mon.c.smithi138.stdout: read_amp_bytes_per_bit: 0 2024-05-03T21:12:19.299 INFO:journalctl@ceph.mon.c.smithi138.stdout: format_version: 5 2024-05-03T21:12:19.299 INFO:journalctl@ceph.mon.c.smithi138.stdout: enable_index_compression: 1 2024-05-03T21:12:19.299 INFO:journalctl@ceph.mon.c.smithi138.stdout: block_align: 0 2024-05-03T21:12:19.299 INFO:journalctl@ceph.mon.c.smithi138.stdout: max_auto_readahead_size: 262144 2024-05-03T21:12:19.299 INFO:journalctl@ceph.mon.c.smithi138.stdout: prepopulate_block_cache: 0 2024-05-03T21:12:19.299 INFO:journalctl@ceph.mon.c.smithi138.stdout: initial_auto_readahead_size: 8192 2024-05-03T21:12:19.299 INFO:journalctl@ceph.mon.c.smithi138.stdout: num_file_reads_for_auto_readahead: 2 2024-05-03T21:12:19.299 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:19 smithi138 ceph-mon[33110]: rocksdb: Options.write_buffer_size: 33554432 2024-05-03T21:12:19.299 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:19 smithi138 ceph-mon[33110]: rocksdb: Options.max_write_buffer_number: 2 2024-05-03T21:12:19.299 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:19 smithi138 ceph-mon[33110]: rocksdb: Options.compression: NoCompression 2024-05-03T21:12:19.299 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:19 smithi138 ceph-mon[33110]: rocksdb: Options.bottommost_compression: Disabled 2024-05-03T21:12:19.299 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:19 smithi138 ceph-mon[33110]: rocksdb: Options.prefix_extractor: nullptr 2024-05-03T21:12:19.300 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:19 smithi138 ceph-mon[33110]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-05-03T21:12:19.300 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:19 smithi138 ceph-mon[33110]: rocksdb: Options.num_levels: 7 2024-05-03T21:12:19.300 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:19 smithi138 ceph-mon[33110]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-05-03T21:12:19.300 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:19 smithi138 ceph-mon[33110]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-05-03T21:12:19.300 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:19 smithi138 ceph-mon[33110]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-05-03T21:12:19.300 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:19 smithi138 ceph-mon[33110]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-05-03T21:12:19.300 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:19 smithi138 ceph-mon[33110]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-05-03T21:12:19.300 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:19 smithi138 ceph-mon[33110]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-05-03T21:12:19.300 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:19 smithi138 ceph-mon[33110]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-05-03T21:12:19.300 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:19 smithi138 ceph-mon[33110]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-05-03T21:12:19.300 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:19 smithi138 ceph-mon[33110]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-05-03T21:12:19.300 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:19 smithi138 ceph-mon[33110]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-05-03T21:12:19.300 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:19 smithi138 ceph-mon[33110]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-05-03T21:12:19.300 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:19 smithi138 ceph-mon[33110]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-05-03T21:12:19.301 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:19 smithi138 ceph-mon[33110]: rocksdb: Options.compression_opts.window_bits: -14 2024-05-03T21:12:19.301 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:19 smithi138 ceph-mon[33110]: rocksdb: Options.compression_opts.level: 32767 2024-05-03T21:12:19.301 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:19 smithi138 ceph-mon[33110]: rocksdb: Options.compression_opts.strategy: 0 2024-05-03T21:12:19.301 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:19 smithi138 ceph-mon[33110]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-05-03T21:12:19.301 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:19 smithi138 ceph-mon[33110]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-05-03T21:12:19.301 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:19 smithi138 ceph-mon[33110]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-05-03T21:12:19.301 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:19 smithi138 ceph-mon[33110]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-05-03T21:12:19.301 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:19 smithi138 ceph-mon[33110]: rocksdb: Options.compression_opts.enabled: false 2024-05-03T21:12:19.301 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:19 smithi138 ceph-mon[33110]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-05-03T21:12:19.301 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:19 smithi138 ceph-mon[33110]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-05-03T21:12:19.301 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:19 smithi138 ceph-mon[33110]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-05-03T21:12:19.301 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:19 smithi138 ceph-mon[33110]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-05-03T21:12:19.301 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:19 smithi138 ceph-mon[33110]: rocksdb: Options.target_file_size_base: 67108864 2024-05-03T21:12:19.302 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:19 smithi138 ceph-mon[33110]: rocksdb: Options.target_file_size_multiplier: 1 2024-05-03T21:12:19.302 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:19 smithi138 ceph-mon[33110]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-05-03T21:12:19.302 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:19 smithi138 ceph-mon[33110]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-05-03T21:12:19.302 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:19 smithi138 ceph-mon[33110]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-05-03T21:12:19.302 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:19 smithi138 ceph-mon[33110]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-05-03T21:12:19.302 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:19 smithi138 ceph-mon[33110]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-05-03T21:12:19.302 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:19 smithi138 ceph-mon[33110]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-05-03T21:12:19.302 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:19 smithi138 ceph-mon[33110]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-05-03T21:12:19.302 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:19 smithi138 ceph-mon[33110]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-05-03T21:12:19.302 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:19 smithi138 ceph-mon[33110]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-05-03T21:12:19.302 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:19 smithi138 ceph-mon[33110]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-05-03T21:12:19.303 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:19 smithi138 ceph-mon[33110]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-05-03T21:12:19.303 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:19 smithi138 ceph-mon[33110]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-05-03T21:12:19.303 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:19 smithi138 ceph-mon[33110]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-05-03T21:12:19.303 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:19 smithi138 ceph-mon[33110]: rocksdb: Options.arena_block_size: 1048576 2024-05-03T21:12:19.303 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:19 smithi138 ceph-mon[33110]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-05-03T21:12:19.303 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:19 smithi138 ceph-mon[33110]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-05-03T21:12:19.303 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:19 smithi138 ceph-mon[33110]: rocksdb: Options.disable_auto_compactions: 0 2024-05-03T21:12:19.303 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:19 smithi138 ceph-mon[33110]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-05-03T21:12:19.303 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:19 smithi138 ceph-mon[33110]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-05-03T21:12:19.303 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:19 smithi138 ceph-mon[33110]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-05-03T21:12:19.303 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:19 smithi138 ceph-mon[33110]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-05-03T21:12:19.303 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:19 smithi138 ceph-mon[33110]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-05-03T21:12:19.303 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:19 smithi138 ceph-mon[33110]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-05-03T21:12:19.303 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:19 smithi138 ceph-mon[33110]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-05-03T21:12:19.304 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:19 smithi138 ceph-mon[33110]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-05-03T21:12:19.304 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:19 smithi138 ceph-mon[33110]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-05-03T21:12:19.304 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:19 smithi138 ceph-mon[33110]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-05-03T21:12:19.304 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:19 smithi138 ceph-mon[33110]: rocksdb: Options.table_properties_collectors: CompactOnDeletionCollector (Sliding window size = 32768 Deletion trigger = 16384 Deletion ratio = 0); 2024-05-03T21:12:19.304 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:19 smithi138 ceph-mon[33110]: rocksdb: Options.inplace_update_support: 0 2024-05-03T21:12:19.304 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:19 smithi138 ceph-mon[33110]: rocksdb: Options.inplace_update_num_locks: 10000 2024-05-03T21:12:19.304 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:19 smithi138 ceph-mon[33110]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-05-03T21:12:19.304 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:19 smithi138 ceph-mon[33110]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-05-03T21:12:19.304 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:19 smithi138 ceph-mon[33110]: rocksdb: Options.memtable_huge_page_size: 0 2024-05-03T21:12:19.304 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:19 smithi138 ceph-mon[33110]: rocksdb: Options.bloom_locality: 0 2024-05-03T21:12:19.304 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:19 smithi138 ceph-mon[33110]: rocksdb: Options.max_successive_merges: 0 2024-05-03T21:12:19.304 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:19 smithi138 ceph-mon[33110]: rocksdb: Options.optimize_filters_for_hits: 0 2024-05-03T21:12:19.304 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:19 smithi138 ceph-mon[33110]: rocksdb: Options.paranoid_file_checks: 0 2024-05-03T21:12:19.304 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:19 smithi138 ceph-mon[33110]: rocksdb: Options.force_consistency_checks: 1 2024-05-03T21:12:19.304 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:19 smithi138 ceph-mon[33110]: rocksdb: Options.report_bg_io_stats: 0 2024-05-03T21:12:19.305 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:19 smithi138 ceph-mon[33110]: rocksdb: Options.ttl: 2592000 2024-05-03T21:12:19.305 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:19 smithi138 ceph-mon[33110]: rocksdb: Options.periodic_compaction_seconds: 0 2024-05-03T21:12:19.305 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:19 smithi138 ceph-mon[33110]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-05-03T21:12:19.305 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:19 smithi138 ceph-mon[33110]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-05-03T21:12:19.305 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:19 smithi138 ceph-mon[33110]: rocksdb: Options.enable_blob_files: false 2024-05-03T21:12:19.305 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:19 smithi138 ceph-mon[33110]: rocksdb: Options.min_blob_size: 0 2024-05-03T21:12:19.305 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:19 smithi138 ceph-mon[33110]: rocksdb: Options.blob_file_size: 268435456 2024-05-03T21:12:19.305 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:19 smithi138 ceph-mon[33110]: rocksdb: Options.blob_compression_type: NoCompression 2024-05-03T21:12:19.305 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:19 smithi138 ceph-mon[33110]: rocksdb: Options.enable_blob_garbage_collection: false 2024-05-03T21:12:19.305 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:19 smithi138 ceph-mon[33110]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-05-03T21:12:19.305 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:19 smithi138 ceph-mon[33110]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-05-03T21:12:19.305 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:19 smithi138 ceph-mon[33110]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-05-03T21:12:19.305 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:19 smithi138 ceph-mon[33110]: rocksdb: Options.blob_file_starting_level: 0 2024-05-03T21:12:19.306 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:19 smithi138 ceph-mon[33110]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-05-03T21:12:19.306 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:19 smithi138 ceph-mon[33110]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000005 succeeded,manifest_file_number is 5, next_file_number is 7, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-05-03T21:12:19.306 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:19 smithi138 ceph-mon[33110]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 0 2024-05-03T21:12:19.306 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:19 smithi138 ceph-mon[33110]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 248de74c-0845-4736-971e-40ef5e45dd7a 2024-05-03T21:12:19.306 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:19 smithi138 ceph-mon[33110]: rocksdb: EVENT_LOG_v1 {"time_micros": 1714770739254094, "job": 1, "event": "recovery_started", "wal_files": [4]} 2024-05-03T21:12:19.306 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:19 smithi138 ceph-mon[33110]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #4 mode 2 2024-05-03T21:12:19.306 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:19 smithi138 ceph-mon[33110]: rocksdb: EVENT_LOG_v1 {"time_micros": 1714770739254523, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 8, "file_size": 1648, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1, "largest_seqno": 5, "table_properties": {"data_size": 523, "index_size": 31, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 401, "raw_average_value_size": 80, "num_data_blocks": 1, "num_entries": 5, "num_filter_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[CompactOnDeletionCollector]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1714770739, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "248de74c-0845-4736-971e-40ef5e45dd7a", "db_session_id": "0HNDXC0MG2ZP1SQYM610", "orig_file_number": 8, "seqno_to_time_mapping": "N/A"}} 2024-05-03T21:12:19.306 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:19 smithi138 ceph-mon[33110]: rocksdb: EVENT_LOG_v1 {"time_micros": 1714770739254603, "job": 1, "event": "recovery_finished"} 2024-05-03T21:12:19.306 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:19 smithi138 ceph-mon[33110]: rocksdb: [db/version_set.cc:5047] Creating manifest 10 2024-05-03T21:12:19.306 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:19 smithi138 ceph-mon[33110]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-c/store.db/000004.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-05-03T21:12:19.307 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:19 smithi138 ceph-mon[33110]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x564afb6ea000 2024-05-03T21:12:19.307 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:19 smithi138 ceph-mon[33110]: rocksdb: DB pointer 0x564afb6d6000 2024-05-03T21:12:19.307 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:19 smithi138 ceph-mon[33110]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-05-03T21:12:19.307 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:19 smithi138 ceph-mon[33110]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-05-03T21:12:19.307 INFO:journalctl@ceph.mon.c.smithi138.stdout: ** DB Stats ** 2024-05-03T21:12:19.307 INFO:journalctl@ceph.mon.c.smithi138.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-05-03T21:12:19.307 INFO:journalctl@ceph.mon.c.smithi138.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-05-03T21:12:19.307 INFO:journalctl@ceph.mon.c.smithi138.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-05-03T21:12:19.307 INFO:journalctl@ceph.mon.c.smithi138.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-05-03T21:12:19.307 INFO:journalctl@ceph.mon.c.smithi138.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-05-03T21:12:19.307 INFO:journalctl@ceph.mon.c.smithi138.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-05-03T21:12:19.308 INFO:journalctl@ceph.mon.c.smithi138.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-05-03T21:12:19.308 INFO:journalctl@ceph.mon.c.smithi138.stdout: 2024-05-03T21:12:19.308 INFO:journalctl@ceph.mon.c.smithi138.stdout: ** Compaction Stats [default] ** 2024-05-03T21:12:19.308 INFO:journalctl@ceph.mon.c.smithi138.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-05-03T21:12:19.308 INFO:journalctl@ceph.mon.c.smithi138.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-05-03T21:12:19.308 INFO:journalctl@ceph.mon.c.smithi138.stdout: L0 1/0 1.61 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.8 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-05-03T21:12:19.308 INFO:journalctl@ceph.mon.c.smithi138.stdout: Sum 1/0 1.61 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.8 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-05-03T21:12:19.308 INFO:journalctl@ceph.mon.c.smithi138.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.8 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-05-03T21:12:19.308 INFO:journalctl@ceph.mon.c.smithi138.stdout: 2024-05-03T21:12:19.308 INFO:journalctl@ceph.mon.c.smithi138.stdout: ** Compaction Stats [default] ** 2024-05-03T21:12:19.308 INFO:journalctl@ceph.mon.c.smithi138.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-05-03T21:12:19.308 INFO:journalctl@ceph.mon.c.smithi138.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-05-03T21:12:19.308 INFO:journalctl@ceph.mon.c.smithi138.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 3.8 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-05-03T21:12:19.308 INFO:journalctl@ceph.mon.c.smithi138.stdout: 2024-05-03T21:12:19.309 INFO:journalctl@ceph.mon.c.smithi138.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-05-03T21:12:19.309 INFO:journalctl@ceph.mon.c.smithi138.stdout: 2024-05-03T21:12:19.309 INFO:journalctl@ceph.mon.c.smithi138.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-05-03T21:12:19.309 INFO:journalctl@ceph.mon.c.smithi138.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-05-03T21:12:19.309 INFO:journalctl@ceph.mon.c.smithi138.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-05-03T21:12:19.309 INFO:journalctl@ceph.mon.c.smithi138.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-05-03T21:12:19.309 INFO:journalctl@ceph.mon.c.smithi138.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-05-03T21:12:19.309 INFO:journalctl@ceph.mon.c.smithi138.stdout: AddFile(Keys): cumulative 0, interval 0 2024-05-03T21:12:19.309 INFO:journalctl@ceph.mon.c.smithi138.stdout: Cumulative compaction: 0.00 GB write, 0.65 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-05-03T21:12:19.309 INFO:journalctl@ceph.mon.c.smithi138.stdout: Interval compaction: 0.00 GB write, 0.65 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-05-03T21:12:19.310 INFO:journalctl@ceph.mon.c.smithi138.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-05-03T21:12:19.310 INFO:journalctl@ceph.mon.c.smithi138.stdout: Block cache BinnedLRUCache@0x564afb5cd1f0#2 capacity: 512.00 MB usage: 0.22 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 9e-06 secs_since: 0 2024-05-03T21:12:19.310 INFO:journalctl@ceph.mon.c.smithi138.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,0.11 KB,2.08616e-05%) IndexBlock(1,0.11 KB,2.08616e-05%) Misc(1,0.00 KB,0%) 2024-05-03T21:12:19.310 INFO:journalctl@ceph.mon.c.smithi138.stdout: 2024-05-03T21:12:19.310 INFO:journalctl@ceph.mon.c.smithi138.stdout: ** File Read Latency Histogram By Level [default] ** 2024-05-03T21:12:19.310 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:19 smithi138 ceph-mon[33110]: mon.c does not exist in monmap, will attempt to join an existing cluster 2024-05-03T21:12:19.310 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:19 smithi138 ceph-mon[33110]: using public_addr v2:172.21.15.138:0/0 -> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] 2024-05-03T21:12:19.310 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:19 smithi138 ceph-mon[33110]: starting mon.c rank -1 at public addrs [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] at bind addrs [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon_data /var/lib/ceph/mon/ceph-c fsid 713bc57a-0991-11ef-bc97-c7b262605968 2024-05-03T21:12:19.310 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:19 smithi138 ceph-mon[33110]: mon.c@-1(???) e0 preinit fsid 713bc57a-0991-11ef-bc97-c7b262605968 2024-05-03T21:12:19.310 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:19 smithi138 ceph-mon[33110]: mon.c@-1(synchronizing).mds e1 new map 2024-05-03T21:12:19.310 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:19 smithi138 ceph-mon[33110]: mon.c@-1(synchronizing).mds e1 print_map 2024-05-03T21:12:19.311 INFO:journalctl@ceph.mon.c.smithi138.stdout: e1 2024-05-03T21:12:19.311 INFO:journalctl@ceph.mon.c.smithi138.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-05-03T21:12:19.311 INFO:journalctl@ceph.mon.c.smithi138.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-05-03T21:12:19.311 INFO:journalctl@ceph.mon.c.smithi138.stdout: legacy client fscid: -1 2024-05-03T21:12:19.311 INFO:journalctl@ceph.mon.c.smithi138.stdout: 2024-05-03T21:12:19.311 INFO:journalctl@ceph.mon.c.smithi138.stdout: No filesystems configured 2024-05-03T21:12:19.311 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:19 smithi138 ceph-mon[33110]: mon.c@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2024-05-03T21:12:19.311 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:19 smithi138 ceph-mon[33110]: mon.c@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2024-05-03T21:12:19.311 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:19 smithi138 ceph-mon[33110]: mon.c@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2024-05-03T21:12:19.311 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:19 smithi138 ceph-mon[33110]: mon.c@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2024-05-03T21:12:19.311 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:19 smithi138 ceph-mon[33110]: mon.c@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2024-05-03T21:12:19.311 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:19 smithi138 ceph-mon[33110]: mon.c@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2024-05-03T21:12:19.311 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:19 smithi138 ceph-mon[33110]: mon.c@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2024-05-03T21:12:19.311 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:19 smithi138 ceph-mon[33110]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-05-03T21:12:19.312 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:19 smithi138 ceph-mon[33110]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-05-03T21:12:19.312 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:19 smithi138 ceph-mon[33110]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-05-03T21:12:19.312 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:19 smithi138 ceph-mon[33110]: from='client.14178 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-03T21:12:19.312 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:19 smithi138 ceph-mon[33110]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-03T21:12:19.312 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:19 smithi138 ceph-mon[33110]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi138", "target": ["mon-mgr", ""]}]: dispatch 2024-05-03T21:12:19.312 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:19 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:12:19.312 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:19 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:12:19.312 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:19 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:12:19.312 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:19 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:12:19.312 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:19 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi082", "name": "osd_memory_target"}]: dispatch 2024-05-03T21:12:19.312 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:19 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T21:12:19.312 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:19 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T21:12:19.312 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:19 smithi138 ceph-mon[33110]: Updating smithi082:/etc/ceph/ceph.conf 2024-05-03T21:12:19.312 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:19 smithi138 ceph-mon[33110]: Updating smithi082:/var/lib/ceph/713bc57a-0991-11ef-bc97-c7b262605968/config/ceph.conf 2024-05-03T21:12:19.312 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:19 smithi138 ceph-mon[33110]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-03T21:12:19.313 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:19 smithi138 ceph-mon[33110]: Deploying cephadm binary to smithi138 2024-05-03T21:12:19.313 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:19 smithi138 ceph-mon[33110]: Updating smithi082:/etc/ceph/ceph.client.admin.keyring 2024-05-03T21:12:19.313 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:19 smithi138 ceph-mon[33110]: Updating smithi082:/var/lib/ceph/713bc57a-0991-11ef-bc97-c7b262605968/config/ceph.client.admin.keyring 2024-05-03T21:12:19.313 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:19 smithi138 ceph-mon[33110]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-03T21:12:19.313 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:19 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:12:19.313 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:19 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:12:19.313 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:19 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:12:19.313 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:19 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:12:19.313 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:19 smithi138 ceph-mon[33110]: Added host smithi138 2024-05-03T21:12:19.313 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:19 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T21:12:19.313 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:19 smithi138 ceph-mon[33110]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-03T21:12:19.313 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:19 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:12:19.313 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:19 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:12:19.313 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:19 smithi138 ceph-mon[33110]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-03T21:12:19.313 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:19 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:12:19.314 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:19 smithi138 ceph-mon[33110]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-03T21:12:19.314 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:19 smithi138 ceph-mon[33110]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-03T21:12:19.314 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:19 smithi138 ceph-mon[33110]: from='client.? 172.21.15.67:0/1569214377' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-05-03T21:12:19.314 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:19 smithi138 ceph-mon[33110]: from='client.? 172.21.15.67:0/1569214377' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-05-03T21:12:19.314 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:19 smithi138 ceph-mon[33110]: osdmap e4: 0 total, 0 up, 0 in 2024-05-03T21:12:19.314 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:19 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:12:19.314 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:19 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:12:19.314 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:19 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:12:19.314 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:19 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:12:19.314 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:19 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi138", "name": "osd_memory_target"}]: dispatch 2024-05-03T21:12:19.314 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:19 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T21:12:19.315 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:19 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T21:12:19.315 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:19 smithi138 ceph-mon[33110]: Updating smithi138:/etc/ceph/ceph.conf 2024-05-03T21:12:19.315 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:19 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:12:19.315 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:19 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:12:19.315 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:19 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:12:19.315 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:19 smithi138 ceph-mon[33110]: Updating smithi138:/var/lib/ceph/713bc57a-0991-11ef-bc97-c7b262605968/config/ceph.conf 2024-05-03T21:12:19.315 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:19 smithi138 ceph-mon[33110]: Updating smithi138:/etc/ceph/ceph.client.admin.keyring 2024-05-03T21:12:19.315 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:19 smithi138 ceph-mon[33110]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-03T21:12:19.315 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:19 smithi138 ceph-mon[33110]: Updating smithi138:/var/lib/ceph/713bc57a-0991-11ef-bc97-c7b262605968/config/ceph.client.admin.keyring 2024-05-03T21:12:19.315 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:19 smithi138 ceph-mon[33110]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-03T21:12:19.315 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:19 smithi138 ceph-mon[33110]: from='client.14186 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi067:172.21.15.67=a;smithi082:172.21.15.82=b;smithi138:172.21.15.138=c", "target": ["mon-mgr", ""]}]: dispatch 2024-05-03T21:12:19.315 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:19 smithi138 ceph-mon[33110]: Saving service mon spec with placement smithi067:172.21.15.67=a;smithi082:172.21.15.82=b;smithi138:172.21.15.138=c;count:3 2024-05-03T21:12:19.315 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:19 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:12:19.315 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:19 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T21:12:19.315 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:19 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T21:12:19.316 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:19 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T21:12:19.316 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:19 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:12:19.316 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:19 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-05-03T21:12:19.316 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:19 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T21:12:19.316 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:19 smithi138 ceph-mon[33110]: Deploying daemon mon.c on smithi138 2024-05-03T21:12:19.316 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:19 smithi138 ceph-mon[33110]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-03T21:12:19.316 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:19 smithi138 ceph-mon[33110]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-03T21:12:19.316 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:19 smithi138 ceph-mon[33110]: from='client.? 172.21.15.138:0/492594802' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-05-03T21:12:19.316 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:19 smithi138 ceph-mon[33110]: mon.c@-1(synchronizing).paxosservice(auth 1..3) refresh upgraded, format 0 -> 3 2024-05-03T21:12:19.316 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:19 smithi138 bash[33090]: b833bba2b5cf358958269b1691b789d1894d99c30b42d1dbccfc1a54ea56a1fb 2024-05-03T21:12:19.317 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:19 smithi138 systemd[1]: Started Ceph mon.c for 713bc57a-0991-11ef-bc97-c7b262605968. 2024-05-03T21:12:19.879 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-05-03T21:12:19.879 DEBUG:teuthology.orchestra.run.smithi138:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9a047fe7599a549042d06a3fb5b6813a203eabea shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 713bc57a-0991-11ef-bc97-c7b262605968 -- ceph mon dump -f json 2024-05-03T21:12:20.110 INFO:teuthology.orchestra.run.smithi138.stderr:Inferring config /var/lib/ceph/713bc57a-0991-11ef-bc97-c7b262605968/mon.c/config 2024-05-03T21:12:24.693 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:12:24.692+0000 7f37fedea700 1 -- 172.21.15.138:0/2952192231 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f37e80035c0 msgr2=0x7f37e8005a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:12:24.694 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:12:24.692+0000 7f37fedea700 1 --2- 172.21.15.138:0/2952192231 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f37e80035c0 0x7f37e8005a50 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f37f8102ef0 tx=0x7f37ec00feb0 comp rx=0 tx=0).stop 2024-05-03T21:12:24.694 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:12:24.692+0000 7f37fedea700 1 -- 172.21.15.138:0/2952192231 shutdown_connections 2024-05-03T21:12:24.694 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:12:24.692+0000 7f37fedea700 1 --2- 172.21.15.138:0/2952192231 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f37e80035c0 0x7f37e8005a50 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:12:24.694 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:12:24.692+0000 7f37fedea700 1 -- 172.21.15.138:0/2952192231 >> 172.21.15.138:0/2952192231 conn(0x7f37f80feab0 msgr2=0x7f37f8100ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T21:12:24.694 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:12:24.692+0000 7f37fedea700 1 -- 172.21.15.138:0/2952192231 shutdown_connections 2024-05-03T21:12:24.694 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:12:24.692+0000 7f37fedea700 1 -- 172.21.15.138:0/2952192231 wait complete. 2024-05-03T21:12:24.694 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:12:24.693+0000 7f37fedea700 1 Processor -- start 2024-05-03T21:12:24.695 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:24 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-03T21:12:24.695 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:24 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-03T21:12:24.695 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:24 smithi138 ceph-mon[33110]: mon.a calling monitor election 2024-05-03T21:12:24.695 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:24 smithi138 ceph-mon[33110]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-03T21:12:24.695 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:24 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-03T21:12:24.695 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:24 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-03T21:12:24.695 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:24 smithi138 ceph-mon[33110]: mon.c calling monitor election 2024-05-03T21:12:24.695 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:24 smithi138 ceph-mon[33110]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-03T21:12:24.695 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:24 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-03T21:12:24.695 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:24 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-03T21:12:24.695 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:24 smithi138 ceph-mon[33110]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-03T21:12:24.696 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:24 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-03T21:12:24.696 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:24 smithi138 ceph-mon[33110]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-05-03T21:12:24.696 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:24 smithi138 ceph-mon[33110]: monmap epoch 2 2024-05-03T21:12:24.696 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:24 smithi138 ceph-mon[33110]: fsid 713bc57a-0991-11ef-bc97-c7b262605968 2024-05-03T21:12:24.696 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:24 smithi138 ceph-mon[33110]: last_changed 2024-05-03T21:12:19.278299+0000 2024-05-03T21:12:24.696 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:24 smithi138 ceph-mon[33110]: created 2024-05-03T21:10:31.243292+0000 2024-05-03T21:12:24.696 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:24 smithi138 ceph-mon[33110]: min_mon_release 19 (squid) 2024-05-03T21:12:24.696 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:24 smithi138 ceph-mon[33110]: election_strategy: 1 2024-05-03T21:12:24.696 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:24 smithi138 ceph-mon[33110]: 0: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.a 2024-05-03T21:12:24.696 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:24 smithi138 ceph-mon[33110]: 1: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.c 2024-05-03T21:12:24.696 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:24 smithi138 ceph-mon[33110]: fsmap 2024-05-03T21:12:24.696 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:24 smithi138 ceph-mon[33110]: osdmap e4: 0 total, 0 up, 0 in 2024-05-03T21:12:24.696 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:24 smithi138 ceph-mon[33110]: mgrmap e13: x(active, since 50s) 2024-05-03T21:12:24.696 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:24 smithi138 ceph-mon[33110]: overall HEALTH_OK 2024-05-03T21:12:24.696 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:24 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:12:24.697 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:24 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:12:24.697 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:24 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:12:24.697 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:24 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-05-03T21:12:24.697 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:24 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T21:12:24.699 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:12:24.693+0000 7f37fedea700 1 -- start start 2024-05-03T21:12:24.699 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:12:24.693+0000 7f37fedea700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f37e80035c0 0x7f37f80757a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:12:24.699 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:12:24.693+0000 7f37fedea700 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f37f8075ce0 0x7f37f8076160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:12:24.699 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:12:24.693+0000 7f37fedea700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f37f810b1a0 con 0x7f37e80035c0 2024-05-03T21:12:24.699 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:12:24.693+0000 7f37fedea700 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f37f810b020 con 0x7f37f8075ce0 2024-05-03T21:12:24.699 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:12:24.693+0000 7f37f7fff700 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f37f8075ce0 0x7f37f8076160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:12:24.699 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:12:24.693+0000 7f37f7fff700 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f37f8075ce0 0x7f37f8076160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3300/0 says I am v2:172.21.15.138:54874/0 (socket says 172.21.15.138:54874) 2024-05-03T21:12:24.699 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:12:24.693+0000 7f37f7fff700 1 -- 172.21.15.138:0/1330608780 learned_addr learned my addr 172.21.15.138:0/1330608780 (peer_addr_for_me v2:172.21.15.138:0/0) 2024-05-03T21:12:24.699 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:12:24.693+0000 7f37fcb86700 1 --2- 172.21.15.138:0/1330608780 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f37e80035c0 0x7f37f80757a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:12:24.700 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:12:24.693+0000 7f37f7fff700 1 -- 172.21.15.138:0/1330608780 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f37f8075ce0 msgr2=0x7f37f8076160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-05-03T21:12:24.700 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:12:24.693+0000 7f37f7fff700 1 -- 172.21.15.138:0/1330608780 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f37f8075ce0 msgr2=0x7f37f8076160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-05-03T21:12:24.700 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:12:24.693+0000 7f37f7fff700 1 --2- 172.21.15.138:0/1330608780 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f37f8075ce0 0x7f37f8076160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-05-03T21:12:24.700 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:12:24.693+0000 7f37f7fff700 1 --2- 172.21.15.138:0/1330608780 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f37f8075ce0 0x7f37f8076160 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-03T21:12:24.700 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:12:24.694+0000 7f37fcb86700 1 -- 172.21.15.138:0/1330608780 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f37f8075ce0 msgr2=0x7f37f8076160 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-03T21:12:24.700 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:12:24.694+0000 7f37fcb86700 1 --2- 172.21.15.138:0/1330608780 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f37f8075ce0 0x7f37f8076160 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:12:24.700 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:12:24.694+0000 7f37fcb86700 1 -- 172.21.15.138:0/1330608780 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f37f80776c0 con 0x7f37e80035c0 2024-05-03T21:12:24.700 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:12:24.694+0000 7f37fcb86700 1 --2- 172.21.15.138:0/1330608780 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f37e80035c0 0x7f37f80757a0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f37ec010010 tx=0x7f37ec0047a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T21:12:24.700 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:12:24.694+0000 7f37f5ffb700 1 -- 172.21.15.138:0/1330608780 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 309+0+0 (secure 0 0 0) 0x7f37ec013b00 con 0x7f37e80035c0 2024-05-03T21:12:24.700 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:12:24.694+0000 7f37f5ffb700 1 -- 172.21.15.138:0/1330608780 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(28 keys) ==== 1170+0+0 (secure 0 0 0) 0x7f37ec004530 con 0x7f37e80035c0 2024-05-03T21:12:24.700 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:12:24.694+0000 7f37fedea700 1 -- 172.21.15.138:0/1330608780 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f37f81a4eb0 con 0x7f37e80035c0 2024-05-03T21:12:24.700 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:12:24.694+0000 7f37f5ffb700 1 -- 172.21.15.138:0/1330608780 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 309+0+0 (secure 0 0 0) 0x7f37ec003e10 con 0x7f37e80035c0 2024-05-03T21:12:24.700 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:12:24.694+0000 7f37fedea700 1 -- 172.21.15.138:0/1330608780 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f37f81a5390 con 0x7f37e80035c0 2024-05-03T21:12:24.701 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:12:24.696+0000 7f37fedea700 1 -- 172.21.15.138:0/1330608780 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f37f8104700 con 0x7f37e80035c0 2024-05-03T21:12:24.701 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:12:24.696+0000 7f37f5ffb700 1 -- 172.21.15.138:0/1330608780 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 13) ==== 50278+0+0 (secure 0 0 0) 0x7f37ec023070 con 0x7f37e80035c0 2024-05-03T21:12:24.701 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:12:24.696+0000 7f37f5ffb700 1 --2- 172.21.15.138:0/1330608780 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f37dc03e2b0 0x7f37dc040770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:12:24.701 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:12:24.696+0000 7f37f5ffb700 1 -- 172.21.15.138:0/1330608780 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f37ec020050 con 0x7f37e80035c0 2024-05-03T21:12:24.701 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:12:24.696+0000 7f37f7fff700 1 --2- 172.21.15.138:0/1330608780 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f37dc03e2b0 0x7f37dc040770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:12:24.701 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:12:24.697+0000 7f37f7fff700 1 --2- 172.21.15.138:0/1330608780 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f37dc03e2b0 0x7f37dc040770 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f37e4009180 tx=0x7f37e4008040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T21:12:24.703 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:12:24.701+0000 7f37f5ffb700 1 -- 172.21.15.138:0/1330608780 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191839 (secure 0 0 0) 0x7f37ec028020 con 0x7f37e80035c0 2024-05-03T21:12:24.721 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:12:24 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-03T21:12:24.721 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:12:24 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-03T21:12:24.721 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:12:24 smithi067 ceph-mon[29730]: mon.a calling monitor election 2024-05-03T21:12:24.721 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:12:24 smithi067 ceph-mon[29730]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-03T21:12:24.721 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:12:24 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-03T21:12:24.721 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:12:24 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-03T21:12:24.722 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:12:24 smithi067 ceph-mon[29730]: mon.c calling monitor election 2024-05-03T21:12:24.722 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:12:24 smithi067 ceph-mon[29730]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-03T21:12:24.722 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:12:24 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-03T21:12:24.722 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:12:24 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-03T21:12:24.722 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:12:24 smithi067 ceph-mon[29730]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-03T21:12:24.722 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:12:24 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-03T21:12:24.722 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:12:24 smithi067 ceph-mon[29730]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-05-03T21:12:24.722 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:12:24 smithi067 ceph-mon[29730]: monmap epoch 2 2024-05-03T21:12:24.722 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:12:24 smithi067 ceph-mon[29730]: fsid 713bc57a-0991-11ef-bc97-c7b262605968 2024-05-03T21:12:24.722 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:12:24 smithi067 ceph-mon[29730]: last_changed 2024-05-03T21:12:19.278299+0000 2024-05-03T21:12:24.722 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:12:24 smithi067 ceph-mon[29730]: created 2024-05-03T21:10:31.243292+0000 2024-05-03T21:12:24.722 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:12:24 smithi067 ceph-mon[29730]: min_mon_release 19 (squid) 2024-05-03T21:12:24.722 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:12:24 smithi067 ceph-mon[29730]: election_strategy: 1 2024-05-03T21:12:24.722 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:12:24 smithi067 ceph-mon[29730]: 0: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.a 2024-05-03T21:12:24.723 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:12:24 smithi067 ceph-mon[29730]: 1: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.c 2024-05-03T21:12:24.723 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:12:24 smithi067 ceph-mon[29730]: fsmap 2024-05-03T21:12:24.723 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:12:24 smithi067 ceph-mon[29730]: osdmap e4: 0 total, 0 up, 0 in 2024-05-03T21:12:24.723 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:12:24 smithi067 ceph-mon[29730]: mgrmap e13: x(active, since 50s) 2024-05-03T21:12:24.723 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:12:24 smithi067 ceph-mon[29730]: overall HEALTH_OK 2024-05-03T21:12:24.723 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:12:24 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:12:24.723 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:12:24 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:12:24.723 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:12:24 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:12:24.723 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:12:24 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-05-03T21:12:24.723 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:12:24 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T21:12:24.991 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:12:24.989+0000 7f37fedea700 1 -- 172.21.15.138:0/1330608780 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) -- 0x7f37f802e510 con 0x7f37e80035c0 2024-05-03T21:12:24.992 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:12:24.990+0000 7f37f5ffb700 1 -- 172.21.15.138:0/1330608780 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 2 v2) ==== 95+0+1015 (secure 0 0 0) 0x7f37ec0105b0 con 0x7f37e80035c0 2024-05-03T21:12:24.992 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-05-03T21:12:24.992 INFO:teuthology.orchestra.run.smithi138.stdout:{"epoch":2,"fsid":"713bc57a-0991-11ef-bc97-c7b262605968","modified":"2024-05-03T21:12:19.278299Z","created":"2024-05-03T21:10:31.243292Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:3300","nonce":0},{"type":"v1","addr":"172.21.15.67:6789","nonce":0}]},"addr":"172.21.15.67:6789/0","public_addr":"172.21.15.67:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:3300","nonce":0},{"type":"v1","addr":"172.21.15.138:6789","nonce":0}]},"addr":"172.21.15.138:6789/0","public_addr":"172.21.15.138:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-05-03T21:12:24.995 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:12:24.993+0000 7f37fedea700 1 -- 172.21.15.138:0/1330608780 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f37dc03e2b0 msgr2=0x7f37dc040770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:12:24.995 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:12:24.993+0000 7f37fedea700 1 --2- 172.21.15.138:0/1330608780 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f37dc03e2b0 0x7f37dc040770 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f37e4009180 tx=0x7f37e4008040 comp rx=0 tx=0).stop 2024-05-03T21:12:24.995 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:12:24.993+0000 7f37fedea700 1 -- 172.21.15.138:0/1330608780 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f37e80035c0 msgr2=0x7f37f80757a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:12:24.995 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:12:24.993+0000 7f37fedea700 1 --2- 172.21.15.138:0/1330608780 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f37e80035c0 0x7f37f80757a0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f37ec010010 tx=0x7f37ec0047a0 comp rx=0 tx=0).stop 2024-05-03T21:12:24.995 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:12:24.993+0000 7f37fedea700 1 -- 172.21.15.138:0/1330608780 shutdown_connections 2024-05-03T21:12:24.995 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:12:24.993+0000 7f37fedea700 1 --2- 172.21.15.138:0/1330608780 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f37dc03e2b0 0x7f37dc040770 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:12:24.995 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:12:24.993+0000 7f37fedea700 1 --2- 172.21.15.138:0/1330608780 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f37e80035c0 0x7f37f80757a0 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:12:24.995 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:12:24.993+0000 7f37fedea700 1 --2- 172.21.15.138:0/1330608780 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f37f8075ce0 0x7f37f8076160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:12:24.995 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:12:24.993+0000 7f37fedea700 1 -- 172.21.15.138:0/1330608780 >> 172.21.15.138:0/1330608780 conn(0x7f37f80feab0 msgr2=0x7f37f80ff6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T21:12:24.995 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:12:24.993+0000 7f37fedea700 1 -- 172.21.15.138:0/1330608780 shutdown_connections 2024-05-03T21:12:24.995 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:12:24.993+0000 7f37fedea700 1 -- 172.21.15.138:0/1330608780 wait complete. 2024-05-03T21:12:24.996 INFO:teuthology.orchestra.run.smithi138.stderr:dumped monmap epoch 2 2024-05-03T21:12:25.563 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:25 smithi138 ceph-mon[33110]: Deploying daemon mon.b on smithi082 2024-05-03T21:12:25.563 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:25 smithi138 ceph-mon[33110]: from='client.? 172.21.15.138:0/1330608780' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-05-03T21:12:25.563 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:25 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-03T21:12:25.622 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:12:25 smithi067 ceph-mon[29730]: Deploying daemon mon.b on smithi082 2024-05-03T21:12:25.622 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:12:25 smithi067 ceph-mon[29730]: from='client.? 172.21.15.138:0/1330608780' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-05-03T21:12:25.622 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:12:25 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-03T21:12:26.565 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-05-03T21:12:26.565 DEBUG:teuthology.orchestra.run.smithi138:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9a047fe7599a549042d06a3fb5b6813a203eabea shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 713bc57a-0991-11ef-bc97-c7b262605968 -- ceph mon dump -f json 2024-05-03T21:12:26.638 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:12:26 smithi067 ceph-mon[29730]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-03T21:12:26.773 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:26 smithi138 ceph-mon[33110]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-03T21:12:26.809 INFO:teuthology.orchestra.run.smithi138.stderr:Inferring config /var/lib/ceph/713bc57a-0991-11ef-bc97-c7b262605968/mon.c/config 2024-05-03T21:12:28.259 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:12:28.256+0000 7f908da57700 1 -- 172.21.15.138:0/3551238807 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9088103220 msgr2=0x7f9088103600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:12:28.259 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:12:28.256+0000 7f908da57700 1 --2- 172.21.15.138:0/3551238807 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9088103220 0x7f9088103600 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f9078009910 tx=0x7f9078009c20 comp rx=0 tx=0).stop 2024-05-03T21:12:28.259 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:12:28.256+0000 7f908da57700 1 -- 172.21.15.138:0/3551238807 shutdown_connections 2024-05-03T21:12:28.259 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:12:28.256+0000 7f908da57700 1 --2- 172.21.15.138:0/3551238807 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9088103220 0x7f9088103600 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:12:28.260 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:12:28.256+0000 7f908da57700 1 -- 172.21.15.138:0/3551238807 >> 172.21.15.138:0/3551238807 conn(0x7f90880feab0 msgr2=0x7f9088100ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T21:12:28.260 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:12:28.256+0000 7f908da57700 1 -- 172.21.15.138:0/3551238807 shutdown_connections 2024-05-03T21:12:28.260 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:12:28.256+0000 7f908da57700 1 -- 172.21.15.138:0/3551238807 wait complete. 2024-05-03T21:12:28.260 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:12:28.257+0000 7f908da57700 1 Processor -- start 2024-05-03T21:12:28.260 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:12:28.257+0000 7f908da57700 1 -- start start 2024-05-03T21:12:28.260 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:12:28.257+0000 7f908da57700 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f9088103220 0x7f9088198d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:12:28.260 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:12:28.257+0000 7f908da57700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9088199260 0x7f908819d5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:12:28.260 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:12:28.257+0000 7f908da57700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f908810d730 con 0x7f9088199260 2024-05-03T21:12:28.260 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:12:28.257+0000 7f908da57700 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f908810d5b0 con 0x7f9088103220 2024-05-03T21:12:28.260 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:12:28.257+0000 7f9086ffd700 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f9088103220 0x7f9088198d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:12:28.260 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:12:28.257+0000 7f9086ffd700 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f9088103220 0x7f9088198d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3300/0 says I am v2:172.21.15.138:54880/0 (socket says 172.21.15.138:54880) 2024-05-03T21:12:28.260 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:12:28.257+0000 7f9086ffd700 1 -- 172.21.15.138:0/2442840544 learned_addr learned my addr 172.21.15.138:0/2442840544 (peer_addr_for_me v2:172.21.15.138:0/0) 2024-05-03T21:12:28.260 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:12:28.257+0000 7f90867fc700 1 --2- 172.21.15.138:0/2442840544 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9088199260 0x7f908819d5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:12:28.260 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:12:28.257+0000 7f9086ffd700 1 -- 172.21.15.138:0/2442840544 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f9088103220 msgr2=0x7f9088198d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-05-03T21:12:28.261 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:12:28.257+0000 7f9086ffd700 1 -- 172.21.15.138:0/2442840544 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f9088103220 msgr2=0x7f9088198d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-05-03T21:12:28.261 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:12:28.257+0000 7f9086ffd700 1 --2- 172.21.15.138:0/2442840544 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f9088103220 0x7f9088198d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-05-03T21:12:28.261 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:12:28.257+0000 7f9086ffd700 1 --2- 172.21.15.138:0/2442840544 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f9088103220 0x7f9088198d20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-03T21:12:28.261 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:12:28.258+0000 7f90867fc700 1 -- 172.21.15.138:0/2442840544 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f9088103220 msgr2=0x7f9088198d20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-03T21:12:28.261 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:12:28.258+0000 7f90867fc700 1 --2- 172.21.15.138:0/2442840544 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f9088103220 0x7f9088198d20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:12:28.261 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:12:28.258+0000 7f90867fc700 1 -- 172.21.15.138:0/2442840544 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f90881998a0 con 0x7f9088199260 2024-05-03T21:12:28.261 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:12:28.258+0000 7f90867fc700 1 --2- 172.21.15.138:0/2442840544 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9088199260 0x7f908819d5f0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f907c00b4d0 tx=0x7f907c00b890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T21:12:28.263 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:12:28.258+0000 7f908ca55700 1 -- 172.21.15.138:0/2442840544 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 309+0+0 (secure 0 0 0) 0x7f907c011630 con 0x7f9088199260 2024-05-03T21:12:28.264 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:12:28.258+0000 7f908da57700 1 -- 172.21.15.138:0/2442840544 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f908819db30 con 0x7f9088199260 2024-05-03T21:12:28.264 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:12:28.258+0000 7f908da57700 1 -- 172.21.15.138:0/2442840544 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f908819e090 con 0x7f9088199260 2024-05-03T21:12:28.264 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:12:28.258+0000 7f908ca55700 1 -- 172.21.15.138:0/2442840544 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(28 keys) ==== 1170+0+0 (secure 0 0 0) 0x7f907c011cd0 con 0x7f9088199260 2024-05-03T21:12:28.264 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:12:28.258+0000 7f908ca55700 1 -- 172.21.15.138:0/2442840544 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 309+0+0 (secure 0 0 0) 0x7f907c0193b0 con 0x7f9088199260 2024-05-03T21:12:28.264 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:12:28.259+0000 7f908da57700 1 -- 172.21.15.138:0/2442840544 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9074005330 con 0x7f9088199260 2024-05-03T21:12:28.264 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:12:28.259+0000 7f908ca55700 1 -- 172.21.15.138:0/2442840544 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 13) ==== 50278+0+0 (secure 0 0 0) 0x7f907c018020 con 0x7f9088199260 2024-05-03T21:12:28.264 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:12:28.259+0000 7f908ca55700 1 --2- 172.21.15.138:0/2442840544 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f907003e2b0 0x7f9070040770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:12:28.264 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:12:28.259+0000 7f908ca55700 1 -- 172.21.15.138:0/2442840544 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f907c052070 con 0x7f9088199260 2024-05-03T21:12:28.264 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:12:28.260+0000 7f9086ffd700 1 --2- 172.21.15.138:0/2442840544 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f907003e2b0 0x7f9070040770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:12:28.264 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:12:28.260+0000 7f9086ffd700 1 --2- 172.21.15.138:0/2442840544 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f907003e2b0 0x7f9070040770 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f9078009750 tx=0x7f9078009f40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T21:12:28.266 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:12:28.265+0000 7f908ca55700 1 -- 172.21.15.138:0/2442840544 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191839 (secure 0 0 0) 0x7f907c01f020 con 0x7f9088199260 2024-05-03T21:12:28.379 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:12:28.377+0000 7f908ca55700 1 -- 172.21.15.138:0/2442840544 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f907c0174a0 con 0x7f9088199260 2024-05-03T21:12:28.559 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:12:28.557+0000 7f908da57700 1 -- 172.21.15.138:0/2442840544 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) -- 0x7f9074005620 con 0x7f9088199260 2024-05-03T21:12:33.392 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:12:33.391+0000 7f908ca55700 1 -- 172.21.15.138:0/2442840544 <== mon.0 v2:172.21.15.67:3300/0 8 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 3 v3) ==== 95+0+1281 (secure 0 0 0) 0x7f907c018770 con 0x7f9088199260 2024-05-03T21:12:33.393 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-05-03T21:12:33.393 INFO:teuthology.orchestra.run.smithi138.stdout:{"epoch":3,"fsid":"713bc57a-0991-11ef-bc97-c7b262605968","modified":"2024-05-03T21:12:28.377266Z","created":"2024-05-03T21:10:31.243292Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:3300","nonce":0},{"type":"v1","addr":"172.21.15.67:6789","nonce":0}]},"addr":"172.21.15.67:6789/0","public_addr":"172.21.15.67:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:3300","nonce":0},{"type":"v1","addr":"172.21.15.138:6789","nonce":0}]},"addr":"172.21.15.138:6789/0","public_addr":"172.21.15.138:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:3300","nonce":0},{"type":"v1","addr":"172.21.15.82:6789","nonce":0}]},"addr":"172.21.15.82:6789/0","public_addr":"172.21.15.82:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1,2]} 2024-05-03T21:12:33.396 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:12:33.394+0000 7f908da57700 1 -- 172.21.15.138:0/2442840544 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f907003e2b0 msgr2=0x7f9070040770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:12:33.396 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:12:33.394+0000 7f908da57700 1 --2- 172.21.15.138:0/2442840544 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f907003e2b0 0x7f9070040770 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f9078009750 tx=0x7f9078009f40 comp rx=0 tx=0).stop 2024-05-03T21:12:33.396 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:12:33.394+0000 7f908da57700 1 -- 172.21.15.138:0/2442840544 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9088199260 msgr2=0x7f908819d5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:12:33.396 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:12:33.394+0000 7f908da57700 1 --2- 172.21.15.138:0/2442840544 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9088199260 0x7f908819d5f0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f907c00b4d0 tx=0x7f907c00b890 comp rx=0 tx=0).stop 2024-05-03T21:12:33.396 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:12:33.394+0000 7f908da57700 1 -- 172.21.15.138:0/2442840544 shutdown_connections 2024-05-03T21:12:33.396 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:12:33.394+0000 7f908da57700 1 --2- 172.21.15.138:0/2442840544 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f907003e2b0 0x7f9070040770 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:12:33.396 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:12:33.394+0000 7f908da57700 1 --2- 172.21.15.138:0/2442840544 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f9088103220 0x7f9088198d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:12:33.396 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:12:33.394+0000 7f908da57700 1 --2- 172.21.15.138:0/2442840544 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9088199260 0x7f908819d5f0 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:12:33.396 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:12:33.394+0000 7f908da57700 1 -- 172.21.15.138:0/2442840544 >> 172.21.15.138:0/2442840544 conn(0x7f90880feab0 msgr2=0x7f9088104c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T21:12:33.396 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:12:33.394+0000 7f908da57700 1 -- 172.21.15.138:0/2442840544 shutdown_connections 2024-05-03T21:12:33.397 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:12:33.394+0000 7f908da57700 1 -- 172.21.15.138:0/2442840544 wait complete. 2024-05-03T21:12:33.397 INFO:teuthology.orchestra.run.smithi138.stderr:dumped monmap epoch 3 2024-05-03T21:12:33.721 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:12:33 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-03T21:12:33.721 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:12:33 smithi067 ceph-mon[29730]: mon.a calling monitor election 2024-05-03T21:12:33.721 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:12:33 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-03T21:12:33.721 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:12:33 smithi067 ceph-mon[29730]: mon.c calling monitor election 2024-05-03T21:12:33.722 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:12:33 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-03T21:12:33.722 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:12:33 smithi067 ceph-mon[29730]: from='client.? 172.21.15.138:0/2442840544' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-05-03T21:12:33.722 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:12:33 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-03T21:12:33.722 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:12:33 smithi067 ceph-mon[29730]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-03T21:12:33.722 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:12:33 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-03T21:12:33.722 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:12:33 smithi067 ceph-mon[29730]: mon.b calling monitor election 2024-05-03T21:12:33.722 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:12:33 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-03T21:12:33.722 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:12:33 smithi067 ceph-mon[29730]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-03T21:12:33.722 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:12:33 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-03T21:12:33.722 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:12:33 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-03T21:12:33.722 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:12:33 smithi067 ceph-mon[29730]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-05-03T21:12:33.722 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:12:33 smithi067 ceph-mon[29730]: monmap epoch 3 2024-05-03T21:12:33.722 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:12:33 smithi067 ceph-mon[29730]: fsid 713bc57a-0991-11ef-bc97-c7b262605968 2024-05-03T21:12:33.723 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:12:33 smithi067 ceph-mon[29730]: last_changed 2024-05-03T21:12:28.377266+0000 2024-05-03T21:12:33.723 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:12:33 smithi067 ceph-mon[29730]: created 2024-05-03T21:10:31.243292+0000 2024-05-03T21:12:33.723 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:12:33 smithi067 ceph-mon[29730]: min_mon_release 19 (squid) 2024-05-03T21:12:33.723 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:12:33 smithi067 ceph-mon[29730]: election_strategy: 1 2024-05-03T21:12:33.723 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:12:33 smithi067 ceph-mon[29730]: 0: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.a 2024-05-03T21:12:33.723 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:12:33 smithi067 ceph-mon[29730]: 1: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.c 2024-05-03T21:12:33.723 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:12:33 smithi067 ceph-mon[29730]: 2: [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] mon.b 2024-05-03T21:12:33.723 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:12:33 smithi067 ceph-mon[29730]: fsmap 2024-05-03T21:12:33.723 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:12:33 smithi067 ceph-mon[29730]: osdmap e4: 0 total, 0 up, 0 in 2024-05-03T21:12:33.723 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:12:33 smithi067 ceph-mon[29730]: mgrmap e13: x(active, since 59s) 2024-05-03T21:12:33.723 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:12:33 smithi067 ceph-mon[29730]: overall HEALTH_OK 2024-05-03T21:12:33.723 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:12:33 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:12:33.723 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:12:33 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:12:33.745 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:12:33 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:12:33.824 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:33 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-03T21:12:33.824 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:33 smithi138 ceph-mon[33110]: mon.a calling monitor election 2024-05-03T21:12:33.825 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:33 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-03T21:12:33.825 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:33 smithi138 ceph-mon[33110]: mon.c calling monitor election 2024-05-03T21:12:33.825 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:33 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-03T21:12:33.825 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:33 smithi138 ceph-mon[33110]: from='client.? 172.21.15.138:0/2442840544' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-05-03T21:12:33.825 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:33 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-03T21:12:33.825 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:33 smithi138 ceph-mon[33110]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-03T21:12:33.825 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:33 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-03T21:12:33.825 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:33 smithi138 ceph-mon[33110]: mon.b calling monitor election 2024-05-03T21:12:33.825 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:33 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-03T21:12:33.825 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:33 smithi138 ceph-mon[33110]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-03T21:12:33.825 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:33 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-03T21:12:33.825 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:33 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-03T21:12:33.826 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:33 smithi138 ceph-mon[33110]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-05-03T21:12:33.826 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:33 smithi138 ceph-mon[33110]: monmap epoch 3 2024-05-03T21:12:33.826 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:33 smithi138 ceph-mon[33110]: fsid 713bc57a-0991-11ef-bc97-c7b262605968 2024-05-03T21:12:33.826 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:33 smithi138 ceph-mon[33110]: last_changed 2024-05-03T21:12:28.377266+0000 2024-05-03T21:12:33.826 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:33 smithi138 ceph-mon[33110]: created 2024-05-03T21:10:31.243292+0000 2024-05-03T21:12:33.826 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:33 smithi138 ceph-mon[33110]: min_mon_release 19 (squid) 2024-05-03T21:12:33.826 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:33 smithi138 ceph-mon[33110]: election_strategy: 1 2024-05-03T21:12:33.826 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:33 smithi138 ceph-mon[33110]: 0: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.a 2024-05-03T21:12:33.826 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:33 smithi138 ceph-mon[33110]: 1: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.c 2024-05-03T21:12:33.826 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:33 smithi138 ceph-mon[33110]: 2: [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] mon.b 2024-05-03T21:12:33.826 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:33 smithi138 ceph-mon[33110]: fsmap 2024-05-03T21:12:33.826 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:33 smithi138 ceph-mon[33110]: osdmap e4: 0 total, 0 up, 0 in 2024-05-03T21:12:33.826 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:33 smithi138 ceph-mon[33110]: mgrmap e13: x(active, since 59s) 2024-05-03T21:12:33.826 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:33 smithi138 ceph-mon[33110]: overall HEALTH_OK 2024-05-03T21:12:33.826 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:33 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:12:33.827 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:33 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:12:33.906 INFO:tasks.cephadm:Generating final ceph.conf file... 2024-05-03T21:12:33.906 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9a047fe7599a549042d06a3fb5b6813a203eabea shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 713bc57a-0991-11ef-bc97-c7b262605968 -- ceph config generate-minimal-conf 2024-05-03T21:12:34.159 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/713bc57a-0991-11ef-bc97-c7b262605968/mon.a/config 2024-05-03T21:12:34.536 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:12:34 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:12:34.536 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:12:34 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:12:34.536 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:12:34 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T21:12:34.537 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:12:34 smithi067 ceph-mon[29730]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-03T21:12:34.537 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:12:34 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:12:34.537 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:12:34 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-03T21:12:34.745 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:12:34 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:12:34.745 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:12:34 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:12:34.745 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:12:34 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T21:12:34.745 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:12:34 smithi082 ceph-mon[32933]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-03T21:12:34.746 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:12:34 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:12:34.746 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:12:34 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-03T21:12:34.824 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:34 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:12:34.824 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:34 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:12:34.824 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:34 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T21:12:34.824 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:34 smithi138 ceph-mon[33110]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-03T21:12:34.824 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:34 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:12:34.824 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:34 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-03T21:12:35.910 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:12:35 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:12:35.983 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:12:35 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:12:35.983 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:12:35 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:12:35.983 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:12:35 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:12:35.983 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:12:35 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T21:12:35.983 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:12:35 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T21:12:35.983 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:12:35 smithi067 ceph-mon[29730]: Updating smithi067:/etc/ceph/ceph.conf 2024-05-03T21:12:35.983 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:12:35 smithi067 ceph-mon[29730]: Updating smithi082:/etc/ceph/ceph.conf 2024-05-03T21:12:35.983 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:12:35 smithi067 ceph-mon[29730]: Updating smithi138:/etc/ceph/ceph.conf 2024-05-03T21:12:35.983 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:12:35 smithi067 ceph-mon[29730]: Updating smithi082:/var/lib/ceph/713bc57a-0991-11ef-bc97-c7b262605968/config/ceph.conf 2024-05-03T21:12:35.983 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:12:35 smithi067 ceph-mon[29730]: Updating smithi067:/var/lib/ceph/713bc57a-0991-11ef-bc97-c7b262605968/config/ceph.conf 2024-05-03T21:12:35.983 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:12:35 smithi067 ceph-mon[29730]: Updating smithi138:/var/lib/ceph/713bc57a-0991-11ef-bc97-c7b262605968/config/ceph.conf 2024-05-03T21:12:35.984 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:12:35 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:12:35.984 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:12:35 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:12:35.984 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:12:35 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:12:35.984 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:12:35 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:12:35.984 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:12:35 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:12:35.984 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:12:35 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:12:35.984 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:12:35 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:12:35.984 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:12:35 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:12:35.984 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:12:35 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:12:35.984 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:12:35 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:12:35.984 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:12:35 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:12:35.984 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:12:35 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-05-03T21:12:35.984 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:12:35 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-05-03T21:12:35.984 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:12:35 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T21:12:35.985 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:12:35.899+0000 7f06dfa0d700 1 -- 172.21.15.67:0/3509354707 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f06d8109030 msgr2=0x7f06d8109410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:12:35.986 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:12:35.899+0000 7f06dfa0d700 1 --2- 172.21.15.67:0/3509354707 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f06d8109030 0x7f06d8109410 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f06d4009910 tx=0x7f06d4009c20 comp rx=0 tx=0).stop 2024-05-03T21:12:35.986 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:12:35.899+0000 7f06dfa0d700 1 -- 172.21.15.67:0/3509354707 shutdown_connections 2024-05-03T21:12:35.986 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:12:35.899+0000 7f06dfa0d700 1 --2- 172.21.15.67:0/3509354707 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f06d8109030 0x7f06d8109410 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:12:35.986 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:12:35.899+0000 7f06dfa0d700 1 -- 172.21.15.67:0/3509354707 >> 172.21.15.67:0/3509354707 conn(0x7f06d8078520 msgr2=0x7f06d8078930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T21:12:35.986 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:12:35.899+0000 7f06dfa0d700 1 -- 172.21.15.67:0/3509354707 shutdown_connections 2024-05-03T21:12:35.986 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:12:35.899+0000 7f06dfa0d700 1 -- 172.21.15.67:0/3509354707 wait complete. 2024-05-03T21:12:35.986 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:12:35.899+0000 7f06dfa0d700 1 Processor -- start 2024-05-03T21:12:35.986 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:12:35.899+0000 7f06dfa0d700 1 -- start start 2024-05-03T21:12:35.986 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:12:35.900+0000 7f06dfa0d700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f06d8109030 0x7f06d8102610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:12:35.986 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:12:35.900+0000 7f06dfa0d700 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f06d8102b50 0x7f06d8102fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:12:35.986 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:12:35.900+0000 7f06dfa0d700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f06d81a5020 0x7f06d81a7410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:12:35.987 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:12:35.900+0000 7f06dfa0d700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f06d810d720 con 0x7f06d81a5020 2024-05-03T21:12:35.987 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:12:35.900+0000 7f06dfa0d700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f06d810d5a0 con 0x7f06d8109030 2024-05-03T21:12:35.987 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:12:35.900+0000 7f06dfa0d700 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f06d810d8a0 con 0x7f06d8102b50 2024-05-03T21:12:35.987 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:12:35.900+0000 7f06ddfaa700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f06d81a5020 0x7f06d81a7410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:12:35.987 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:12:35.900+0000 7f06ddfaa700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f06d81a5020 0x7f06d81a7410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:45712/0 (socket says 172.21.15.67:45712) 2024-05-03T21:12:35.987 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:12:35.900+0000 7f06ddfaa700 1 -- 172.21.15.67:0/26043750 learned_addr learned my addr 172.21.15.67:0/26043750 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-03T21:12:35.987 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:12:35.900+0000 7f06dd7a9700 1 --2- 172.21.15.67:0/26043750 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f06d8109030 0x7f06d8102610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:12:35.987 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:12:35.900+0000 7f06dcfa8700 1 --2- 172.21.15.67:0/26043750 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f06d8102b50 0x7f06d8102fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:12:35.987 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:12:35.900+0000 7f06ddfaa700 1 -- 172.21.15.67:0/26043750 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f06d8102b50 msgr2=0x7f06d8102fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:12:35.987 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:12:35.900+0000 7f06ddfaa700 1 --2- 172.21.15.67:0/26043750 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f06d8102b50 0x7f06d8102fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:12:35.987 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:12:35.900+0000 7f06ddfaa700 1 -- 172.21.15.67:0/26043750 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f06d8109030 msgr2=0x7f06d8102610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:12:35.987 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:12:35.900+0000 7f06ddfaa700 1 --2- 172.21.15.67:0/26043750 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f06d8109030 0x7f06d8102610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:12:35.988 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:12:35.900+0000 7f06ddfaa700 1 -- 172.21.15.67:0/26043750 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f06d81a7ae0 con 0x7f06d81a5020 2024-05-03T21:12:35.988 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:12:35.900+0000 7f06ddfaa700 1 --2- 172.21.15.67:0/26043750 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f06d81a5020 0x7f06d81a7410 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f06d000b9e0 tx=0x7f06d000bcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T21:12:35.988 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:12:35.900+0000 7f06ce7fc700 1 -- 172.21.15.67:0/26043750 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06d0019070 con 0x7f06d81a5020 2024-05-03T21:12:35.988 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:12:35.901+0000 7f06dfa0d700 1 -- 172.21.15.67:0/26043750 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f06d81a7dd0 con 0x7f06d81a5020 2024-05-03T21:12:35.988 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:12:35.901+0000 7f06ce7fc700 1 -- 172.21.15.67:0/26043750 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(28 keys) ==== 1170+0+0 (secure 0 0 0) 0x7f06d000ce30 con 0x7f06d81a5020 2024-05-03T21:12:35.988 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:12:35.901+0000 7f06ce7fc700 1 -- 172.21.15.67:0/26043750 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06d00075b0 con 0x7f06d81a5020 2024-05-03T21:12:35.988 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:12:35.901+0000 7f06dfa0d700 1 -- 172.21.15.67:0/26043750 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f06d81a8230 con 0x7f06d81a5020 2024-05-03T21:12:35.988 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:12:35.902+0000 7f06dfa0d700 1 -- 172.21.15.67:0/26043750 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f06d8064b00 con 0x7f06d81a5020 2024-05-03T21:12:35.988 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:12:35.902+0000 7f06ce7fc700 1 -- 172.21.15.67:0/26043750 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 13) ==== 50278+0+0 (secure 0 0 0) 0x7f06d001c070 con 0x7f06d81a5020 2024-05-03T21:12:35.988 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:12:35.902+0000 7f06ce7fc700 1 --2- 172.21.15.67:0/26043750 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f06c403e9a0 0x7f06c4040e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:12:35.989 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:12:35.902+0000 7f06ce7fc700 1 -- 172.21.15.67:0/26043750 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f06d0051ce0 con 0x7f06d81a5020 2024-05-03T21:12:35.989 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:12:35.902+0000 7f06dd7a9700 1 --2- 172.21.15.67:0/26043750 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f06c403e9a0 0x7f06c4040e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:12:35.989 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:12:35.902+0000 7f06dd7a9700 1 --2- 172.21.15.67:0/26043750 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f06c403e9a0 0x7f06c4040e60 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f06d4009750 tx=0x7f06d400b580 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T21:12:35.989 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:12:35.907+0000 7f06ce7fc700 1 -- 172.21.15.67:0/26043750 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191839 (secure 0 0 0) 0x7f06d0012c60 con 0x7f06d81a5020 2024-05-03T21:12:36.123 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:12:36.121+0000 7f06dfa0d700 1 -- 172.21.15.67:0/26043750 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) -- 0x7f06d81a8620 con 0x7f06d81a5020 2024-05-03T21:12:36.123 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:12:36.121+0000 7f06ce7fc700 1 -- 172.21.15.67:0/26043750 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v9) ==== 76+0+273 (secure 0 0 0) 0x7f06d0014760 con 0x7f06d81a5020 2024-05-03T21:12:36.123 INFO:teuthology.orchestra.run.smithi067.stdout:# minimal ceph.conf for 713bc57a-0991-11ef-bc97-c7b262605968 2024-05-03T21:12:36.124 INFO:teuthology.orchestra.run.smithi067.stdout:[global] 2024-05-03T21:12:36.124 INFO:teuthology.orchestra.run.smithi067.stdout: fsid = 713bc57a-0991-11ef-bc97-c7b262605968 2024-05-03T21:12:36.124 INFO:teuthology.orchestra.run.smithi067.stdout: mon_host = [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] 2024-05-03T21:12:36.126 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:12:36.123+0000 7f06dfa0d700 1 -- 172.21.15.67:0/26043750 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f06c403e9a0 msgr2=0x7f06c4040e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:12:36.126 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:12:36.123+0000 7f06dfa0d700 1 --2- 172.21.15.67:0/26043750 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f06c403e9a0 0x7f06c4040e60 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f06d4009750 tx=0x7f06d400b580 comp rx=0 tx=0).stop 2024-05-03T21:12:36.126 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:12:36.123+0000 7f06dfa0d700 1 -- 172.21.15.67:0/26043750 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f06d81a5020 msgr2=0x7f06d81a7410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:12:36.126 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:12:36.123+0000 7f06dfa0d700 1 --2- 172.21.15.67:0/26043750 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f06d81a5020 0x7f06d81a7410 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f06d000b9e0 tx=0x7f06d000bcf0 comp rx=0 tx=0).stop 2024-05-03T21:12:36.126 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:12:36.124+0000 7f06dfa0d700 1 -- 172.21.15.67:0/26043750 shutdown_connections 2024-05-03T21:12:36.126 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:12:36.124+0000 7f06dfa0d700 1 --2- 172.21.15.67:0/26043750 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f06c403e9a0 0x7f06c4040e60 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:12:36.126 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:12:36.124+0000 7f06dfa0d700 1 --2- 172.21.15.67:0/26043750 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f06d81a5020 0x7f06d81a7410 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:12:36.126 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:12:36.124+0000 7f06dfa0d700 1 --2- 172.21.15.67:0/26043750 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f06d8109030 0x7f06d8102610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:12:36.126 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:12:36.124+0000 7f06dfa0d700 1 --2- 172.21.15.67:0/26043750 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f06d8102b50 0x7f06d8102fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:12:36.126 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:12:36.124+0000 7f06dfa0d700 1 -- 172.21.15.67:0/26043750 >> 172.21.15.67:0/26043750 conn(0x7f06d8078520 msgr2=0x7f06d8104ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T21:12:36.126 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:12:36.124+0000 7f06dfa0d700 1 -- 172.21.15.67:0/26043750 shutdown_connections 2024-05-03T21:12:36.126 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:12:36.124+0000 7f06dfa0d700 1 -- 172.21.15.67:0/26043750 wait complete. 2024-05-03T21:12:36.245 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:12:35 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:12:36.245 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:12:35 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:12:36.245 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:12:35 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:12:36.246 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:12:35 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:12:36.246 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:12:35 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T21:12:36.246 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:12:35 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T21:12:36.246 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:12:35 smithi082 ceph-mon[32933]: Updating smithi067:/etc/ceph/ceph.conf 2024-05-03T21:12:36.246 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:12:35 smithi082 ceph-mon[32933]: Updating smithi082:/etc/ceph/ceph.conf 2024-05-03T21:12:36.246 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:12:35 smithi082 ceph-mon[32933]: Updating smithi138:/etc/ceph/ceph.conf 2024-05-03T21:12:36.246 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:12:35 smithi082 ceph-mon[32933]: Updating smithi082:/var/lib/ceph/713bc57a-0991-11ef-bc97-c7b262605968/config/ceph.conf 2024-05-03T21:12:36.246 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:12:35 smithi082 ceph-mon[32933]: Updating smithi067:/var/lib/ceph/713bc57a-0991-11ef-bc97-c7b262605968/config/ceph.conf 2024-05-03T21:12:36.246 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:12:35 smithi082 ceph-mon[32933]: Updating smithi138:/var/lib/ceph/713bc57a-0991-11ef-bc97-c7b262605968/config/ceph.conf 2024-05-03T21:12:36.246 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:12:35 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:12:36.246 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:12:35 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:12:36.246 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:12:35 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:12:36.246 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:12:35 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:12:36.246 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:12:35 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:12:36.247 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:12:35 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:12:36.247 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:12:35 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:12:36.247 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:12:35 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:12:36.247 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:12:35 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:12:36.247 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:12:35 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:12:36.247 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:12:35 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:12:36.247 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:12:35 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-05-03T21:12:36.247 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:12:35 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-05-03T21:12:36.247 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:12:35 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T21:12:36.324 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:35 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:12:36.325 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:35 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:12:36.325 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:35 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:12:36.325 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:35 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:12:36.325 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:35 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T21:12:36.325 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:35 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T21:12:36.325 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:35 smithi138 ceph-mon[33110]: Updating smithi067:/etc/ceph/ceph.conf 2024-05-03T21:12:36.325 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:35 smithi138 ceph-mon[33110]: Updating smithi082:/etc/ceph/ceph.conf 2024-05-03T21:12:36.325 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:35 smithi138 ceph-mon[33110]: Updating smithi138:/etc/ceph/ceph.conf 2024-05-03T21:12:36.325 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:35 smithi138 ceph-mon[33110]: Updating smithi082:/var/lib/ceph/713bc57a-0991-11ef-bc97-c7b262605968/config/ceph.conf 2024-05-03T21:12:36.326 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:35 smithi138 ceph-mon[33110]: Updating smithi067:/var/lib/ceph/713bc57a-0991-11ef-bc97-c7b262605968/config/ceph.conf 2024-05-03T21:12:36.326 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:35 smithi138 ceph-mon[33110]: Updating smithi138:/var/lib/ceph/713bc57a-0991-11ef-bc97-c7b262605968/config/ceph.conf 2024-05-03T21:12:36.326 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:35 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:12:36.326 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:35 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:12:36.326 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:35 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:12:36.326 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:35 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:12:36.326 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:35 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:12:36.326 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:35 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:12:36.326 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:35 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:12:36.326 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:35 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:12:36.326 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:35 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:12:36.326 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:35 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:12:36.326 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:35 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:12:36.326 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:35 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-05-03T21:12:36.327 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:35 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-05-03T21:12:36.327 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:35 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T21:12:36.753 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2024-05-03T21:12:36.753 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-05-03T21:12:36.753 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd of=/etc/ceph/ceph.conf 2024-05-03T21:12:36.781 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-05-03T21:12:36.781 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-05-03T21:12:36.847 DEBUG:teuthology.orchestra.run.smithi082:> set -ex 2024-05-03T21:12:36.847 DEBUG:teuthology.orchestra.run.smithi082:> sudo dd of=/etc/ceph/ceph.conf 2024-05-03T21:12:36.876 DEBUG:teuthology.orchestra.run.smithi082:> set -ex 2024-05-03T21:12:36.877 DEBUG:teuthology.orchestra.run.smithi082:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-05-03T21:12:36.943 DEBUG:teuthology.orchestra.run.smithi138:> set -ex 2024-05-03T21:12:36.943 DEBUG:teuthology.orchestra.run.smithi138:> sudo dd of=/etc/ceph/ceph.conf 2024-05-03T21:12:36.974 DEBUG:teuthology.orchestra.run.smithi138:> set -ex 2024-05-03T21:12:36.974 DEBUG:teuthology.orchestra.run.smithi138:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-05-03T21:12:37.042 INFO:tasks.cephadm:Adding mgr.x on smithi067 2024-05-03T21:12:37.042 INFO:tasks.cephadm:Adding mgr.y on smithi082 2024-05-03T21:12:37.042 INFO:tasks.cephadm:Adding mgr.z on smithi138 2024-05-03T21:12:37.042 DEBUG:teuthology.orchestra.run.smithi138:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9a047fe7599a549042d06a3fb5b6813a203eabea shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 713bc57a-0991-11ef-bc97-c7b262605968 -- ceph orch apply mgr '3;smithi067=x;smithi082=y;smithi138=z' 2024-05-03T21:12:37.104 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:36 smithi138 ceph-mon[33110]: Reconfiguring mon.a (unknown last config time)... 2024-05-03T21:12:37.104 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:36 smithi138 ceph-mon[33110]: Reconfiguring daemon mon.a on smithi067 2024-05-03T21:12:37.104 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:36 smithi138 ceph-mon[33110]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-03T21:12:37.104 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:36 smithi138 ceph-mon[33110]: from='client.? 172.21.15.67:0/26043750' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T21:12:37.152 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:12:36 smithi067 ceph-mon[29730]: Reconfiguring mon.a (unknown last config time)... 2024-05-03T21:12:37.152 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:12:36 smithi067 ceph-mon[29730]: Reconfiguring daemon mon.a on smithi067 2024-05-03T21:12:37.152 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:12:36 smithi067 ceph-mon[29730]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-03T21:12:37.152 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:12:36 smithi067 ceph-mon[29730]: from='client.? 172.21.15.67:0/26043750' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T21:12:37.245 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:12:36 smithi082 ceph-mon[32933]: Reconfiguring mon.a (unknown last config time)... 2024-05-03T21:12:37.245 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:12:36 smithi082 ceph-mon[32933]: Reconfiguring daemon mon.a on smithi067 2024-05-03T21:12:37.245 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:12:36 smithi082 ceph-mon[32933]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-03T21:12:37.245 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:12:36 smithi082 ceph-mon[32933]: from='client.? 172.21.15.67:0/26043750' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T21:12:37.311 INFO:teuthology.orchestra.run.smithi138.stderr:Inferring config /var/lib/ceph/713bc57a-0991-11ef-bc97-c7b262605968/mon.c/config 2024-05-03T21:12:38.647 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:12:38.643+0000 7f7468b36700 1 -- 172.21.15.138:0/1412562260 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7464103010 msgr2=0x7f7464103430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:12:38.647 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:12:38.643+0000 7f7468b36700 1 --2- 172.21.15.138:0/1412562260 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7464103010 0x7f7464103430 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f7458009910 tx=0x7f7458009c20 comp rx=0 tx=0).stop 2024-05-03T21:12:38.648 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:12:38.643+0000 7f7468b36700 1 -- 172.21.15.138:0/1412562260 shutdown_connections 2024-05-03T21:12:38.648 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:12:38.643+0000 7f7468b36700 1 --2- 172.21.15.138:0/1412562260 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7464103010 0x7f7464103430 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:12:38.648 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:12:38.643+0000 7f7468b36700 1 -- 172.21.15.138:0/1412562260 >> 172.21.15.138:0/1412562260 conn(0x7f74640fe5d0 msgr2=0x7f74641009f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T21:12:38.648 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:12:38.644+0000 7f7468b36700 1 -- 172.21.15.138:0/1412562260 shutdown_connections 2024-05-03T21:12:38.648 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:12:38.644+0000 7f7468b36700 1 -- 172.21.15.138:0/1412562260 wait complete. 2024-05-03T21:12:38.648 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:12:38.644+0000 7f7468b36700 1 Processor -- start 2024-05-03T21:12:38.648 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:12:38.644+0000 7f7468b36700 1 -- start start 2024-05-03T21:12:38.648 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:12:38.644+0000 7f7468b36700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7464103010 0x7f7464198910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:12:38.648 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:12:38.644+0000 7f7468b36700 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f7464198e50 0x7f746419dec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:12:38.648 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:12:38.644+0000 7f7468b36700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f74641992d0 0x7f74641a0450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:12:38.648 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:12:38.644+0000 7f7468b36700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f7464109190 con 0x7f7464103010 2024-05-03T21:12:38.648 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:12:38.644+0000 7f7468b36700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f7464109010 con 0x7f74641992d0 2024-05-03T21:12:38.648 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:12:38.644+0000 7f7468b36700 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f7464109310 con 0x7f7464198e50 2024-05-03T21:12:38.649 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:12:38.645+0000 7f7461d9b700 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f7464198e50 0x7f746419dec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:12:38.649 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:12:38.645+0000 7f7461d9b700 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f7464198e50 0x7f746419dec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3300/0 says I am v2:172.21.15.138:39738/0 (socket says 172.21.15.138:39738) 2024-05-03T21:12:38.649 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:12:38.645+0000 7f7461d9b700 1 -- 172.21.15.138:0/1299695620 learned_addr learned my addr 172.21.15.138:0/1299695620 (peer_addr_for_me v2:172.21.15.138:0/0) 2024-05-03T21:12:38.649 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:12:38.645+0000 7f746259c700 1 --2- 172.21.15.138:0/1299695620 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7464103010 0x7f7464198910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:12:38.649 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:12:38.645+0000 7f7462d9d700 1 --2- 172.21.15.138:0/1299695620 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f74641992d0 0x7f74641a0450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:12:38.649 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:12:38.645+0000 7f7461d9b700 1 -- 172.21.15.138:0/1299695620 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f74641992d0 msgr2=0x7f74641a0450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:12:38.649 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:12:38.645+0000 7f7461d9b700 1 --2- 172.21.15.138:0/1299695620 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f74641992d0 0x7f74641a0450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:12:38.649 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:12:38.645+0000 7f7461d9b700 1 -- 172.21.15.138:0/1299695620 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7464103010 msgr2=0x7f7464198910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:12:38.649 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:12:38.645+0000 7f7461d9b700 1 --2- 172.21.15.138:0/1299695620 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7464103010 0x7f7464198910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:12:38.649 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:12:38.645+0000 7f7461d9b700 1 -- 172.21.15.138:0/1299695620 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f74641a0d20 con 0x7f7464198e50 2024-05-03T21:12:38.649 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:12:38.645+0000 7f7461d9b700 1 --2- 172.21.15.138:0/1299695620 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f7464198e50 0x7f746419dec0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f745400b780 tx=0x7f745400ba90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T21:12:38.649 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:12:38.645+0000 7f74537fe700 1 -- 172.21.15.138:0/1299695620 <== mon.1 v2:172.21.15.138:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7454004230 con 0x7f7464198e50 2024-05-03T21:12:38.649 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:12:38.645+0000 7f7468b36700 1 -- 172.21.15.138:0/1299695620 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f74641a0fb0 con 0x7f7464198e50 2024-05-03T21:12:38.649 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:12:38.645+0000 7f74537fe700 1 -- 172.21.15.138:0/1299695620 <== mon.1 v2:172.21.15.138:3300/0 2 ==== config(28 keys) ==== 1170+0+0 (secure 0 0 0) 0x7f74540043d0 con 0x7f7464198e50 2024-05-03T21:12:38.649 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:12:38.645+0000 7f74537fe700 1 -- 172.21.15.138:0/1299695620 <== mon.1 v2:172.21.15.138:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7454011d30 con 0x7f7464198e50 2024-05-03T21:12:38.650 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:12:38.645+0000 7f7468b36700 1 -- 172.21.15.138:0/1299695620 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f74641a1540 con 0x7f7464198e50 2024-05-03T21:12:38.650 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:12:38.646+0000 7f74537fe700 1 -- 172.21.15.138:0/1299695620 <== mon.1 v2:172.21.15.138:3300/0 4 ==== mgrmap(e 13) ==== 50278+0+0 (secure 0 0 0) 0x7f745401a400 con 0x7f7464198e50 2024-05-03T21:12:38.650 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:12:38.646+0000 7f7468b36700 1 -- 172.21.15.138:0/1299695620 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f746402e510 con 0x7f7464198e50 2024-05-03T21:12:38.651 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:12:38.646+0000 7f74537fe700 1 --2- 172.21.15.138:0/1299695620 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f744c03e9a0 0x7f744c040e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:12:38.651 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:12:38.646+0000 7f74537fe700 1 -- 172.21.15.138:0/1299695620 <== mon.1 v2:172.21.15.138:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f7454052440 con 0x7f7464198e50 2024-05-03T21:12:38.651 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:12:38.647+0000 7f746259c700 1 --2- 172.21.15.138:0/1299695620 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f744c03e9a0 0x7f744c040e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:12:38.651 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:12:38.647+0000 7f746259c700 1 --2- 172.21.15.138:0/1299695620 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f744c03e9a0 0x7f744c040e60 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f745800b5c0 tx=0x7f745801a040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T21:12:38.653 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:12:38.652+0000 7f74537fe700 1 -- 172.21.15.138:0/1299695620 <== mon.1 v2:172.21.15.138:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191839 (secure 0 0 0) 0x7f7454020740 con 0x7f7464198e50 2024-05-03T21:12:38.869 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:12:38.867+0000 7f7468b36700 1 -- 172.21.15.138:0/1299695620 --> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi067=x;smithi082=y;smithi138=z", "target": ["mon-mgr", ""]}) -- 0x7f74640638e0 con 0x7f744c03e9a0 2024-05-03T21:12:38.877 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:12:38.875+0000 7f74537fe700 1 -- 172.21.15.138:0/1299695620 <== mgr.14150 v2:172.21.15.67:6800/604451592 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7f74640638e0 con 0x7f744c03e9a0 2024-05-03T21:12:38.877 INFO:teuthology.orchestra.run.smithi138.stdout:Scheduled mgr update... 2024-05-03T21:12:38.880 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:12:38.878+0000 7f7468b36700 1 -- 172.21.15.138:0/1299695620 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f744c03e9a0 msgr2=0x7f744c040e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:12:38.880 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:12:38.878+0000 7f7468b36700 1 --2- 172.21.15.138:0/1299695620 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f744c03e9a0 0x7f744c040e60 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f745800b5c0 tx=0x7f745801a040 comp rx=0 tx=0).stop 2024-05-03T21:12:38.880 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:12:38.878+0000 7f7468b36700 1 -- 172.21.15.138:0/1299695620 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f7464198e50 msgr2=0x7f746419dec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:12:38.880 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:12:38.878+0000 7f7468b36700 1 --2- 172.21.15.138:0/1299695620 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f7464198e50 0x7f746419dec0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f745400b780 tx=0x7f745400ba90 comp rx=0 tx=0).stop 2024-05-03T21:12:38.880 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:12:38.878+0000 7f7468b36700 1 -- 172.21.15.138:0/1299695620 shutdown_connections 2024-05-03T21:12:38.880 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:12:38.878+0000 7f7468b36700 1 --2- 172.21.15.138:0/1299695620 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f744c03e9a0 0x7f744c040e60 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:12:38.880 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:12:38.878+0000 7f7468b36700 1 --2- 172.21.15.138:0/1299695620 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7464103010 0x7f7464198910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:12:38.880 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:12:38.878+0000 7f7468b36700 1 --2- 172.21.15.138:0/1299695620 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f74641992d0 0x7f74641a0450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:12:38.880 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:12:38.878+0000 7f7468b36700 1 --2- 172.21.15.138:0/1299695620 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f7464198e50 0x7f746419dec0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:12:38.880 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:12:38.878+0000 7f7468b36700 1 -- 172.21.15.138:0/1299695620 >> 172.21.15.138:0/1299695620 conn(0x7f74640fe5d0 msgr2=0x7f74640ff1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T21:12:38.880 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:12:38.878+0000 7f7468b36700 1 -- 172.21.15.138:0/1299695620 shutdown_connections 2024-05-03T21:12:38.880 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:12:38.878+0000 7f7468b36700 1 -- 172.21.15.138:0/1299695620 wait complete. 2024-05-03T21:12:38.995 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:12:38 smithi082 ceph-mon[32933]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-03T21:12:38.996 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:12:38 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:12:38.996 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:12:38 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:12:38.996 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:12:38 smithi082 ceph-mon[32933]: Reconfiguring mon.b (monmap changed)... 2024-05-03T21:12:38.996 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:12:38 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-05-03T21:12:38.996 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:12:38 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-05-03T21:12:38.996 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:12:38 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T21:12:38.996 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:12:38 smithi082 ceph-mon[32933]: Reconfiguring daemon mon.b on smithi082 2024-05-03T21:12:39.074 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:38 smithi138 ceph-mon[33110]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-03T21:12:39.074 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:38 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:12:39.074 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:38 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:12:39.074 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:38 smithi138 ceph-mon[33110]: Reconfiguring mon.b (monmap changed)... 2024-05-03T21:12:39.075 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:38 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-05-03T21:12:39.075 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:38 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-05-03T21:12:39.075 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:38 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T21:12:39.075 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:38 smithi138 ceph-mon[33110]: Reconfiguring daemon mon.b on smithi082 2024-05-03T21:12:39.221 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:12:38 smithi067 ceph-mon[29730]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-03T21:12:39.221 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:12:38 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:12:39.221 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:12:38 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:12:39.221 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:12:38 smithi067 ceph-mon[29730]: Reconfiguring mon.b (monmap changed)... 2024-05-03T21:12:39.221 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:12:38 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-05-03T21:12:39.221 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:12:38 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-05-03T21:12:39.221 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:12:38 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T21:12:39.221 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:12:38 smithi067 ceph-mon[29730]: Reconfiguring daemon mon.b on smithi082 2024-05-03T21:12:39.426 DEBUG:teuthology.orchestra.run.smithi082:mgr.y> sudo journalctl -f -n 0 -u ceph-713bc57a-0991-11ef-bc97-c7b262605968@mgr.y.service 2024-05-03T21:12:39.429 DEBUG:teuthology.orchestra.run.smithi138:mgr.z> sudo journalctl -f -n 0 -u ceph-713bc57a-0991-11ef-bc97-c7b262605968@mgr.z.service 2024-05-03T21:12:39.469 INFO:tasks.cephadm:Deploying OSDs... 2024-05-03T21:12:39.469 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-05-03T21:12:39.469 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/scratch_devs of=/dev/stdout 2024-05-03T21:12:39.490 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-05-03T21:12:39.490 DEBUG:teuthology.orchestra.run.smithi067:> stat /dev/vg_nvme/lv_1 2024-05-03T21:12:39.547 INFO:teuthology.orchestra.run.smithi067.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-05-03T21:12:39.547 INFO:teuthology.orchestra.run.smithi067.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-05-03T21:12:39.548 INFO:teuthology.orchestra.run.smithi067.stdout:Device: 5h/5d Inode: 663 Links: 1 2024-05-03T21:12:39.548 INFO:teuthology.orchestra.run.smithi067.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-03T21:12:39.548 INFO:teuthology.orchestra.run.smithi067.stdout:Context: system_u:object_r:device_t:s0 2024-05-03T21:12:39.548 INFO:teuthology.orchestra.run.smithi067.stdout:Access: 2024-05-03 21:11:44.103002749 +0000 2024-05-03T21:12:39.548 INFO:teuthology.orchestra.run.smithi067.stdout:Modify: 2024-05-03 21:07:06.224420154 +0000 2024-05-03T21:12:39.548 INFO:teuthology.orchestra.run.smithi067.stdout:Change: 2024-05-03 21:07:06.224420154 +0000 2024-05-03T21:12:39.548 INFO:teuthology.orchestra.run.smithi067.stdout: Birth: 2024-05-03 21:07:06.224420154 +0000 2024-05-03T21:12:39.548 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-05-03T21:12:39.615 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-05-03T21:12:39.615 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-05-03T21:12:39.615 INFO:teuthology.orchestra.run.smithi067.stderr:512 bytes copied, 0.000217775 s, 2.4 MB/s 2024-05-03T21:12:39.616 DEBUG:teuthology.orchestra.run.smithi067:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-05-03T21:12:39.675 DEBUG:teuthology.orchestra.run.smithi067:> stat /dev/vg_nvme/lv_2 2024-05-03T21:12:39.732 INFO:teuthology.orchestra.run.smithi067.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-05-03T21:12:39.733 INFO:teuthology.orchestra.run.smithi067.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-05-03T21:12:39.733 INFO:teuthology.orchestra.run.smithi067.stdout:Device: 5h/5d Inode: 685 Links: 1 2024-05-03T21:12:39.733 INFO:teuthology.orchestra.run.smithi067.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-03T21:12:39.733 INFO:teuthology.orchestra.run.smithi067.stdout:Context: system_u:object_r:device_t:s0 2024-05-03T21:12:39.733 INFO:teuthology.orchestra.run.smithi067.stdout:Access: 2024-05-03 21:11:44.132002113 +0000 2024-05-03T21:12:39.733 INFO:teuthology.orchestra.run.smithi067.stdout:Modify: 2024-05-03 21:07:06.567412193 +0000 2024-05-03T21:12:39.733 INFO:teuthology.orchestra.run.smithi067.stdout:Change: 2024-05-03 21:07:06.567412193 +0000 2024-05-03T21:12:39.733 INFO:teuthology.orchestra.run.smithi067.stdout: Birth: 2024-05-03 21:07:06.567412193 +0000 2024-05-03T21:12:39.733 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-05-03T21:12:39.798 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-05-03T21:12:39.798 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-05-03T21:12:39.799 INFO:teuthology.orchestra.run.smithi067.stderr:512 bytes copied, 0.000206979 s, 2.5 MB/s 2024-05-03T21:12:39.800 DEBUG:teuthology.orchestra.run.smithi067:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-05-03T21:12:39.858 DEBUG:teuthology.orchestra.run.smithi067:> stat /dev/vg_nvme/lv_3 2024-05-03T21:12:39.914 INFO:teuthology.orchestra.run.smithi067.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-05-03T21:12:39.914 INFO:teuthology.orchestra.run.smithi067.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-05-03T21:12:39.914 INFO:teuthology.orchestra.run.smithi067.stdout:Device: 5h/5d Inode: 702 Links: 1 2024-05-03T21:12:39.914 INFO:teuthology.orchestra.run.smithi067.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-03T21:12:39.914 INFO:teuthology.orchestra.run.smithi067.stdout:Context: system_u:object_r:device_t:s0 2024-05-03T21:12:39.915 INFO:teuthology.orchestra.run.smithi067.stdout:Access: 2024-05-03 21:11:44.158001542 +0000 2024-05-03T21:12:39.915 INFO:teuthology.orchestra.run.smithi067.stdout:Modify: 2024-05-03 21:07:06.924403907 +0000 2024-05-03T21:12:39.915 INFO:teuthology.orchestra.run.smithi067.stdout:Change: 2024-05-03 21:07:06.924403907 +0000 2024-05-03T21:12:39.915 INFO:teuthology.orchestra.run.smithi067.stdout: Birth: 2024-05-03 21:07:06.924403907 +0000 2024-05-03T21:12:39.915 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-05-03T21:12:39.979 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-05-03T21:12:39.979 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-05-03T21:12:39.979 INFO:teuthology.orchestra.run.smithi067.stderr:512 bytes copied, 0.000219348 s, 2.3 MB/s 2024-05-03T21:12:39.980 DEBUG:teuthology.orchestra.run.smithi067:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-05-03T21:12:40.038 DEBUG:teuthology.orchestra.run.smithi067:> stat /dev/vg_nvme/lv_4 2024-05-03T21:12:40.093 INFO:teuthology.orchestra.run.smithi067.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-05-03T21:12:40.094 INFO:teuthology.orchestra.run.smithi067.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-05-03T21:12:40.094 INFO:teuthology.orchestra.run.smithi067.stdout:Device: 5h/5d Inode: 723 Links: 1 2024-05-03T21:12:40.094 INFO:teuthology.orchestra.run.smithi067.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-03T21:12:40.094 INFO:teuthology.orchestra.run.smithi067.stdout:Context: system_u:object_r:device_t:s0 2024-05-03T21:12:40.094 INFO:teuthology.orchestra.run.smithi067.stdout:Access: 2024-05-03 21:11:44.185000950 +0000 2024-05-03T21:12:40.094 INFO:teuthology.orchestra.run.smithi067.stdout:Modify: 2024-05-03 21:07:07.269395900 +0000 2024-05-03T21:12:40.094 INFO:teuthology.orchestra.run.smithi067.stdout:Change: 2024-05-03 21:07:07.269395900 +0000 2024-05-03T21:12:40.094 INFO:teuthology.orchestra.run.smithi067.stdout: Birth: 2024-05-03 21:07:07.269395900 +0000 2024-05-03T21:12:40.094 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-05-03T21:12:40.155 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:12:39 smithi067 ceph-mon[29730]: from='client.24100 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi067=x;smithi082=y;smithi138=z", "target": ["mon-mgr", ""]}]: dispatch 2024-05-03T21:12:40.155 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:12:39 smithi067 ceph-mon[29730]: Saving service mgr spec with placement smithi067=x;smithi082=y;smithi138=z;count:3 2024-05-03T21:12:40.155 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:12:39 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:12:40.155 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:12:39 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:12:40.155 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:12:39 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:12:40.155 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:12:39 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-05-03T21:12:40.156 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:12:39 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-05-03T21:12:40.156 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:12:39 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T21:12:40.159 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-05-03T21:12:40.159 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-05-03T21:12:40.159 INFO:teuthology.orchestra.run.smithi067.stderr:512 bytes copied, 0.000197671 s, 2.6 MB/s 2024-05-03T21:12:40.159 DEBUG:teuthology.orchestra.run.smithi067:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-05-03T21:12:40.216 DEBUG:teuthology.orchestra.run.smithi082:> set -ex 2024-05-03T21:12:40.216 DEBUG:teuthology.orchestra.run.smithi082:> dd if=/scratch_devs of=/dev/stdout 2024-05-03T21:12:40.218 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:39 smithi138 ceph-mon[33110]: from='client.24100 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi067=x;smithi082=y;smithi138=z", "target": ["mon-mgr", ""]}]: dispatch 2024-05-03T21:12:40.218 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:39 smithi138 ceph-mon[33110]: Saving service mgr spec with placement smithi067=x;smithi082=y;smithi138=z;count:3 2024-05-03T21:12:40.218 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:39 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:12:40.218 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:39 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:12:40.218 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:39 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:12:40.218 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:39 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-05-03T21:12:40.218 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:39 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-05-03T21:12:40.218 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:39 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T21:12:40.232 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-05-03T21:12:40.232 DEBUG:teuthology.orchestra.run.smithi082:> stat /dev/vg_nvme/lv_1 2024-05-03T21:12:40.245 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:12:39 smithi082 ceph-mon[32933]: from='client.24100 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi067=x;smithi082=y;smithi138=z", "target": ["mon-mgr", ""]}]: dispatch 2024-05-03T21:12:40.245 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:12:39 smithi082 ceph-mon[32933]: Saving service mgr spec with placement smithi067=x;smithi082=y;smithi138=z;count:3 2024-05-03T21:12:40.246 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:12:39 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:12:40.246 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:12:39 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:12:40.246 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:12:39 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:12:40.246 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:12:39 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-05-03T21:12:40.246 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:12:39 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-05-03T21:12:40.246 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:12:39 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T21:12:40.260 INFO:teuthology.orchestra.run.smithi082.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-05-03T21:12:40.260 INFO:teuthology.orchestra.run.smithi082.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-05-03T21:12:40.260 INFO:teuthology.orchestra.run.smithi082.stdout:Device: 5h/5d Inode: 662 Links: 1 2024-05-03T21:12:40.260 INFO:teuthology.orchestra.run.smithi082.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-03T21:12:40.260 INFO:teuthology.orchestra.run.smithi082.stdout:Context: system_u:object_r:device_t:s0 2024-05-03T21:12:40.260 INFO:teuthology.orchestra.run.smithi082.stdout:Access: 2024-05-03 21:12:27.803997281 +0000 2024-05-03T21:12:40.260 INFO:teuthology.orchestra.run.smithi082.stdout:Modify: 2024-05-03 21:07:16.597778951 +0000 2024-05-03T21:12:40.260 INFO:teuthology.orchestra.run.smithi082.stdout:Change: 2024-05-03 21:07:16.597778951 +0000 2024-05-03T21:12:40.260 INFO:teuthology.orchestra.run.smithi082.stdout: Birth: 2024-05-03 21:07:16.597778951 +0000 2024-05-03T21:12:40.261 DEBUG:teuthology.orchestra.run.smithi082:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-05-03T21:12:40.322 INFO:teuthology.orchestra.run.smithi082.stderr:1+0 records in 2024-05-03T21:12:40.323 INFO:teuthology.orchestra.run.smithi082.stderr:1+0 records out 2024-05-03T21:12:40.323 INFO:teuthology.orchestra.run.smithi082.stderr:512 bytes copied, 0.000276226 s, 1.9 MB/s 2024-05-03T21:12:40.324 DEBUG:teuthology.orchestra.run.smithi082:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-05-03T21:12:40.380 DEBUG:teuthology.orchestra.run.smithi082:> stat /dev/vg_nvme/lv_2 2024-05-03T21:12:40.435 INFO:teuthology.orchestra.run.smithi082.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-05-03T21:12:40.435 INFO:teuthology.orchestra.run.smithi082.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-05-03T21:12:40.435 INFO:teuthology.orchestra.run.smithi082.stdout:Device: 5h/5d Inode: 678 Links: 1 2024-05-03T21:12:40.435 INFO:teuthology.orchestra.run.smithi082.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-03T21:12:40.435 INFO:teuthology.orchestra.run.smithi082.stdout:Context: system_u:object_r:device_t:s0 2024-05-03T21:12:40.435 INFO:teuthology.orchestra.run.smithi082.stdout:Access: 2024-05-03 21:12:27.803997281 +0000 2024-05-03T21:12:40.435 INFO:teuthology.orchestra.run.smithi082.stdout:Modify: 2024-05-03 21:07:16.933772210 +0000 2024-05-03T21:12:40.435 INFO:teuthology.orchestra.run.smithi082.stdout:Change: 2024-05-03 21:07:16.933772210 +0000 2024-05-03T21:12:40.436 INFO:teuthology.orchestra.run.smithi082.stdout: Birth: 2024-05-03 21:07:16.933772210 +0000 2024-05-03T21:12:40.436 DEBUG:teuthology.orchestra.run.smithi082:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-05-03T21:12:40.497 INFO:teuthology.orchestra.run.smithi082.stderr:1+0 records in 2024-05-03T21:12:40.498 INFO:teuthology.orchestra.run.smithi082.stderr:1+0 records out 2024-05-03T21:12:40.498 INFO:teuthology.orchestra.run.smithi082.stderr:512 bytes copied, 0.000216103 s, 2.4 MB/s 2024-05-03T21:12:40.499 DEBUG:teuthology.orchestra.run.smithi082:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-05-03T21:12:40.554 DEBUG:teuthology.orchestra.run.smithi082:> stat /dev/vg_nvme/lv_3 2024-05-03T21:12:40.609 INFO:teuthology.orchestra.run.smithi082.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-05-03T21:12:40.609 INFO:teuthology.orchestra.run.smithi082.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-05-03T21:12:40.609 INFO:teuthology.orchestra.run.smithi082.stdout:Device: 5h/5d Inode: 697 Links: 1 2024-05-03T21:12:40.609 INFO:teuthology.orchestra.run.smithi082.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-03T21:12:40.609 INFO:teuthology.orchestra.run.smithi082.stdout:Context: system_u:object_r:device_t:s0 2024-05-03T21:12:40.609 INFO:teuthology.orchestra.run.smithi082.stdout:Access: 2024-05-03 21:12:27.804997262 +0000 2024-05-03T21:12:40.610 INFO:teuthology.orchestra.run.smithi082.stdout:Modify: 2024-05-03 21:07:17.279765268 +0000 2024-05-03T21:12:40.610 INFO:teuthology.orchestra.run.smithi082.stdout:Change: 2024-05-03 21:07:17.279765268 +0000 2024-05-03T21:12:40.610 INFO:teuthology.orchestra.run.smithi082.stdout: Birth: 2024-05-03 21:07:17.279765268 +0000 2024-05-03T21:12:40.610 DEBUG:teuthology.orchestra.run.smithi082:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-05-03T21:12:40.673 INFO:teuthology.orchestra.run.smithi082.stderr:1+0 records in 2024-05-03T21:12:40.673 INFO:teuthology.orchestra.run.smithi082.stderr:1+0 records out 2024-05-03T21:12:40.673 INFO:teuthology.orchestra.run.smithi082.stderr:512 bytes copied, 0.000177592 s, 2.9 MB/s 2024-05-03T21:12:40.674 DEBUG:teuthology.orchestra.run.smithi082:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-05-03T21:12:40.729 DEBUG:teuthology.orchestra.run.smithi082:> stat /dev/vg_nvme/lv_4 2024-05-03T21:12:40.783 INFO:teuthology.orchestra.run.smithi082.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-05-03T21:12:40.784 INFO:teuthology.orchestra.run.smithi082.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-05-03T21:12:40.784 INFO:teuthology.orchestra.run.smithi082.stdout:Device: 5h/5d Inode: 718 Links: 1 2024-05-03T21:12:40.784 INFO:teuthology.orchestra.run.smithi082.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-03T21:12:40.784 INFO:teuthology.orchestra.run.smithi082.stdout:Context: system_u:object_r:device_t:s0 2024-05-03T21:12:40.784 INFO:teuthology.orchestra.run.smithi082.stdout:Access: 2024-05-03 21:12:27.804997262 +0000 2024-05-03T21:12:40.784 INFO:teuthology.orchestra.run.smithi082.stdout:Modify: 2024-05-03 21:07:17.623758367 +0000 2024-05-03T21:12:40.784 INFO:teuthology.orchestra.run.smithi082.stdout:Change: 2024-05-03 21:07:17.623758367 +0000 2024-05-03T21:12:40.784 INFO:teuthology.orchestra.run.smithi082.stdout: Birth: 2024-05-03 21:07:17.623758367 +0000 2024-05-03T21:12:40.784 DEBUG:teuthology.orchestra.run.smithi082:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-05-03T21:12:40.846 INFO:teuthology.orchestra.run.smithi082.stderr:1+0 records in 2024-05-03T21:12:40.847 INFO:teuthology.orchestra.run.smithi082.stderr:1+0 records out 2024-05-03T21:12:40.847 INFO:teuthology.orchestra.run.smithi082.stderr:512 bytes copied, 0.000220957 s, 2.3 MB/s 2024-05-03T21:12:40.848 DEBUG:teuthology.orchestra.run.smithi082:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-05-03T21:12:40.904 DEBUG:teuthology.orchestra.run.smithi138:> set -ex 2024-05-03T21:12:40.904 DEBUG:teuthology.orchestra.run.smithi138:> dd if=/scratch_devs of=/dev/stdout 2024-05-03T21:12:40.920 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-05-03T21:12:40.920 DEBUG:teuthology.orchestra.run.smithi138:> stat /dev/vg_nvme/lv_1 2024-05-03T21:12:40.975 INFO:teuthology.orchestra.run.smithi138.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-05-03T21:12:40.975 INFO:teuthology.orchestra.run.smithi138.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-05-03T21:12:40.975 INFO:teuthology.orchestra.run.smithi138.stdout:Device: 5h/5d Inode: 666 Links: 1 2024-05-03T21:12:40.975 INFO:teuthology.orchestra.run.smithi138.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-03T21:12:40.976 INFO:teuthology.orchestra.run.smithi138.stdout:Context: system_u:object_r:device_t:s0 2024-05-03T21:12:40.976 INFO:teuthology.orchestra.run.smithi138.stdout:Access: 2024-05-03 21:12:18.327456809 +0000 2024-05-03T21:12:40.976 INFO:teuthology.orchestra.run.smithi138.stdout:Modify: 2024-05-03 21:07:17.845499304 +0000 2024-05-03T21:12:40.976 INFO:teuthology.orchestra.run.smithi138.stdout:Change: 2024-05-03 21:07:17.845499304 +0000 2024-05-03T21:12:40.976 INFO:teuthology.orchestra.run.smithi138.stdout: Birth: 2024-05-03 21:07:17.845499304 +0000 2024-05-03T21:12:40.976 DEBUG:teuthology.orchestra.run.smithi138:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-05-03T21:12:40.995 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:12:40 smithi082 ceph-mon[32933]: Reconfiguring mon.c (monmap changed)... 2024-05-03T21:12:40.996 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:12:40 smithi082 ceph-mon[32933]: Reconfiguring daemon mon.c on smithi138 2024-05-03T21:12:40.996 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:12:40 smithi082 ceph-mon[32933]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-03T21:12:40.996 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:12:40 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:12:40.996 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:12:40 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:12:41.038 INFO:teuthology.orchestra.run.smithi138.stderr:1+0 records in 2024-05-03T21:12:41.038 INFO:teuthology.orchestra.run.smithi138.stderr:1+0 records out 2024-05-03T21:12:41.039 INFO:teuthology.orchestra.run.smithi138.stderr:512 bytes copied, 0.000281577 s, 1.8 MB/s 2024-05-03T21:12:41.040 DEBUG:teuthology.orchestra.run.smithi138:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-05-03T21:12:41.074 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:40 smithi138 ceph-mon[33110]: Reconfiguring mon.c (monmap changed)... 2024-05-03T21:12:41.075 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:40 smithi138 ceph-mon[33110]: Reconfiguring daemon mon.c on smithi138 2024-05-03T21:12:41.075 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:40 smithi138 ceph-mon[33110]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-03T21:12:41.075 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:40 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:12:41.075 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:40 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:12:41.090 DEBUG:teuthology.orchestra.run.smithi138:> stat /dev/vg_nvme/lv_2 2024-05-03T21:12:41.145 INFO:teuthology.orchestra.run.smithi138.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-05-03T21:12:41.145 INFO:teuthology.orchestra.run.smithi138.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-05-03T21:12:41.145 INFO:teuthology.orchestra.run.smithi138.stdout:Device: 5h/5d Inode: 684 Links: 1 2024-05-03T21:12:41.145 INFO:teuthology.orchestra.run.smithi138.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-03T21:12:41.146 INFO:teuthology.orchestra.run.smithi138.stdout:Context: system_u:object_r:device_t:s0 2024-05-03T21:12:41.146 INFO:teuthology.orchestra.run.smithi138.stdout:Access: 2024-05-03 21:12:18.327456809 +0000 2024-05-03T21:12:41.146 INFO:teuthology.orchestra.run.smithi138.stdout:Modify: 2024-05-03 21:07:18.189491632 +0000 2024-05-03T21:12:41.146 INFO:teuthology.orchestra.run.smithi138.stdout:Change: 2024-05-03 21:07:18.189491632 +0000 2024-05-03T21:12:41.146 INFO:teuthology.orchestra.run.smithi138.stdout: Birth: 2024-05-03 21:07:18.189491632 +0000 2024-05-03T21:12:41.146 DEBUG:teuthology.orchestra.run.smithi138:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-05-03T21:12:41.207 INFO:teuthology.orchestra.run.smithi138.stderr:1+0 records in 2024-05-03T21:12:41.207 INFO:teuthology.orchestra.run.smithi138.stderr:1+0 records out 2024-05-03T21:12:41.208 INFO:teuthology.orchestra.run.smithi138.stderr:512 bytes copied, 0.000205359 s, 2.5 MB/s 2024-05-03T21:12:41.209 DEBUG:teuthology.orchestra.run.smithi138:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-05-03T21:12:41.220 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:12:40 smithi067 ceph-mon[29730]: Reconfiguring mon.c (monmap changed)... 2024-05-03T21:12:41.221 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:12:40 smithi067 ceph-mon[29730]: Reconfiguring daemon mon.c on smithi138 2024-05-03T21:12:41.221 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:12:40 smithi067 ceph-mon[29730]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-03T21:12:41.221 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:12:40 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:12:41.221 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:12:40 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:12:41.265 DEBUG:teuthology.orchestra.run.smithi138:> stat /dev/vg_nvme/lv_3 2024-05-03T21:12:41.320 INFO:teuthology.orchestra.run.smithi138.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-05-03T21:12:41.320 INFO:teuthology.orchestra.run.smithi138.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-05-03T21:12:41.320 INFO:teuthology.orchestra.run.smithi138.stdout:Device: 5h/5d Inode: 702 Links: 1 2024-05-03T21:12:41.320 INFO:teuthology.orchestra.run.smithi138.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-03T21:12:41.320 INFO:teuthology.orchestra.run.smithi138.stdout:Context: system_u:object_r:device_t:s0 2024-05-03T21:12:41.320 INFO:teuthology.orchestra.run.smithi138.stdout:Access: 2024-05-03 21:12:18.327456809 +0000 2024-05-03T21:12:41.320 INFO:teuthology.orchestra.run.smithi138.stdout:Modify: 2024-05-03 21:07:18.525484137 +0000 2024-05-03T21:12:41.320 INFO:teuthology.orchestra.run.smithi138.stdout:Change: 2024-05-03 21:07:18.525484137 +0000 2024-05-03T21:12:41.321 INFO:teuthology.orchestra.run.smithi138.stdout: Birth: 2024-05-03 21:07:18.525484137 +0000 2024-05-03T21:12:41.321 DEBUG:teuthology.orchestra.run.smithi138:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-05-03T21:12:41.382 INFO:teuthology.orchestra.run.smithi138.stderr:1+0 records in 2024-05-03T21:12:41.382 INFO:teuthology.orchestra.run.smithi138.stderr:1+0 records out 2024-05-03T21:12:41.382 INFO:teuthology.orchestra.run.smithi138.stderr:512 bytes copied, 0.000202776 s, 2.5 MB/s 2024-05-03T21:12:41.383 DEBUG:teuthology.orchestra.run.smithi138:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-05-03T21:12:41.439 DEBUG:teuthology.orchestra.run.smithi138:> stat /dev/vg_nvme/lv_4 2024-05-03T21:12:41.494 INFO:teuthology.orchestra.run.smithi138.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-05-03T21:12:41.495 INFO:teuthology.orchestra.run.smithi138.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-05-03T21:12:41.495 INFO:teuthology.orchestra.run.smithi138.stdout:Device: 5h/5d Inode: 721 Links: 1 2024-05-03T21:12:41.495 INFO:teuthology.orchestra.run.smithi138.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-03T21:12:41.495 INFO:teuthology.orchestra.run.smithi138.stdout:Context: system_u:object_r:device_t:s0 2024-05-03T21:12:41.495 INFO:teuthology.orchestra.run.smithi138.stdout:Access: 2024-05-03 21:12:18.328456788 +0000 2024-05-03T21:12:41.495 INFO:teuthology.orchestra.run.smithi138.stdout:Modify: 2024-05-03 21:07:18.855476777 +0000 2024-05-03T21:12:41.495 INFO:teuthology.orchestra.run.smithi138.stdout:Change: 2024-05-03 21:07:18.855476777 +0000 2024-05-03T21:12:41.495 INFO:teuthology.orchestra.run.smithi138.stdout: Birth: 2024-05-03 21:07:18.855476777 +0000 2024-05-03T21:12:41.495 DEBUG:teuthology.orchestra.run.smithi138:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-05-03T21:12:41.556 INFO:teuthology.orchestra.run.smithi138.stderr:1+0 records in 2024-05-03T21:12:41.556 INFO:teuthology.orchestra.run.smithi138.stderr:1+0 records out 2024-05-03T21:12:41.556 INFO:teuthology.orchestra.run.smithi138.stderr:512 bytes copied, 0.000178942 s, 2.9 MB/s 2024-05-03T21:12:41.557 DEBUG:teuthology.orchestra.run.smithi138:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-05-03T21:12:41.614 INFO:tasks.cephadm:Deploying osd.0 on smithi067 with /dev/vg_nvme/lv_4... 2024-05-03T21:12:41.615 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9a047fe7599a549042d06a3fb5b6813a203eabea shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 713bc57a-0991-11ef-bc97-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-05-03T21:12:41.852 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/713bc57a-0991-11ef-bc97-c7b262605968/mon.a/config 2024-05-03T21:12:41.971 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:12:41 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T21:12:41.971 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:12:41 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T21:12:41.971 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:12:41 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T21:12:41.971 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:12:41 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:12:41.971 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:12:41 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-05-03T21:12:41.971 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:12:41 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-05-03T21:12:41.971 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:12:41 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-05-03T21:12:41.971 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:12:41 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T21:12:42.140 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:41 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T21:12:42.140 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:41 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T21:12:42.140 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:41 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T21:12:42.140 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:41 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:12:42.140 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:41 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-05-03T21:12:42.140 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:41 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-05-03T21:12:42.140 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:41 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-05-03T21:12:42.141 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:41 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T21:12:42.245 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:12:41 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T21:12:42.245 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:12:41 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T21:12:42.245 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:12:41 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T21:12:42.245 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:12:41 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:12:42.246 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:12:41 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-05-03T21:12:42.246 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:12:41 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-05-03T21:12:42.246 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:12:41 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-05-03T21:12:42.246 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:12:41 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T21:12:42.997 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:42 smithi138 ceph-mon[33110]: Deploying daemon mgr.z on smithi138 2024-05-03T21:12:42.997 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:42 smithi138 ceph-mon[33110]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-03T21:12:43.157 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:12:42 smithi067 ceph-mon[29730]: Deploying daemon mgr.z on smithi138 2024-05-03T21:12:43.157 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:12:42 smithi067 ceph-mon[29730]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-03T21:12:43.245 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:12:42 smithi082 ceph-mon[32933]: Deploying daemon mgr.z on smithi138 2024-05-03T21:12:43.245 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:12:42 smithi082 ceph-mon[32933]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-03T21:12:43.574 INFO:journalctl@ceph.mgr.z.smithi138.stdout:May 03 21:12:43 smithi138 systemd[1]: Starting Ceph mgr.z for 713bc57a-0991-11ef-bc97-c7b262605968... 2024-05-03T21:12:43.751 INFO:teuthology.orchestra.run.smithi067.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-05-03T21:12:43.751 INFO:teuthology.orchestra.run.smithi067.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-05-03T21:12:43.751 INFO:teuthology.orchestra.run.smithi067.stderr: stderr: 10+0 records in 2024-05-03T21:12:43.751 INFO:teuthology.orchestra.run.smithi067.stderr:10+0 records out 2024-05-03T21:12:43.751 INFO:teuthology.orchestra.run.smithi067.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0112836 s, 929 MB/s 2024-05-03T21:12:43.751 INFO:teuthology.orchestra.run.smithi067.stderr:--> Zapping successful for: 2024-05-03T21:12:43.977 INFO:journalctl@ceph.mgr.z.smithi138.stdout:May 03 21:12:43 smithi138 podman[34950]: 2024-05-03 21:12:43.55752655 +0000 UTC m=+0.017530254 image pull ee4d846202496ba6096593e8e8becc0700a5eb7ebd95536031187f5f337d4012 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:68e15ae152ef5c805c733e76da8fc1d3dd4ee669d5ba025f457e05725baa24dd 2024-05-03T21:12:43.977 INFO:journalctl@ceph.mgr.z.smithi138.stdout:May 03 21:12:43 smithi138 podman[34950]: 2024-05-03 21:12:43.690352868 +0000 UTC m=+0.150356548 container create b67b795589f4586dbbb621dcdb239448062167b48f6ea916c9cce4134e9f0324 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:68e15ae152ef5c805c733e76da8fc1d3dd4ee669d5ba025f457e05725baa24dd, name=ceph-713bc57a-0991-11ef-bc97-c7b262605968-mgr-z, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_CLEAN=True, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.build-date=20240430, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-rishabh-testing-20240501.193033-9a047fe, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, CEPH_POINT_RELEASE=) 2024-05-03T21:12:43.977 INFO:journalctl@ceph.mgr.z.smithi138.stdout:May 03 21:12:43 smithi138 podman[34950]: 2024-05-03 21:12:43.875757802 +0000 UTC m=+0.335761487 container init b67b795589f4586dbbb621dcdb239448062167b48f6ea916c9cce4134e9f0324 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:68e15ae152ef5c805c733e76da8fc1d3dd4ee669d5ba025f457e05725baa24dd, name=ceph-713bc57a-0991-11ef-bc97-c7b262605968-mgr-z, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, RELEASE=wip-rishabh-testing-20240501.193033-9a047fe, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240430, ceph=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image) 2024-05-03T21:12:43.977 INFO:journalctl@ceph.mgr.z.smithi138.stdout:May 03 21:12:43 smithi138 podman[34950]: 2024-05-03 21:12:43.878357151 +0000 UTC m=+0.338360840 container start b67b795589f4586dbbb621dcdb239448062167b48f6ea916c9cce4134e9f0324 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:68e15ae152ef5c805c733e76da8fc1d3dd4ee669d5ba025f457e05725baa24dd, name=ceph-713bc57a-0991-11ef-bc97-c7b262605968-mgr-z, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.build-date=20240430, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, RELEASE=wip-rishabh-testing-20240501.193033-9a047fe, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-05-03T21:12:43.977 INFO:journalctl@ceph.mgr.z.smithi138.stdout:May 03 21:12:43 smithi138 ceph-mgr[34969]: -- 172.21.15.138:0/1072613446 <== mon.1 v2:172.21.15.138:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x562d614905a0 con 0x562d61464800 2024-05-03T21:12:43.977 INFO:journalctl@ceph.mgr.z.smithi138.stdout:May 03 21:12:43 smithi138 bash[34950]: b67b795589f4586dbbb621dcdb239448062167b48f6ea916c9cce4134e9f0324 2024-05-03T21:12:44.316 INFO:journalctl@ceph.mgr.z.smithi138.stdout:May 03 21:12:43 smithi138 systemd[1]: Started Ceph mgr.z for 713bc57a-0991-11ef-bc97-c7b262605968. 2024-05-03T21:12:44.316 INFO:journalctl@ceph.mgr.z.smithi138.stdout:May 03 21:12:44 smithi138 ceph-713bc57a-0991-11ef-bc97-c7b262605968-mgr-z[34965]: 2024-05-03T21:12:44.029+0000 7faed2ad6200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-05-03T21:12:44.316 INFO:journalctl@ceph.mgr.z.smithi138.stdout:May 03 21:12:44 smithi138 ceph-713bc57a-0991-11ef-bc97-c7b262605968-mgr-z[34965]: 2024-05-03T21:12:44.213+0000 7faed2ad6200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-05-03T21:12:44.316 INFO:journalctl@ceph.mgr.z.smithi138.stdout:May 03 21:12:44 smithi138 ceph-713bc57a-0991-11ef-bc97-c7b262605968-mgr-z[34965]: 2024-05-03T21:12:44.313+0000 7faed2ad6200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-05-03T21:12:44.354 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9a047fe7599a549042d06a3fb5b6813a203eabea shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 713bc57a-0991-11ef-bc97-c7b262605968 -- ceph orch daemon add osd smithi067:vg_nvme/lv_4 2024-05-03T21:12:44.594 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/713bc57a-0991-11ef-bc97-c7b262605968/mon.a/config 2024-05-03T21:12:45.046 INFO:journalctl@ceph.mgr.z.smithi138.stdout:May 03 21:12:44 smithi138 ceph-713bc57a-0991-11ef-bc97-c7b262605968-mgr-z[34965]: 2024-05-03T21:12:44.617+0000 7faed2ad6200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-05-03T21:12:45.046 INFO:journalctl@ceph.mgr.z.smithi138.stdout:May 03 21:12:44 smithi138 ceph-713bc57a-0991-11ef-bc97-c7b262605968-mgr-z[34965]: 2024-05-03T21:12:44.699+0000 7faed2ad6200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-05-03T21:12:45.246 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:12:45 smithi082 ceph-mon[32933]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-03T21:12:45.246 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:12:45 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:12:45.246 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:12:45 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:12:45.246 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:12:45 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:12:45.246 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:12:45 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-05-03T21:12:45.246 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:12:45 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-05-03T21:12:45.246 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:12:45 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-05-03T21:12:45.247 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:12:45 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T21:12:45.247 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:12:45 smithi082 ceph-mon[32933]: Deploying daemon mgr.y on smithi082 2024-05-03T21:12:45.324 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:45 smithi138 ceph-mon[33110]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-03T21:12:45.324 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:45 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:12:45.325 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:45 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:12:45.325 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:45 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:12:45.325 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:45 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-05-03T21:12:45.325 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:45 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-05-03T21:12:45.325 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:45 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-05-03T21:12:45.325 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:45 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T21:12:45.325 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:45 smithi138 ceph-mon[33110]: Deploying daemon mgr.y on smithi082 2024-05-03T21:12:45.400 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:12:45 smithi067 ceph-mon[29730]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-03T21:12:45.400 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:12:45 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:12:45.400 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:12:45 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:12:45.400 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:12:45 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:12:45.400 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:12:45 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-05-03T21:12:45.401 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:12:45 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-05-03T21:12:45.401 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:12:45 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-05-03T21:12:45.401 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:12:45 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T21:12:45.401 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:12:45 smithi067 ceph-mon[29730]: Deploying daemon mgr.y on smithi082 2024-05-03T21:12:46.164 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:12:46.161+0000 7f6411620700 1 -- 172.21.15.67:0/2847872663 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f640c103040 msgr2=0x7f640c10a8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:12:46.164 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:12:46.161+0000 7f6411620700 1 --2- 172.21.15.67:0/2847872663 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f640c103040 0x7f640c10a8e0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f640400b0a0 tx=0x7f640400b3b0 comp rx=0 tx=0).stop 2024-05-03T21:12:46.164 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:12:46.161+0000 7f6411620700 1 -- 172.21.15.67:0/2847872663 shutdown_connections 2024-05-03T21:12:46.164 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:12:46.161+0000 7f6411620700 1 --2- 172.21.15.67:0/2847872663 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f640c103040 0x7f640c10a8e0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:12:46.164 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:12:46.161+0000 7f6411620700 1 --2- 172.21.15.67:0/2847872663 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f640c103a20 0x7f640c103e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:12:46.164 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:12:46.161+0000 7f6411620700 1 --2- 172.21.15.67:0/2847872663 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f640c102680 0x7f640c102b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:12:46.164 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:12:46.161+0000 7f6411620700 1 -- 172.21.15.67:0/2847872663 >> 172.21.15.67:0/2847872663 conn(0x7f640c0fd110 msgr2=0x7f640c0ff570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T21:12:46.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:12:46.161+0000 7f6411620700 1 -- 172.21.15.67:0/2847872663 shutdown_connections 2024-05-03T21:12:46.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:12:46.161+0000 7f6411620700 1 -- 172.21.15.67:0/2847872663 wait complete. 2024-05-03T21:12:46.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:12:46.162+0000 7f6411620700 1 Processor -- start 2024-05-03T21:12:46.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:12:46.162+0000 7f6411620700 1 -- start start 2024-05-03T21:12:46.167 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:12:46.162+0000 7f6411620700 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f640c102680 0x7f640c199730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:12:46.167 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:12:46.162+0000 7f6411620700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f640c103040 0x7f640c199c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:12:46.167 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:12:46.162+0000 7f6411620700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f640c103a20 0x7f640c1a0cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:12:46.167 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:12:46.162+0000 7f6411620700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f640c10e100 con 0x7f640c103040 2024-05-03T21:12:46.167 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:12:46.162+0000 7f6411620700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f640c10df80 con 0x7f640c103a20 2024-05-03T21:12:46.167 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:12:46.162+0000 7f6411620700 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f640c10e280 con 0x7f640c102680 2024-05-03T21:12:46.167 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:12:46.163+0000 7f640b7fe700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f640c103a20 0x7f640c1a0cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:12:46.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:12:46.163+0000 7f640affd700 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f640c102680 0x7f640c199730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:12:46.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:12:46.163+0000 7f640b7fe700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f640c103a20 0x7f640c1a0cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.67:51418/0 (socket says 172.21.15.67:51418) 2024-05-03T21:12:46.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:12:46.163+0000 7f640b7fe700 1 -- 172.21.15.67:0/159173635 learned_addr learned my addr 172.21.15.67:0/159173635 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-03T21:12:46.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:12:46.163+0000 7f640a7fc700 1 --2- 172.21.15.67:0/159173635 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f640c103040 0x7f640c199c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:12:46.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:12:46.163+0000 7f640b7fe700 1 -- 172.21.15.67:0/159173635 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f640c102680 msgr2=0x7f640c199730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:12:46.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:12:46.163+0000 7f640b7fe700 1 --2- 172.21.15.67:0/159173635 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f640c102680 0x7f640c199730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:12:46.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:12:46.163+0000 7f640b7fe700 1 -- 172.21.15.67:0/159173635 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f640c103040 msgr2=0x7f640c199c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:12:46.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:12:46.163+0000 7f640b7fe700 1 --2- 172.21.15.67:0/159173635 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f640c103040 0x7f640c199c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:12:46.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:12:46.163+0000 7f640b7fe700 1 -- 172.21.15.67:0/159173635 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f640c1a1390 con 0x7f640c103a20 2024-05-03T21:12:46.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:12:46.163+0000 7f640b7fe700 1 --2- 172.21.15.67:0/159173635 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f640c103a20 0x7f640c1a0cf0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f6404006070 tx=0x7f6404007d10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T21:12:46.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:12:46.163+0000 7f63f3fff700 1 -- 172.21.15.67:0/159173635 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6404003e10 con 0x7f640c103a20 2024-05-03T21:12:46.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:12:46.163+0000 7f6411620700 1 -- 172.21.15.67:0/159173635 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f640c1a1620 con 0x7f640c103a20 2024-05-03T21:12:46.169 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:12:46.163+0000 7f63f3fff700 1 -- 172.21.15.67:0/159173635 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(28 keys) ==== 1170+0+0 (secure 0 0 0) 0x7f6404022070 con 0x7f640c103a20 2024-05-03T21:12:46.169 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:12:46.163+0000 7f63f3fff700 1 -- 172.21.15.67:0/159173635 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f640401b9c0 con 0x7f640c103a20 2024-05-03T21:12:46.169 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:12:46.164+0000 7f6411620700 1 -- 172.21.15.67:0/159173635 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f640c1a1ad0 con 0x7f640c103a20 2024-05-03T21:12:46.169 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:12:46.164+0000 7f63f3fff700 1 -- 172.21.15.67:0/159173635 <== mon.2 v2:172.21.15.82:3300/0 4 ==== mgrmap(e 13) ==== 50278+0+0 (secure 0 0 0) 0x7f640401bb60 con 0x7f640c103a20 2024-05-03T21:12:46.169 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:12:46.164+0000 7f6411620700 1 -- 172.21.15.67:0/159173635 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f640c06a190 con 0x7f640c103a20 2024-05-03T21:12:46.169 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:12:46.165+0000 7f63f3fff700 1 --2- 172.21.15.67:0/159173635 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f63f403e750 0x7f63f4040c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:12:46.169 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:12:46.165+0000 7f63f3fff700 1 -- 172.21.15.67:0/159173635 <== mon.2 v2:172.21.15.82:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f640400f020 con 0x7f640c103a20 2024-05-03T21:12:46.169 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:12:46.165+0000 7f640affd700 1 --2- 172.21.15.67:0/159173635 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f63f403e750 0x7f63f4040c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:12:46.169 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:12:46.165+0000 7f640affd700 1 --2- 172.21.15.67:0/159173635 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f63f403e750 0x7f63f4040c10 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f640c06ba30 tx=0x7f63fc007400 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T21:12:46.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:12:46.170+0000 7f63f3fff700 1 -- 172.21.15.67:0/159173635 <== mon.2 v2:172.21.15.82:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191839 (secure 0 0 0) 0x7f64040197c0 con 0x7f640c103a20 2024-05-03T21:12:46.245 INFO:journalctl@ceph.mgr.y.smithi082.stdout:May 03 21:12:46 smithi082 systemd[1]: Starting Ceph mgr.y for 713bc57a-0991-11ef-bc97-c7b262605968... 2024-05-03T21:12:46.259 INFO:journalctl@ceph.mgr.z.smithi138.stdout:May 03 21:12:45 smithi138 ceph-713bc57a-0991-11ef-bc97-c7b262605968-mgr-z[34965]: 2024-05-03T21:12:45.912+0000 7faed2ad6200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-05-03T21:12:46.259 INFO:journalctl@ceph.mgr.z.smithi138.stdout:May 03 21:12:46 smithi138 ceph-713bc57a-0991-11ef-bc97-c7b262605968-mgr-z[34965]: 2024-05-03T21:12:46.084+0000 7faed2ad6200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-05-03T21:12:46.259 INFO:journalctl@ceph.mgr.z.smithi138.stdout:May 03 21:12:46 smithi138 ceph-713bc57a-0991-11ef-bc97-c7b262605968-mgr-z[34965]: 2024-05-03T21:12:46.256+0000 7faed2ad6200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-05-03T21:12:46.398 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:12:46.396+0000 7f6411620700 1 -- 172.21.15.67:0/159173635 --> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi067:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) -- 0x7f640c0ff200 con 0x7f63f403e750 2024-05-03T21:12:46.496 INFO:journalctl@ceph.mgr.y.smithi082.stdout:May 03 21:12:46 smithi082 podman[34307]: 2024-05-03 21:12:46.145958938 +0000 UTC m=+0.017895441 image pull ee4d846202496ba6096593e8e8becc0700a5eb7ebd95536031187f5f337d4012 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:68e15ae152ef5c805c733e76da8fc1d3dd4ee669d5ba025f457e05725baa24dd 2024-05-03T21:12:46.496 INFO:journalctl@ceph.mgr.y.smithi082.stdout:May 03 21:12:46 smithi082 podman[34307]: 2024-05-03 21:12:46.243996921 +0000 UTC m=+0.115933419 container create cea5d1d3105e675ee00f76cede266bc1e9010f7b555cb401dd7a0ff360c447c8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:68e15ae152ef5c805c733e76da8fc1d3dd4ee669d5ba025f457e05725baa24dd, name=ceph-713bc57a-0991-11ef-bc97-c7b262605968-mgr-y, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240430, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_BRANCH=HEAD, RELEASE=wip-rishabh-testing-20240501.193033-9a047fe, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, io.buildah.version=1.33.5, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS) 2024-05-03T21:12:46.496 INFO:journalctl@ceph.mgr.y.smithi082.stdout:May 03 21:12:46 smithi082 podman[34307]: 2024-05-03 21:12:46.471543873 +0000 UTC m=+0.343480370 container init cea5d1d3105e675ee00f76cede266bc1e9010f7b555cb401dd7a0ff360c447c8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:68e15ae152ef5c805c733e76da8fc1d3dd4ee669d5ba025f457e05725baa24dd, name=ceph-713bc57a-0991-11ef-bc97-c7b262605968-mgr-y, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240430, RELEASE=wip-rishabh-testing-20240501.193033-9a047fe) 2024-05-03T21:12:46.496 INFO:journalctl@ceph.mgr.y.smithi082.stdout:May 03 21:12:46 smithi082 podman[34307]: 2024-05-03 21:12:46.47417969 +0000 UTC m=+0.346116188 container start cea5d1d3105e675ee00f76cede266bc1e9010f7b555cb401dd7a0ff360c447c8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:68e15ae152ef5c805c733e76da8fc1d3dd4ee669d5ba025f457e05725baa24dd, name=ceph-713bc57a-0991-11ef-bc97-c7b262605968-mgr-y, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-rishabh-testing-20240501.193033-9a047fe, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.build-date=20240430, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_CLEAN=True, ceph=True, CEPH_POINT_RELEASE=) 2024-05-03T21:12:46.543 INFO:journalctl@ceph.mgr.z.smithi138.stdout:May 03 21:12:46 smithi138 ceph-713bc57a-0991-11ef-bc97-c7b262605968-mgr-z[34965]: 2024-05-03T21:12:46.541+0000 7faed2ad6200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-05-03T21:12:46.788 INFO:journalctl@ceph.mgr.y.smithi082.stdout:May 03 21:12:46 smithi082 ceph-mgr[34328]: -- 172.21.15.82:0/3694851884 <== mon.2 v2:172.21.15.82:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55b8ccee85a0 con 0x55b8ccebc800 2024-05-03T21:12:46.788 INFO:journalctl@ceph.mgr.y.smithi082.stdout:May 03 21:12:46 smithi082 bash[34307]: cea5d1d3105e675ee00f76cede266bc1e9010f7b555cb401dd7a0ff360c447c8 2024-05-03T21:12:46.788 INFO:journalctl@ceph.mgr.y.smithi082.stdout:May 03 21:12:46 smithi082 systemd[1]: Started Ceph mgr.y for 713bc57a-0991-11ef-bc97-c7b262605968. 2024-05-03T21:12:46.788 INFO:journalctl@ceph.mgr.y.smithi082.stdout:May 03 21:12:46 smithi082 ceph-713bc57a-0991-11ef-bc97-c7b262605968-mgr-y[34323]: 2024-05-03T21:12:46.629+0000 7f5320d77200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-05-03T21:12:46.824 INFO:journalctl@ceph.mgr.z.smithi138.stdout:May 03 21:12:46 smithi138 ceph-713bc57a-0991-11ef-bc97-c7b262605968-mgr-z[34965]: 2024-05-03T21:12:46.631+0000 7faed2ad6200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-05-03T21:12:47.049 INFO:journalctl@ceph.mgr.y.smithi082.stdout:May 03 21:12:46 smithi082 ceph-713bc57a-0991-11ef-bc97-c7b262605968-mgr-y[34323]: 2024-05-03T21:12:46.828+0000 7f5320d77200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-05-03T21:12:47.049 INFO:journalctl@ceph.mgr.y.smithi082.stdout:May 03 21:12:46 smithi082 ceph-713bc57a-0991-11ef-bc97-c7b262605968-mgr-y[34323]: 2024-05-03T21:12:46.936+0000 7f5320d77200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-05-03T21:12:47.306 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:12:47 smithi067 ceph-mon[29730]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-03T21:12:47.307 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:12:47 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-03T21:12:47.307 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:12:47 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-03T21:12:47.307 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:12:47 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T21:12:47.307 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:12:47 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:12:47.307 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:12:47 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:12:47.307 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:12:47 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:12:47.307 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:12:47 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:12:47.307 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:12:47 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T21:12:47.324 INFO:journalctl@ceph.mgr.z.smithi138.stdout:May 03 21:12:46 smithi138 ceph-713bc57a-0991-11ef-bc97-c7b262605968-mgr-z[34965]: 2024-05-03T21:12:46.835+0000 7faed2ad6200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-05-03T21:12:47.324 INFO:journalctl@ceph.mgr.z.smithi138.stdout:May 03 21:12:47 smithi138 ceph-713bc57a-0991-11ef-bc97-c7b262605968-mgr-z[34965]: 2024-05-03T21:12:47.036+0000 7faed2ad6200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-05-03T21:12:47.324 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:47 smithi138 ceph-mon[33110]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-03T21:12:47.324 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:47 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-03T21:12:47.324 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:47 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-03T21:12:47.324 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:47 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T21:12:47.325 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:47 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:12:47.325 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:47 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:12:47.325 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:47 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:12:47.325 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:47 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:12:47.325 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:47 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T21:12:47.368 INFO:journalctl@ceph.mgr.y.smithi082.stdout:May 03 21:12:47 smithi082 ceph-713bc57a-0991-11ef-bc97-c7b262605968-mgr-y[34323]: 2024-05-03T21:12:47.264+0000 7f5320d77200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-05-03T21:12:47.368 INFO:journalctl@ceph.mgr.y.smithi082.stdout:May 03 21:12:47 smithi082 ceph-713bc57a-0991-11ef-bc97-c7b262605968-mgr-y[34323]: 2024-05-03T21:12:47.366+0000 7f5320d77200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-05-03T21:12:47.368 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:12:47 smithi082 ceph-mon[32933]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-03T21:12:47.368 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:12:47 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-03T21:12:47.368 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:12:47 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-03T21:12:47.368 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:12:47 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T21:12:47.369 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:12:47 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:12:47.369 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:12:47 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:12:47.369 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:12:47 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:12:47.369 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:12:47 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:12:47.369 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:12:47 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T21:12:47.912 INFO:journalctl@ceph.mgr.z.smithi138.stdout:May 03 21:12:47 smithi138 ceph-713bc57a-0991-11ef-bc97-c7b262605968-mgr-z[34965]: 2024-05-03T21:12:47.686+0000 7faed2ad6200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-05-03T21:12:47.912 INFO:journalctl@ceph.mgr.z.smithi138.stdout:May 03 21:12:47 smithi138 ceph-713bc57a-0991-11ef-bc97-c7b262605968-mgr-z[34965]: 2024-05-03T21:12:47.909+0000 7faed2ad6200 -1 mgr[py] Module smb has missing NOTIFY_TYPES member 2024-05-03T21:12:48.156 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:12:48 smithi067 ceph-mon[29730]: from='client.24110 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi067:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-05-03T21:12:48.324 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:48 smithi138 ceph-mon[33110]: from='client.24110 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi067:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-05-03T21:12:48.495 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:12:48 smithi082 ceph-mon[32933]: from='client.24110 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi067:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-05-03T21:12:48.824 INFO:journalctl@ceph.mgr.z.smithi138.stdout:May 03 21:12:48 smithi138 ceph-713bc57a-0991-11ef-bc97-c7b262605968-mgr-z[34965]: 2024-05-03T21:12:48.361+0000 7faed2ad6200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-05-03T21:12:48.824 INFO:journalctl@ceph.mgr.z.smithi138.stdout:May 03 21:12:48 smithi138 ceph-713bc57a-0991-11ef-bc97-c7b262605968-mgr-z[34965]: 2024-05-03T21:12:48.455+0000 7faed2ad6200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-05-03T21:12:48.824 INFO:journalctl@ceph.mgr.z.smithi138.stdout:May 03 21:12:48 smithi138 ceph-713bc57a-0991-11ef-bc97-c7b262605968-mgr-z[34965]: 2024-05-03T21:12:48.821+0000 7faed2ad6200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-05-03T21:12:48.951 INFO:journalctl@ceph.mgr.y.smithi082.stdout:May 03 21:12:48 smithi082 ceph-713bc57a-0991-11ef-bc97-c7b262605968-mgr-y[34323]: 2024-05-03T21:12:48.614+0000 7f5320d77200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-05-03T21:12:48.951 INFO:journalctl@ceph.mgr.y.smithi082.stdout:May 03 21:12:48 smithi082 ceph-713bc57a-0991-11ef-bc97-c7b262605968-mgr-y[34323]: 2024-05-03T21:12:48.781+0000 7f5320d77200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-05-03T21:12:48.951 INFO:journalctl@ceph.mgr.y.smithi082.stdout:May 03 21:12:48 smithi082 ceph-713bc57a-0991-11ef-bc97-c7b262605968-mgr-y[34323]: 2024-05-03T21:12:48.949+0000 7f5320d77200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-05-03T21:12:49.237 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:12:49 smithi082 ceph-mon[32933]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-03T21:12:49.237 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:12:49 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:12:49.237 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:12:49 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:12:49.237 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:12:49 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:12:49.237 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:12:49 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:12:49.237 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:12:49 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T21:12:49.237 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:12:49 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T21:12:49.237 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:12:49 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:12:49.237 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:12:49 smithi082 ceph-mon[32933]: Reconfiguring mgr.x (unknown last config time)... 2024-05-03T21:12:49.237 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:12:49 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-05-03T21:12:49.237 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:12:49 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-05-03T21:12:49.238 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:12:49 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T21:12:49.238 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:12:49 smithi082 ceph-mon[32933]: Reconfiguring daemon mgr.x on smithi067 2024-05-03T21:12:49.238 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:12:49 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:12:49.238 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:12:49 smithi082 ceph-mon[32933]: from='client.? 172.21.15.67:0/2954145588' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7939f31c-6fb7-42bd-af96-88a448d11bd2"}]: dispatch 2024-05-03T21:12:49.238 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:12:49 smithi082 ceph-mon[32933]: from='client.? 172.21.15.67:0/2954145588' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "7939f31c-6fb7-42bd-af96-88a448d11bd2"}]': finished 2024-05-03T21:12:49.238 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:12:49 smithi082 ceph-mon[32933]: osdmap e5: 1 total, 0 up, 1 in 2024-05-03T21:12:49.238 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:12:49 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-03T21:12:49.384 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:12:49 smithi067 ceph-mon[29730]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-03T21:12:49.385 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:12:49 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:12:49.385 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:12:49 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:12:49.385 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:12:49 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:12:49.385 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:12:49 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:12:49.385 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:12:49 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T21:12:49.385 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:12:49 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T21:12:49.385 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:12:49 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:12:49.385 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:12:49 smithi067 ceph-mon[29730]: Reconfiguring mgr.x (unknown last config time)... 2024-05-03T21:12:49.385 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:12:49 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-05-03T21:12:49.385 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:12:49 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-05-03T21:12:49.385 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:12:49 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T21:12:49.385 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:12:49 smithi067 ceph-mon[29730]: Reconfiguring daemon mgr.x on smithi067 2024-05-03T21:12:49.386 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:12:49 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:12:49.386 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:12:49 smithi067 ceph-mon[29730]: from='client.? 172.21.15.67:0/2954145588' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7939f31c-6fb7-42bd-af96-88a448d11bd2"}]: dispatch 2024-05-03T21:12:49.386 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:12:49 smithi067 ceph-mon[29730]: from='client.? 172.21.15.67:0/2954145588' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "7939f31c-6fb7-42bd-af96-88a448d11bd2"}]': finished 2024-05-03T21:12:49.386 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:12:49 smithi067 ceph-mon[29730]: osdmap e5: 1 total, 0 up, 1 in 2024-05-03T21:12:49.386 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:12:49 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-03T21:12:49.495 INFO:journalctl@ceph.mgr.y.smithi082.stdout:May 03 21:12:49 smithi082 ceph-713bc57a-0991-11ef-bc97-c7b262605968-mgr-y[34323]: 2024-05-03T21:12:49.234+0000 7f5320d77200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-05-03T21:12:49.495 INFO:journalctl@ceph.mgr.y.smithi082.stdout:May 03 21:12:49 smithi082 ceph-713bc57a-0991-11ef-bc97-c7b262605968-mgr-y[34323]: 2024-05-03T21:12:49.326+0000 7f5320d77200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-05-03T21:12:49.498 INFO:journalctl@ceph.mgr.z.smithi138.stdout:May 03 21:12:49 smithi138 ceph-713bc57a-0991-11ef-bc97-c7b262605968-mgr-z[34965]: 2024-05-03T21:12:49.169+0000 7faed2ad6200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-05-03T21:12:49.498 INFO:journalctl@ceph.mgr.z.smithi138.stdout:May 03 21:12:49 smithi138 ceph-713bc57a-0991-11ef-bc97-c7b262605968-mgr-z[34965]: 2024-05-03T21:12:49.259+0000 7faed2ad6200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-05-03T21:12:49.499 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:49 smithi138 ceph-mon[33110]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-03T21:12:49.499 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:49 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:12:49.499 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:49 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:12:49.499 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:49 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:12:49.499 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:49 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:12:49.499 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:49 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T21:12:49.499 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:49 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T21:12:49.499 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:49 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:12:49.500 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:49 smithi138 ceph-mon[33110]: Reconfiguring mgr.x (unknown last config time)... 2024-05-03T21:12:49.500 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:49 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-05-03T21:12:49.500 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:49 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-05-03T21:12:49.500 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:49 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T21:12:49.500 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:49 smithi138 ceph-mon[33110]: Reconfiguring daemon mgr.x on smithi067 2024-05-03T21:12:49.500 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:49 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:12:49.500 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:49 smithi138 ceph-mon[33110]: from='client.? 172.21.15.67:0/2954145588' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7939f31c-6fb7-42bd-af96-88a448d11bd2"}]: dispatch 2024-05-03T21:12:49.500 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:49 smithi138 ceph-mon[33110]: from='client.? 172.21.15.67:0/2954145588' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "7939f31c-6fb7-42bd-af96-88a448d11bd2"}]': finished 2024-05-03T21:12:49.500 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:49 smithi138 ceph-mon[33110]: osdmap e5: 1 total, 0 up, 1 in 2024-05-03T21:12:49.500 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:49 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-03T21:12:49.824 INFO:journalctl@ceph.mgr.z.smithi138.stdout:May 03 21:12:49 smithi138 ceph-713bc57a-0991-11ef-bc97-c7b262605968-mgr-z[34965]: 2024-05-03T21:12:49.496+0000 7faed2ad6200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-05-03T21:12:49.824 INFO:journalctl@ceph.mgr.z.smithi138.stdout:May 03 21:12:49 smithi138 ceph-713bc57a-0991-11ef-bc97-c7b262605968-mgr-z[34965]: 2024-05-03T21:12:49.625+0000 7faed2ad6200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-05-03T21:12:49.995 INFO:journalctl@ceph.mgr.y.smithi082.stdout:May 03 21:12:49 smithi082 ceph-713bc57a-0991-11ef-bc97-c7b262605968-mgr-y[34323]: 2024-05-03T21:12:49.528+0000 7f5320d77200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-05-03T21:12:49.995 INFO:journalctl@ceph.mgr.y.smithi082.stdout:May 03 21:12:49 smithi082 ceph-713bc57a-0991-11ef-bc97-c7b262605968-mgr-y[34323]: 2024-05-03T21:12:49.717+0000 7f5320d77200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-05-03T21:12:50.324 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:50 smithi138 ceph-mon[33110]: from='client.? 172.21.15.67:0/304683814' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-03T21:12:50.324 INFO:journalctl@ceph.mgr.z.smithi138.stdout:May 03 21:12:49 smithi138 ceph-713bc57a-0991-11ef-bc97-c7b262605968-mgr-z[34965]: 2024-05-03T21:12:49.928+0000 7faed2ad6200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-05-03T21:12:50.324 INFO:journalctl@ceph.mgr.z.smithi138.stdout:May 03 21:12:50 smithi138 ceph-713bc57a-0991-11ef-bc97-c7b262605968-mgr-z[34965]: 2024-05-03T21:12:50.041+0000 7faed2ad6200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-05-03T21:12:50.324 INFO:journalctl@ceph.mgr.z.smithi138.stdout:May 03 21:12:50 smithi138 ceph-713bc57a-0991-11ef-bc97-c7b262605968-mgr-z[34965]: 2024-05-03T21:12:50.130+0000 7faed2ad6200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-05-03T21:12:50.471 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:12:50 smithi067 ceph-mon[29730]: from='client.? 172.21.15.67:0/304683814' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-03T21:12:50.495 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:12:50 smithi082 ceph-mon[32933]: from='client.? 172.21.15.67:0/304683814' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-03T21:12:50.495 INFO:journalctl@ceph.mgr.y.smithi082.stdout:May 03 21:12:50 smithi082 ceph-713bc57a-0991-11ef-bc97-c7b262605968-mgr-y[34323]: 2024-05-03T21:12:50.312+0000 7f5320d77200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-05-03T21:12:50.955 INFO:journalctl@ceph.mgr.y.smithi082.stdout:May 03 21:12:50 smithi082 ceph-713bc57a-0991-11ef-bc97-c7b262605968-mgr-y[34323]: 2024-05-03T21:12:50.533+0000 7f5320d77200 -1 mgr[py] Module smb has missing NOTIFY_TYPES member 2024-05-03T21:12:51.074 INFO:journalctl@ceph.mgr.z.smithi138.stdout:May 03 21:12:50 smithi138 ceph-713bc57a-0991-11ef-bc97-c7b262605968-mgr-z[34965]: 2024-05-03T21:12:50.716+0000 7faed2ad6200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-05-03T21:12:51.074 INFO:journalctl@ceph.mgr.z.smithi138.stdout:May 03 21:12:50 smithi138 ceph-713bc57a-0991-11ef-bc97-c7b262605968-mgr-z[34965]: 2024-05-03T21:12:50.926+0000 7faed2ad6200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-05-03T21:12:51.130 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:12:51.128+0000 7f63f3fff700 1 -- 172.21.15.67:0/159173635 <== mon.2 v2:172.21.15.82:3300/0 7 ==== mgrmap(e 14) ==== 99937+0+0 (secure 0 0 0) 0x7f6404025c90 con 0x7f640c103a20 2024-05-03T21:12:51.245 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:12:51 smithi082 ceph-mon[32933]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-03T21:12:51.245 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:12:51 smithi082 ceph-mon[32933]: Standby manager daemon z started 2024-05-03T21:12:51.245 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:12:51 smithi082 ceph-mon[32933]: from='mgr.? 172.21.15.138:0/2504648021' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-05-03T21:12:51.245 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:12:51 smithi082 ceph-mon[32933]: from='mgr.? 172.21.15.138:0/2504648021' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-05-03T21:12:51.245 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:12:51 smithi082 ceph-mon[32933]: from='mgr.? 172.21.15.138:0/2504648021' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-05-03T21:12:51.246 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:12:51 smithi082 ceph-mon[32933]: from='mgr.? 172.21.15.138:0/2504648021' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-05-03T21:12:51.246 INFO:journalctl@ceph.mgr.y.smithi082.stdout:May 03 21:12:50 smithi082 ceph-713bc57a-0991-11ef-bc97-c7b262605968-mgr-y[34323]: 2024-05-03T21:12:50.953+0000 7f5320d77200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-05-03T21:12:51.246 INFO:journalctl@ceph.mgr.y.smithi082.stdout:May 03 21:12:51 smithi082 ceph-713bc57a-0991-11ef-bc97-c7b262605968-mgr-y[34323]: 2024-05-03T21:12:51.049+0000 7f5320d77200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-05-03T21:12:51.473 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:12:51 smithi067 ceph-mon[29730]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-03T21:12:51.474 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:12:51 smithi067 ceph-mon[29730]: Standby manager daemon z started 2024-05-03T21:12:51.474 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:12:51 smithi067 ceph-mon[29730]: from='mgr.? 172.21.15.138:0/2504648021' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-05-03T21:12:51.474 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:12:51 smithi067 ceph-mon[29730]: from='mgr.? 172.21.15.138:0/2504648021' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-05-03T21:12:51.474 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:12:51 smithi067 ceph-mon[29730]: from='mgr.? 172.21.15.138:0/2504648021' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-05-03T21:12:51.474 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:12:51 smithi067 ceph-mon[29730]: from='mgr.? 172.21.15.138:0/2504648021' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-05-03T21:12:51.574 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:51 smithi138 ceph-mon[33110]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-03T21:12:51.574 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:51 smithi138 ceph-mon[33110]: Standby manager daemon z started 2024-05-03T21:12:51.575 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:51 smithi138 ceph-mon[33110]: from='mgr.? 172.21.15.138:0/2504648021' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-05-03T21:12:51.575 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:51 smithi138 ceph-mon[33110]: from='mgr.? 172.21.15.138:0/2504648021' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-05-03T21:12:51.575 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:51 smithi138 ceph-mon[33110]: from='mgr.? 172.21.15.138:0/2504648021' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-05-03T21:12:51.575 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:51 smithi138 ceph-mon[33110]: from='mgr.? 172.21.15.138:0/2504648021' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-05-03T21:12:51.745 INFO:journalctl@ceph.mgr.y.smithi082.stdout:May 03 21:12:51 smithi082 ceph-713bc57a-0991-11ef-bc97-c7b262605968-mgr-y[34323]: 2024-05-03T21:12:51.417+0000 7f5320d77200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-05-03T21:12:52.103 INFO:journalctl@ceph.mgr.y.smithi082.stdout:May 03 21:12:51 smithi082 ceph-713bc57a-0991-11ef-bc97-c7b262605968-mgr-y[34323]: 2024-05-03T21:12:51.769+0000 7f5320d77200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-05-03T21:12:52.103 INFO:journalctl@ceph.mgr.y.smithi082.stdout:May 03 21:12:51 smithi082 ceph-713bc57a-0991-11ef-bc97-c7b262605968-mgr-y[34323]: 2024-05-03T21:12:51.860+0000 7f5320d77200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-05-03T21:12:52.471 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:12:52 smithi067 ceph-mon[29730]: mgrmap e14: x(active, since 77s), standbys: z 2024-05-03T21:12:52.471 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:12:52 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-05-03T21:12:52.495 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:12:52 smithi082 ceph-mon[32933]: mgrmap e14: x(active, since 77s), standbys: z 2024-05-03T21:12:52.497 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:12:52 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-05-03T21:12:52.497 INFO:journalctl@ceph.mgr.y.smithi082.stdout:May 03 21:12:52 smithi082 ceph-713bc57a-0991-11ef-bc97-c7b262605968-mgr-y[34323]: 2024-05-03T21:12:52.101+0000 7f5320d77200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-05-03T21:12:52.498 INFO:journalctl@ceph.mgr.y.smithi082.stdout:May 03 21:12:52 smithi082 ceph-713bc57a-0991-11ef-bc97-c7b262605968-mgr-y[34323]: 2024-05-03T21:12:52.235+0000 7f5320d77200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-05-03T21:12:52.574 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:52 smithi138 ceph-mon[33110]: mgrmap e14: x(active, since 77s), standbys: z 2024-05-03T21:12:52.574 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:52 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-05-03T21:12:52.995 INFO:journalctl@ceph.mgr.y.smithi082.stdout:May 03 21:12:52 smithi082 ceph-713bc57a-0991-11ef-bc97-c7b262605968-mgr-y[34323]: 2024-05-03T21:12:52.544+0000 7f5320d77200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-05-03T21:12:52.995 INFO:journalctl@ceph.mgr.y.smithi082.stdout:May 03 21:12:52 smithi082 ceph-713bc57a-0991-11ef-bc97-c7b262605968-mgr-y[34323]: 2024-05-03T21:12:52.657+0000 7f5320d77200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-05-03T21:12:52.995 INFO:journalctl@ceph.mgr.y.smithi082.stdout:May 03 21:12:52 smithi082 ceph-713bc57a-0991-11ef-bc97-c7b262605968-mgr-y[34323]: 2024-05-03T21:12:52.747+0000 7f5320d77200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-05-03T21:12:53.221 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:12:53 smithi067 ceph-mon[29730]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-03T21:12:53.495 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:12:53 smithi082 ceph-mon[32933]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-03T21:12:53.495 INFO:journalctl@ceph.mgr.y.smithi082.stdout:May 03 21:12:53 smithi082 ceph-713bc57a-0991-11ef-bc97-c7b262605968-mgr-y[34323]: 2024-05-03T21:12:53.347+0000 7f5320d77200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-05-03T21:12:53.574 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:53 smithi138 ceph-mon[33110]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-03T21:12:53.995 INFO:journalctl@ceph.mgr.y.smithi082.stdout:May 03 21:12:53 smithi082 ceph-713bc57a-0991-11ef-bc97-c7b262605968-mgr-y[34323]: 2024-05-03T21:12:53.563+0000 7f5320d77200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-05-03T21:12:54.152 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:12:54.149+0000 7f63f3fff700 1 -- 172.21.15.67:0/159173635 <== mon.2 v2:172.21.15.82:3300/0 8 ==== mgrmap(e 15) ==== 149596+0+0 (secure 0 0 0) 0x7f6404011390 con 0x7f640c103a20 2024-05-03T21:12:54.308 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:12:54 smithi067 ceph-mon[29730]: Standby manager daemon y started 2024-05-03T21:12:54.309 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:12:54 smithi067 ceph-mon[29730]: from='mgr.? 172.21.15.82:0/393119178' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-05-03T21:12:54.309 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:12:54 smithi067 ceph-mon[29730]: from='mgr.? 172.21.15.82:0/393119178' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-05-03T21:12:54.309 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:12:54 smithi067 ceph-mon[29730]: from='mgr.? 172.21.15.82:0/393119178' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-05-03T21:12:54.309 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:12:54 smithi067 ceph-mon[29730]: from='mgr.? 172.21.15.82:0/393119178' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-05-03T21:12:54.495 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:12:54 smithi082 ceph-mon[32933]: Standby manager daemon y started 2024-05-03T21:12:54.495 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:12:54 smithi082 ceph-mon[32933]: from='mgr.? 172.21.15.82:0/393119178' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-05-03T21:12:54.495 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:12:54 smithi082 ceph-mon[32933]: from='mgr.? 172.21.15.82:0/393119178' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-05-03T21:12:54.495 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:12:54 smithi082 ceph-mon[32933]: from='mgr.? 172.21.15.82:0/393119178' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-05-03T21:12:54.495 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:12:54 smithi082 ceph-mon[32933]: from='mgr.? 172.21.15.82:0/393119178' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-05-03T21:12:54.574 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:54 smithi138 ceph-mon[33110]: Standby manager daemon y started 2024-05-03T21:12:54.574 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:54 smithi138 ceph-mon[33110]: from='mgr.? 172.21.15.82:0/393119178' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-05-03T21:12:54.574 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:54 smithi138 ceph-mon[33110]: from='mgr.? 172.21.15.82:0/393119178' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-05-03T21:12:54.574 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:54 smithi138 ceph-mon[33110]: from='mgr.? 172.21.15.82:0/393119178' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-05-03T21:12:54.574 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:54 smithi138 ceph-mon[33110]: from='mgr.? 172.21.15.82:0/393119178' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-05-03T21:12:55.398 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:12:55 smithi067 ceph-mon[29730]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-03T21:12:55.398 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:12:55 smithi067 ceph-mon[29730]: mgrmap e15: x(active, since 80s), standbys: z, y 2024-05-03T21:12:55.398 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:12:55 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-05-03T21:12:55.398 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:12:55 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:12:55.398 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:12:55 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:12:55.398 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:12:55 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T21:12:55.398 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:12:55 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T21:12:55.398 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:12:55 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T21:12:55.399 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:12:55 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:12:55.495 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:12:55 smithi082 ceph-mon[32933]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-03T21:12:55.495 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:12:55 smithi082 ceph-mon[32933]: mgrmap e15: x(active, since 80s), standbys: z, y 2024-05-03T21:12:55.495 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:12:55 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-05-03T21:12:55.495 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:12:55 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:12:55.495 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:12:55 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:12:55.496 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:12:55 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T21:12:55.496 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:12:55 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T21:12:55.496 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:12:55 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T21:12:55.496 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:12:55 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:12:55.574 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:55 smithi138 ceph-mon[33110]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-03T21:12:55.574 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:55 smithi138 ceph-mon[33110]: mgrmap e15: x(active, since 80s), standbys: z, y 2024-05-03T21:12:55.574 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:55 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-05-03T21:12:55.574 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:55 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:12:55.574 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:55 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:12:55.575 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:55 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T21:12:55.575 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:55 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T21:12:55.575 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:55 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T21:12:55.575 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:55 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:12:57.429 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:12:57 smithi067 ceph-mon[29730]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-03T21:12:57.495 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:12:57 smithi082 ceph-mon[32933]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-03T21:12:57.574 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:57 smithi138 ceph-mon[33110]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-03T21:12:58.352 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:12:58 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-05-03T21:12:58.352 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:12:58 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T21:12:58.495 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:12:58 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-05-03T21:12:58.495 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:12:58 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T21:12:58.574 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:58 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-05-03T21:12:58.574 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:58 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T21:12:59.319 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:12:59 smithi067 ceph-mon[29730]: Deploying daemon osd.0 on smithi067 2024-05-03T21:12:59.319 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:12:59 smithi067 ceph-mon[29730]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-03T21:12:59.495 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:12:59 smithi082 ceph-mon[32933]: Deploying daemon osd.0 on smithi067 2024-05-03T21:12:59.496 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:12:59 smithi082 ceph-mon[32933]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-03T21:12:59.574 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:59 smithi138 ceph-mon[33110]: Deploying daemon osd.0 on smithi067 2024-05-03T21:12:59.574 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:12:59 smithi138 ceph-mon[33110]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-03T21:13:01.284 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:13:01 smithi067 ceph-mon[29730]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-03T21:13:01.495 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:13:01 smithi082 ceph-mon[32933]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-03T21:13:01.574 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:13:01 smithi138 ceph-mon[33110]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-03T21:13:03.471 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:13:03 smithi067 ceph-mon[29730]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-03T21:13:03.471 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:13:03 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:13:03.471 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:13:03 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:13:03.471 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:13:03 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T21:13:03.495 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:13:03 smithi082 ceph-mon[32933]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-03T21:13:03.495 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:13:03 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:13:03.495 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:13:03 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:13:03.495 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:13:03 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T21:13:03.574 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:13:03 smithi138 ceph-mon[33110]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-03T21:13:03.574 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:13:03 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:13:03.574 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:13:03 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:13:03.574 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:13:03 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T21:13:05.423 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:13:05 smithi067 ceph-mon[29730]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-03T21:13:05.495 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:13:05 smithi082 ceph-mon[32933]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-03T21:13:05.574 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:13:05 smithi138 ceph-mon[33110]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-03T21:13:06.656 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:13:06.654+0000 7f63f3fff700 1 -- 172.21.15.67:0/159173635 <== mgr.14150 v2:172.21.15.67:6800/604451592 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f640c0ff200 con 0x7f63f403e750 2024-05-03T21:13:06.656 INFO:teuthology.orchestra.run.smithi067.stdout:Created osd(s) 0 on host 'smithi067' 2024-05-03T21:13:06.658 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:13:06.656+0000 7f6411620700 1 -- 172.21.15.67:0/159173635 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f63f403e750 msgr2=0x7f63f4040c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:13:06.658 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:13:06.656+0000 7f6411620700 1 --2- 172.21.15.67:0/159173635 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f63f403e750 0x7f63f4040c10 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f640c06ba30 tx=0x7f63fc007400 comp rx=0 tx=0).stop 2024-05-03T21:13:06.658 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:13:06.656+0000 7f6411620700 1 -- 172.21.15.67:0/159173635 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f640c103a20 msgr2=0x7f640c1a0cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:13:06.658 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:13:06.656+0000 7f6411620700 1 --2- 172.21.15.67:0/159173635 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f640c103a20 0x7f640c1a0cf0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f6404006070 tx=0x7f6404007d10 comp rx=0 tx=0).stop 2024-05-03T21:13:06.658 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:13:06.656+0000 7f6411620700 1 -- 172.21.15.67:0/159173635 shutdown_connections 2024-05-03T21:13:06.658 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:13:06.656+0000 7f6411620700 1 --2- 172.21.15.67:0/159173635 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f63f403e750 0x7f63f4040c10 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:13:06.659 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:13:06.656+0000 7f6411620700 1 --2- 172.21.15.67:0/159173635 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f640c103a20 0x7f640c1a0cf0 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:13:06.659 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:13:06.656+0000 7f6411620700 1 --2- 172.21.15.67:0/159173635 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f640c102680 0x7f640c199730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:13:06.659 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:13:06.656+0000 7f6411620700 1 --2- 172.21.15.67:0/159173635 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f640c103040 0x7f640c199c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:13:06.659 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:13:06.656+0000 7f6411620700 1 -- 172.21.15.67:0/159173635 >> 172.21.15.67:0/159173635 conn(0x7f640c0fd110 msgr2=0x7f640c0feaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T21:13:06.659 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:13:06.656+0000 7f6411620700 1 -- 172.21.15.67:0/159173635 shutdown_connections 2024-05-03T21:13:06.659 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:13:06.656+0000 7f6411620700 1 -- 172.21.15.67:0/159173635 wait complete. 2024-05-03T21:13:06.965 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:13:06 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:13:06.965 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:13:06 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:13:06.965 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:13:06 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T21:13:06.965 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:13:06 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T21:13:06.965 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:13:06 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:13:06.965 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:13:06 smithi067 ceph-mon[29730]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-03T21:13:06.966 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:13:06 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:13:06.966 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:13:06 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:13:06.966 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:13:06 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T21:13:06.995 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:13:06 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:13:06.995 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:13:06 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:13:06.995 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:13:06 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T21:13:06.995 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:13:06 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T21:13:06.995 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:13:06 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:13:06.995 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:13:06 smithi082 ceph-mon[32933]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-03T21:13:06.995 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:13:06 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:13:06.996 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:13:06 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:13:06.996 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:13:06 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T21:13:07.074 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:13:06 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:13:07.075 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:13:06 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:13:07.075 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:13:06 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T21:13:07.075 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:13:06 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T21:13:07.075 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:13:06 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:13:07.075 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:13:06 smithi138 ceph-mon[33110]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-03T21:13:07.075 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:13:06 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:13:07.075 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:13:06 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:13:07.075 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:13:06 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T21:13:07.267 DEBUG:teuthology.orchestra.run.smithi067:osd.0> sudo journalctl -f -n 0 -u ceph-713bc57a-0991-11ef-bc97-c7b262605968@osd.0.service 2024-05-03T21:13:07.269 INFO:tasks.cephadm:Deploying osd.1 on smithi082 with /dev/vg_nvme/lv_4... 2024-05-03T21:13:07.269 DEBUG:teuthology.orchestra.run.smithi082:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9a047fe7599a549042d06a3fb5b6813a203eabea shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 713bc57a-0991-11ef-bc97-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-05-03T21:13:07.514 INFO:teuthology.orchestra.run.smithi082.stderr:Inferring config /var/lib/ceph/713bc57a-0991-11ef-bc97-c7b262605968/mon.b/config 2024-05-03T21:13:07.918 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:13:07 smithi082 ceph-mon[32933]: from='osd.0 [v2:172.21.15.67:6802/3692483807,v1:172.21.15.67:6803/3692483807]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-05-03T21:13:07.971 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:13:07 smithi067 ceph-mon[29730]: from='osd.0 [v2:172.21.15.67:6802/3692483807,v1:172.21.15.67:6803/3692483807]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-05-03T21:13:08.074 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:13:07 smithi138 ceph-mon[33110]: from='osd.0 [v2:172.21.15.67:6802/3692483807,v1:172.21.15.67:6803/3692483807]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-05-03T21:13:08.772 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:13:08 smithi082 ceph-mon[32933]: from='osd.0 [v2:172.21.15.67:6802/3692483807,v1:172.21.15.67:6803/3692483807]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-05-03T21:13:08.772 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:13:08 smithi082 ceph-mon[32933]: osdmap e6: 1 total, 0 up, 1 in 2024-05-03T21:13:08.772 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:13:08 smithi082 ceph-mon[32933]: from='osd.0 [v2:172.21.15.67:6802/3692483807,v1:172.21.15.67:6803/3692483807]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi067", "root=default"]}]: dispatch 2024-05-03T21:13:08.772 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:13:08 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-03T21:13:08.772 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:13:08 smithi082 ceph-mon[32933]: pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-03T21:13:08.971 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:13:08 smithi067 ceph-mon[29730]: from='osd.0 [v2:172.21.15.67:6802/3692483807,v1:172.21.15.67:6803/3692483807]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-05-03T21:13:08.971 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:13:08 smithi067 ceph-mon[29730]: osdmap e6: 1 total, 0 up, 1 in 2024-05-03T21:13:08.971 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:13:08 smithi067 ceph-mon[29730]: from='osd.0 [v2:172.21.15.67:6802/3692483807,v1:172.21.15.67:6803/3692483807]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi067", "root=default"]}]: dispatch 2024-05-03T21:13:08.971 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:13:08 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-03T21:13:08.971 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:13:08 smithi067 ceph-mon[29730]: pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-03T21:13:09.074 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:13:08 smithi138 ceph-mon[33110]: from='osd.0 [v2:172.21.15.67:6802/3692483807,v1:172.21.15.67:6803/3692483807]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-05-03T21:13:09.075 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:13:08 smithi138 ceph-mon[33110]: osdmap e6: 1 total, 0 up, 1 in 2024-05-03T21:13:09.075 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:13:08 smithi138 ceph-mon[33110]: from='osd.0 [v2:172.21.15.67:6802/3692483807,v1:172.21.15.67:6803/3692483807]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi067", "root=default"]}]: dispatch 2024-05-03T21:13:09.075 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:13:08 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-03T21:13:09.075 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:13:08 smithi138 ceph-mon[33110]: pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-03T21:13:09.303 INFO:teuthology.orchestra.run.smithi082.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-05-03T21:13:09.303 INFO:teuthology.orchestra.run.smithi082.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-05-03T21:13:09.303 INFO:teuthology.orchestra.run.smithi082.stderr: stderr: 10+0 records in 2024-05-03T21:13:09.303 INFO:teuthology.orchestra.run.smithi082.stderr:10+0 records out 2024-05-03T21:13:09.303 INFO:teuthology.orchestra.run.smithi082.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0128698 s, 815 MB/s 2024-05-03T21:13:09.303 INFO:teuthology.orchestra.run.smithi082.stderr:--> Zapping successful for: 2024-05-03T21:13:09.936 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:13:09 smithi082 ceph-mon[32933]: from='osd.0 [v2:172.21.15.67:6802/3692483807,v1:172.21.15.67:6803/3692483807]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi067", "root=default"]}]': finished 2024-05-03T21:13:09.937 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:13:09 smithi082 ceph-mon[32933]: osdmap e7: 1 total, 0 up, 1 in 2024-05-03T21:13:09.937 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:13:09 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-03T21:13:09.937 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:13:09 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-03T21:13:09.937 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:13:09 smithi082 ceph-mon[32933]: from='osd.0 [v2:172.21.15.67:6802/3692483807,v1:172.21.15.67:6803/3692483807]' entity='osd.0' 2024-05-03T21:13:09.964 DEBUG:teuthology.orchestra.run.smithi082:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9a047fe7599a549042d06a3fb5b6813a203eabea shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 713bc57a-0991-11ef-bc97-c7b262605968 -- ceph orch daemon add osd smithi082:vg_nvme/lv_4 2024-05-03T21:13:09.971 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:13:09 smithi067 ceph-mon[29730]: from='osd.0 [v2:172.21.15.67:6802/3692483807,v1:172.21.15.67:6803/3692483807]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi067", "root=default"]}]': finished 2024-05-03T21:13:09.971 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:13:09 smithi067 ceph-mon[29730]: osdmap e7: 1 total, 0 up, 1 in 2024-05-03T21:13:09.971 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:13:09 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-03T21:13:09.971 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:13:09 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-03T21:13:09.971 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:13:09 smithi067 ceph-mon[29730]: from='osd.0 [v2:172.21.15.67:6802/3692483807,v1:172.21.15.67:6803/3692483807]' entity='osd.0' 2024-05-03T21:13:09.971 INFO:journalctl@ceph.osd.0.smithi067.stdout:May 03 21:13:09 smithi067 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-0[41836]: 2024-05-03T21:13:09.722+0000 7feabf08b700 -1 osd.0 0 waiting for initial osdmap 2024-05-03T21:13:09.971 INFO:journalctl@ceph.osd.0.smithi067.stdout:May 03 21:13:09 smithi067 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-0[41836]: 2024-05-03T21:13:09.829+0000 7feab6e76700 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-05-03T21:13:10.074 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:13:09 smithi138 ceph-mon[33110]: from='osd.0 [v2:172.21.15.67:6802/3692483807,v1:172.21.15.67:6803/3692483807]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi067", "root=default"]}]': finished 2024-05-03T21:13:10.074 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:13:09 smithi138 ceph-mon[33110]: osdmap e7: 1 total, 0 up, 1 in 2024-05-03T21:13:10.074 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:13:09 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-03T21:13:10.075 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:13:09 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-03T21:13:10.075 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:13:09 smithi138 ceph-mon[33110]: from='osd.0 [v2:172.21.15.67:6802/3692483807,v1:172.21.15.67:6803/3692483807]' entity='osd.0' 2024-05-03T21:13:10.203 INFO:teuthology.orchestra.run.smithi082.stderr:Inferring config /var/lib/ceph/713bc57a-0991-11ef-bc97-c7b262605968/mon.b/config 2024-05-03T21:13:10.908 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:13:10 smithi082 ceph-mon[32933]: purged_snaps scrub starts 2024-05-03T21:13:10.909 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:13:10 smithi082 ceph-mon[32933]: purged_snaps scrub ok 2024-05-03T21:13:10.909 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:13:10 smithi082 ceph-mon[32933]: pgmap v45: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-03T21:13:10.909 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:13:10 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-03T21:13:10.909 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:13:10 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:13:10.909 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:13:10 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:13:10.909 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:13:10 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-05-03T21:13:10.909 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:13:10 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:13:10.909 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:13:10 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T21:13:10.909 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:13:10 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T21:13:10.909 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:13:10 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:13:11.074 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:13:10 smithi138 ceph-mon[33110]: purged_snaps scrub starts 2024-05-03T21:13:11.075 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:13:10 smithi138 ceph-mon[33110]: purged_snaps scrub ok 2024-05-03T21:13:11.075 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:13:10 smithi138 ceph-mon[33110]: pgmap v45: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-03T21:13:11.075 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:13:10 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-03T21:13:11.075 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:13:10 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:13:11.075 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:13:10 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:13:11.075 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:13:10 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-05-03T21:13:11.075 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:13:10 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:13:11.075 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:13:10 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T21:13:11.075 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:13:10 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T21:13:11.075 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:13:10 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:13:11.221 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:13:10 smithi067 ceph-mon[29730]: purged_snaps scrub starts 2024-05-03T21:13:11.221 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:13:10 smithi067 ceph-mon[29730]: purged_snaps scrub ok 2024-05-03T21:13:11.221 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:13:10 smithi067 ceph-mon[29730]: pgmap v45: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-03T21:13:11.221 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:13:10 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-03T21:13:11.221 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:13:10 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:13:11.221 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:13:10 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:13:11.221 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:13:10 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-05-03T21:13:11.221 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:13:10 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:13:11.221 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:13:10 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T21:13:11.222 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:13:10 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T21:13:11.222 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:13:10 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:13:11.646 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T21:13:11.641+0000 7f577eb01700 1 -- 172.21.15.82:0/557636498 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f57781017f0 msgr2=0x7f5778101c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:13:11.646 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T21:13:11.641+0000 7f577eb01700 1 --2- 172.21.15.82:0/557636498 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f57781017f0 0x7f5778101c10 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f576c009990 tx=0x7f576c009ca0 comp rx=0 tx=0).stop 2024-05-03T21:13:11.646 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T21:13:11.641+0000 7f577eb01700 1 -- 172.21.15.82:0/557636498 shutdown_connections 2024-05-03T21:13:11.646 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T21:13:11.641+0000 7f577eb01700 1 --2- 172.21.15.82:0/557636498 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f5778105170 0x7f577810ba20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:13:11.646 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T21:13:11.641+0000 7f577eb01700 1 --2- 172.21.15.82:0/557636498 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f57781017f0 0x7f5778101c10 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:13:11.647 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T21:13:11.641+0000 7f577eb01700 1 --2- 172.21.15.82:0/557636498 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f57781047d0 0x7f5778104c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:13:11.647 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T21:13:11.642+0000 7f577eb01700 1 -- 172.21.15.82:0/557636498 >> 172.21.15.82:0/557636498 conn(0x7f57780fd150 msgr2=0x7f57780ff590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T21:13:11.647 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T21:13:11.642+0000 7f577eb01700 1 -- 172.21.15.82:0/557636498 shutdown_connections 2024-05-03T21:13:11.647 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T21:13:11.642+0000 7f577eb01700 1 -- 172.21.15.82:0/557636498 wait complete. 2024-05-03T21:13:11.647 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T21:13:11.642+0000 7f577eb01700 1 Processor -- start 2024-05-03T21:13:11.647 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T21:13:11.642+0000 7f577eb01700 1 -- start start 2024-05-03T21:13:11.647 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T21:13:11.642+0000 7f577eb01700 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f57781017f0 0x7f57781997d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:13:11.647 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T21:13:11.642+0000 7f577eb01700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f57781047d0 0x7f5778199d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:13:11.647 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T21:13:11.642+0000 7f577eb01700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5778105170 0x7f57781a0d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:13:11.647 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T21:13:11.642+0000 7f577eb01700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f577810f240 con 0x7f5778105170 2024-05-03T21:13:11.647 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T21:13:11.642+0000 7f577eb01700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f577810f0c0 con 0x7f57781047d0 2024-05-03T21:13:11.647 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T21:13:11.642+0000 7f577eb01700 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f577810f3c0 con 0x7f57781017f0 2024-05-03T21:13:11.647 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T21:13:11.642+0000 7f5777fff700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f57781047d0 0x7f5778199d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:13:11.648 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T21:13:11.642+0000 7f5777fff700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f57781047d0 0x7f5778199d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.82:39740/0 (socket says 172.21.15.82:39740) 2024-05-03T21:13:11.648 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T21:13:11.642+0000 7f5777fff700 1 -- 172.21.15.82:0/3113341796 learned_addr learned my addr 172.21.15.82:0/3113341796 (peer_addr_for_me v2:172.21.15.82:0/0) 2024-05-03T21:13:11.648 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T21:13:11.643+0000 7f577c89d700 1 --2- 172.21.15.82:0/3113341796 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f57781017f0 0x7f57781997d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:13:11.648 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T21:13:11.643+0000 7f577d09e700 1 --2- 172.21.15.82:0/3113341796 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5778105170 0x7f57781a0d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:13:11.648 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T21:13:11.643+0000 7f5777fff700 1 -- 172.21.15.82:0/3113341796 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f57781017f0 msgr2=0x7f57781997d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:13:11.648 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T21:13:11.643+0000 7f5777fff700 1 --2- 172.21.15.82:0/3113341796 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f57781017f0 0x7f57781997d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:13:11.648 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T21:13:11.643+0000 7f5777fff700 1 -- 172.21.15.82:0/3113341796 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5778105170 msgr2=0x7f57781a0d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:13:11.648 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T21:13:11.643+0000 7f5777fff700 1 --2- 172.21.15.82:0/3113341796 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5778105170 0x7f57781a0d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:13:11.648 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T21:13:11.643+0000 7f5777fff700 1 -- 172.21.15.82:0/3113341796 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f57781a1430 con 0x7f57781047d0 2024-05-03T21:13:11.648 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T21:13:11.643+0000 7f5777fff700 1 --2- 172.21.15.82:0/3113341796 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f57781047d0 0x7f5778199d10 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f576400d950 tx=0x7f576400dc60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T21:13:11.648 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T21:13:11.643+0000 7f5775ffb700 1 -- 172.21.15.82:0/3113341796 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5764016070 con 0x7f57781047d0 2024-05-03T21:13:11.648 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T21:13:11.643+0000 7f577eb01700 1 -- 172.21.15.82:0/3113341796 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f57781a1720 con 0x7f57781047d0 2024-05-03T21:13:11.649 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T21:13:11.643+0000 7f5775ffb700 1 -- 172.21.15.82:0/3113341796 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(28 keys) ==== 1170+0+0 (secure 0 0 0) 0x7f576400fd90 con 0x7f57781047d0 2024-05-03T21:13:11.649 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T21:13:11.643+0000 7f5775ffb700 1 -- 172.21.15.82:0/3113341796 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f576400e610 con 0x7f57781047d0 2024-05-03T21:13:11.649 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T21:13:11.643+0000 7f577eb01700 1 -- 172.21.15.82:0/3113341796 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f57781a1c60 con 0x7f57781047d0 2024-05-03T21:13:11.649 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T21:13:11.644+0000 7f577eb01700 1 -- 172.21.15.82:0/3113341796 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f577806a190 con 0x7f57781047d0 2024-05-03T21:13:11.650 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T21:13:11.645+0000 7f5775ffb700 1 -- 172.21.15.82:0/3113341796 <== mon.2 v2:172.21.15.82:3300/0 4 ==== mgrmap(e 15) ==== 149596+0+0 (secure 0 0 0) 0x7f5764019070 con 0x7f57781047d0 2024-05-03T21:13:11.650 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T21:13:11.646+0000 7f5775ffb700 1 --2- 172.21.15.82:0/3113341796 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f57680b27f0 0x7f57680b4cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:13:11.650 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T21:13:11.646+0000 7f5775ffb700 1 -- 172.21.15.82:0/3113341796 <== mon.2 v2:172.21.15.82:3300/0 5 ==== osd_map(8..8 src has 1..8) ==== 1767+0+0 (secure 0 0 0) 0x7f57640e01f0 con 0x7f57781047d0 2024-05-03T21:13:11.650 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T21:13:11.647+0000 7f577c89d700 1 --2- 172.21.15.82:0/3113341796 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f57680b27f0 0x7f57680b4cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:13:11.650 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T21:13:11.647+0000 7f577c89d700 1 --2- 172.21.15.82:0/3113341796 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f57680b27f0 0x7f57680b4cb0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f576c003860 tx=0x7f576c003930 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T21:13:11.653 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T21:13:11.652+0000 7f5775ffb700 1 -- 172.21.15.82:0/3113341796 <== mon.2 v2:172.21.15.82:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191839 (secure 0 0 0) 0x7f57640a9980 con 0x7f57781047d0 2024-05-03T21:13:11.866 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T21:13:11.864+0000 7f577eb01700 1 -- 172.21.15.82:0/3113341796 --> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi082:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) -- 0x7f57780638e0 con 0x7f57680b27f0 2024-05-03T21:13:12.074 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:13:11 smithi138 ceph-mon[33110]: Detected new or changed devices on smithi067 2024-05-03T21:13:12.074 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:13:11 smithi138 ceph-mon[33110]: Adjusting osd_memory_target on smithi067 to 17088M 2024-05-03T21:13:12.074 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:13:11 smithi138 ceph-mon[33110]: osd.0 [v2:172.21.15.67:6802/3692483807,v1:172.21.15.67:6803/3692483807] boot 2024-05-03T21:13:12.075 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:13:11 smithi138 ceph-mon[33110]: osdmap e8: 1 total, 1 up, 1 in 2024-05-03T21:13:12.075 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:13:11 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-03T21:13:12.221 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:13:11 smithi067 ceph-mon[29730]: Detected new or changed devices on smithi067 2024-05-03T21:13:12.221 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:13:11 smithi067 ceph-mon[29730]: Adjusting osd_memory_target on smithi067 to 17088M 2024-05-03T21:13:12.221 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:13:11 smithi067 ceph-mon[29730]: osd.0 [v2:172.21.15.67:6802/3692483807,v1:172.21.15.67:6803/3692483807] boot 2024-05-03T21:13:12.221 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:13:11 smithi067 ceph-mon[29730]: osdmap e8: 1 total, 1 up, 1 in 2024-05-03T21:13:12.221 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:13:11 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-03T21:13:12.245 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:13:11 smithi082 ceph-mon[32933]: Detected new or changed devices on smithi067 2024-05-03T21:13:12.245 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:13:11 smithi082 ceph-mon[32933]: Adjusting osd_memory_target on smithi067 to 17088M 2024-05-03T21:13:12.245 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:13:11 smithi082 ceph-mon[32933]: osd.0 [v2:172.21.15.67:6802/3692483807,v1:172.21.15.67:6803/3692483807] boot 2024-05-03T21:13:12.245 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:13:11 smithi082 ceph-mon[32933]: osdmap e8: 1 total, 1 up, 1 in 2024-05-03T21:13:12.245 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:13:11 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-03T21:13:13.074 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:13:12 smithi138 ceph-mon[33110]: pgmap v47: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-03T21:13:13.074 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:13:12 smithi138 ceph-mon[33110]: from='client.24140 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi082:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-05-03T21:13:13.074 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:13:12 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-03T21:13:13.074 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:13:12 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-03T21:13:13.075 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:13:12 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T21:13:13.109 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:13:12 smithi082 ceph-mon[32933]: pgmap v47: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-03T21:13:13.109 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:13:12 smithi082 ceph-mon[32933]: from='client.24140 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi082:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-05-03T21:13:13.109 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:13:12 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-03T21:13:13.109 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:13:12 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-03T21:13:13.109 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:13:12 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T21:13:13.221 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:13:12 smithi067 ceph-mon[29730]: pgmap v47: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-03T21:13:13.221 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:13:12 smithi067 ceph-mon[29730]: from='client.24140 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi082:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-05-03T21:13:13.221 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:13:12 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-03T21:13:13.221 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:13:12 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-03T21:13:13.221 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:13:12 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T21:13:14.074 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:13:13 smithi138 ceph-mon[33110]: osdmap e9: 1 total, 1 up, 1 in 2024-05-03T21:13:14.221 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:13:13 smithi067 ceph-mon[29730]: osdmap e9: 1 total, 1 up, 1 in 2024-05-03T21:13:14.245 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:13:13 smithi082 ceph-mon[32933]: osdmap e9: 1 total, 1 up, 1 in 2024-05-03T21:13:15.074 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:13:14 smithi138 ceph-mon[33110]: pgmap v49: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-05-03T21:13:15.074 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:13:14 smithi138 ceph-mon[33110]: from='client.? 172.21.15.82:0/209435309' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "71858193-05a6-4c2c-9bb7-bc3d811d634e"}]: dispatch 2024-05-03T21:13:15.074 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:13:14 smithi138 ceph-mon[33110]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "71858193-05a6-4c2c-9bb7-bc3d811d634e"}]: dispatch 2024-05-03T21:13:15.074 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:13:14 smithi138 ceph-mon[33110]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "71858193-05a6-4c2c-9bb7-bc3d811d634e"}]': finished 2024-05-03T21:13:15.075 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:13:14 smithi138 ceph-mon[33110]: osdmap e10: 2 total, 1 up, 2 in 2024-05-03T21:13:15.075 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:13:14 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-03T21:13:15.221 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:13:14 smithi067 ceph-mon[29730]: pgmap v49: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-05-03T21:13:15.221 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:13:14 smithi067 ceph-mon[29730]: from='client.? 172.21.15.82:0/209435309' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "71858193-05a6-4c2c-9bb7-bc3d811d634e"}]: dispatch 2024-05-03T21:13:15.221 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:13:14 smithi067 ceph-mon[29730]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "71858193-05a6-4c2c-9bb7-bc3d811d634e"}]: dispatch 2024-05-03T21:13:15.221 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:13:14 smithi067 ceph-mon[29730]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "71858193-05a6-4c2c-9bb7-bc3d811d634e"}]': finished 2024-05-03T21:13:15.221 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:13:14 smithi067 ceph-mon[29730]: osdmap e10: 2 total, 1 up, 2 in 2024-05-03T21:13:15.221 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:13:14 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-03T21:13:15.245 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:13:14 smithi082 ceph-mon[32933]: pgmap v49: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-05-03T21:13:15.245 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:13:14 smithi082 ceph-mon[32933]: from='client.? 172.21.15.82:0/209435309' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "71858193-05a6-4c2c-9bb7-bc3d811d634e"}]: dispatch 2024-05-03T21:13:15.245 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:13:14 smithi082 ceph-mon[32933]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "71858193-05a6-4c2c-9bb7-bc3d811d634e"}]: dispatch 2024-05-03T21:13:15.245 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:13:14 smithi082 ceph-mon[32933]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "71858193-05a6-4c2c-9bb7-bc3d811d634e"}]': finished 2024-05-03T21:13:15.245 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:13:14 smithi082 ceph-mon[32933]: osdmap e10: 2 total, 1 up, 2 in 2024-05-03T21:13:15.245 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:13:14 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-03T21:13:16.074 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:13:15 smithi138 ceph-mon[33110]: from='client.? 172.21.15.82:0/513615830' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-03T21:13:16.221 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:13:15 smithi067 ceph-mon[29730]: from='client.? 172.21.15.82:0/513615830' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-03T21:13:16.245 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:13:15 smithi082 ceph-mon[32933]: from='client.? 172.21.15.82:0/513615830' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-03T21:13:17.074 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:13:16 smithi138 ceph-mon[33110]: pgmap v51: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-05-03T21:13:17.221 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:13:16 smithi067 ceph-mon[29730]: pgmap v51: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-05-03T21:13:17.245 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:13:16 smithi082 ceph-mon[32933]: pgmap v51: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-05-03T21:13:18.995 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:13:18 smithi082 ceph-mon[32933]: pgmap v52: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-05-03T21:13:19.074 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:13:18 smithi138 ceph-mon[33110]: pgmap v52: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-05-03T21:13:19.221 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:13:18 smithi067 ceph-mon[29730]: pgmap v52: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-05-03T21:13:21.074 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:13:20 smithi138 ceph-mon[33110]: pgmap v53: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-05-03T21:13:21.178 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:13:20 smithi082 ceph-mon[32933]: pgmap v53: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-05-03T21:13:21.221 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:13:20 smithi067 ceph-mon[29730]: pgmap v53: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-05-03T21:13:21.995 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:13:21 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-05-03T21:13:21.995 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:13:21 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T21:13:22.074 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:13:21 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-05-03T21:13:22.074 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:13:21 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T21:13:22.221 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:13:21 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-05-03T21:13:22.221 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:13:21 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T21:13:22.995 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:13:22 smithi082 ceph-mon[32933]: pgmap v54: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-05-03T21:13:22.995 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:13:22 smithi082 ceph-mon[32933]: Deploying daemon osd.1 on smithi082 2024-05-03T21:13:23.074 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:13:22 smithi138 ceph-mon[33110]: pgmap v54: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-05-03T21:13:23.074 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:13:22 smithi138 ceph-mon[33110]: Deploying daemon osd.1 on smithi082 2024-05-03T21:13:23.221 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:13:22 smithi067 ceph-mon[29730]: pgmap v54: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-05-03T21:13:23.221 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:13:22 smithi067 ceph-mon[29730]: Deploying daemon osd.1 on smithi082 2024-05-03T21:13:25.074 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:13:24 smithi138 ceph-mon[33110]: pgmap v55: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-05-03T21:13:25.180 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:13:24 smithi082 ceph-mon[32933]: pgmap v55: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-05-03T21:13:25.221 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:13:24 smithi067 ceph-mon[29730]: pgmap v55: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-05-03T21:13:26.929 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:13:26 smithi082 ceph-mon[32933]: pgmap v56: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-05-03T21:13:27.074 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:13:26 smithi138 ceph-mon[33110]: pgmap v56: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-05-03T21:13:27.220 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:13:26 smithi067 ceph-mon[29730]: pgmap v56: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-05-03T21:13:28.245 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:13:27 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:13:28.245 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:13:27 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:13:28.245 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:13:27 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T21:13:28.326 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:13:27 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:13:28.326 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:13:27 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:13:28.326 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:13:27 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T21:13:28.471 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:13:27 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:13:28.471 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:13:27 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:13:28.471 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:13:27 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T21:13:28.995 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:13:28 smithi082 ceph-mon[32933]: pgmap v57: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-05-03T21:13:29.324 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:13:28 smithi138 ceph-mon[33110]: pgmap v57: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-05-03T21:13:29.471 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:13:28 smithi067 ceph-mon[29730]: pgmap v57: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-05-03T21:13:30.286 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T21:13:30.285+0000 7f5775ffb700 1 -- 172.21.15.82:0/3113341796 <== mgr.14150 v2:172.21.15.67:6800/604451592 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f57780638e0 con 0x7f57680b27f0 2024-05-03T21:13:30.287 INFO:teuthology.orchestra.run.smithi082.stdout:Created osd(s) 1 on host 'smithi082' 2024-05-03T21:13:30.289 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T21:13:30.287+0000 7f577eb01700 1 -- 172.21.15.82:0/3113341796 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f57680b27f0 msgr2=0x7f57680b4cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:13:30.289 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T21:13:30.287+0000 7f577eb01700 1 --2- 172.21.15.82:0/3113341796 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f57680b27f0 0x7f57680b4cb0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f576c003860 tx=0x7f576c003930 comp rx=0 tx=0).stop 2024-05-03T21:13:30.289 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T21:13:30.287+0000 7f577eb01700 1 -- 172.21.15.82:0/3113341796 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f57781047d0 msgr2=0x7f5778199d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:13:30.289 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T21:13:30.287+0000 7f577eb01700 1 --2- 172.21.15.82:0/3113341796 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f57781047d0 0x7f5778199d10 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f576400d950 tx=0x7f576400dc60 comp rx=0 tx=0).stop 2024-05-03T21:13:30.289 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T21:13:30.287+0000 7f577eb01700 1 -- 172.21.15.82:0/3113341796 shutdown_connections 2024-05-03T21:13:30.289 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T21:13:30.287+0000 7f577eb01700 1 --2- 172.21.15.82:0/3113341796 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f57680b27f0 0x7f57680b4cb0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:13:30.289 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T21:13:30.287+0000 7f577eb01700 1 --2- 172.21.15.82:0/3113341796 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5778105170 0x7f57781a0d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:13:30.289 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T21:13:30.287+0000 7f577eb01700 1 --2- 172.21.15.82:0/3113341796 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f57781017f0 0x7f57781997d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:13:30.289 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T21:13:30.287+0000 7f577eb01700 1 --2- 172.21.15.82:0/3113341796 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f57781047d0 0x7f5778199d10 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:13:30.290 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T21:13:30.287+0000 7f577eb01700 1 -- 172.21.15.82:0/3113341796 >> 172.21.15.82:0/3113341796 conn(0x7f57780fd150 msgr2=0x7f5778107a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T21:13:30.290 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T21:13:30.287+0000 7f577eb01700 1 -- 172.21.15.82:0/3113341796 shutdown_connections 2024-05-03T21:13:30.290 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T21:13:30.287+0000 7f577eb01700 1 -- 172.21.15.82:0/3113341796 wait complete. 2024-05-03T21:13:30.857 DEBUG:teuthology.orchestra.run.smithi082:osd.1> sudo journalctl -f -n 0 -u ceph-713bc57a-0991-11ef-bc97-c7b262605968@osd.1.service 2024-05-03T21:13:30.859 INFO:tasks.cephadm:Deploying osd.2 on smithi138 with /dev/vg_nvme/lv_4... 2024-05-03T21:13:30.859 DEBUG:teuthology.orchestra.run.smithi138:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9a047fe7599a549042d06a3fb5b6813a203eabea shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 713bc57a-0991-11ef-bc97-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-05-03T21:13:31.103 INFO:teuthology.orchestra.run.smithi138.stderr:Inferring config /var/lib/ceph/713bc57a-0991-11ef-bc97-c7b262605968/mon.c/config 2024-05-03T21:13:31.115 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:13:30 smithi082 ceph-mon[32933]: pgmap v58: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-05-03T21:13:31.115 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:13:30 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:13:31.115 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:13:30 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:13:31.115 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:13:30 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T21:13:31.115 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:13:30 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T21:13:31.115 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:13:30 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:13:31.115 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:13:30 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:13:31.115 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:13:30 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:13:31.115 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:13:30 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T21:13:31.115 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:13:30 smithi082 ceph-mon[32933]: from='osd.1 [v2:172.21.15.82:6800/1139376397,v1:172.21.15.82:6801/1139376397]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-05-03T21:13:31.115 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:13:30 smithi082 ceph-mon[32933]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-05-03T21:13:31.221 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:13:30 smithi067 ceph-mon[29730]: pgmap v58: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-05-03T21:13:31.221 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:13:30 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:13:31.221 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:13:30 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:13:31.221 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:13:30 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T21:13:31.221 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:13:30 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T21:13:31.221 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:13:30 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:13:31.221 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:13:30 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:13:31.222 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:13:30 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:13:31.222 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:13:30 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T21:13:31.222 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:13:30 smithi067 ceph-mon[29730]: from='osd.1 [v2:172.21.15.82:6800/1139376397,v1:172.21.15.82:6801/1139376397]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-05-03T21:13:31.222 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:13:30 smithi067 ceph-mon[29730]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-05-03T21:13:31.224 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:13:30 smithi138 ceph-mon[33110]: pgmap v58: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-05-03T21:13:31.224 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:13:30 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:13:31.224 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:13:30 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:13:31.224 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:13:30 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T21:13:31.224 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:13:30 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T21:13:31.225 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:13:30 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:13:31.225 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:13:30 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:13:31.225 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:13:30 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:13:31.225 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:13:30 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T21:13:31.225 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:13:30 smithi138 ceph-mon[33110]: from='osd.1 [v2:172.21.15.82:6800/1139376397,v1:172.21.15.82:6801/1139376397]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-05-03T21:13:31.225 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:13:30 smithi138 ceph-mon[33110]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-05-03T21:13:32.426 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:13:32 smithi138 ceph-mon[33110]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-05-03T21:13:32.426 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:13:32 smithi138 ceph-mon[33110]: from='osd.1 [v2:172.21.15.82:6800/1139376397,v1:172.21.15.82:6801/1139376397]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi082", "root=default"]}]: dispatch 2024-05-03T21:13:32.426 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:13:32 smithi138 ceph-mon[33110]: osdmap e11: 2 total, 1 up, 2 in 2024-05-03T21:13:32.427 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:13:32 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-03T21:13:32.427 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:13:32 smithi138 ceph-mon[33110]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi082", "root=default"]}]: dispatch 2024-05-03T21:13:32.495 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:13:32 smithi082 ceph-mon[32933]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-05-03T21:13:32.496 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:13:32 smithi082 ceph-mon[32933]: from='osd.1 [v2:172.21.15.82:6800/1139376397,v1:172.21.15.82:6801/1139376397]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi082", "root=default"]}]: dispatch 2024-05-03T21:13:32.496 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:13:32 smithi082 ceph-mon[32933]: osdmap e11: 2 total, 1 up, 2 in 2024-05-03T21:13:32.496 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:13:32 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-03T21:13:32.496 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:13:32 smithi082 ceph-mon[32933]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi082", "root=default"]}]: dispatch 2024-05-03T21:13:32.721 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:13:32 smithi067 ceph-mon[29730]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-05-03T21:13:32.721 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:13:32 smithi067 ceph-mon[29730]: from='osd.1 [v2:172.21.15.82:6800/1139376397,v1:172.21.15.82:6801/1139376397]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi082", "root=default"]}]: dispatch 2024-05-03T21:13:32.721 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:13:32 smithi067 ceph-mon[29730]: osdmap e11: 2 total, 1 up, 2 in 2024-05-03T21:13:32.721 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:13:32 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-03T21:13:32.721 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:13:32 smithi067 ceph-mon[29730]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi082", "root=default"]}]: dispatch 2024-05-03T21:13:32.972 INFO:teuthology.orchestra.run.smithi138.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-05-03T21:13:32.972 INFO:teuthology.orchestra.run.smithi138.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-05-03T21:13:32.973 INFO:teuthology.orchestra.run.smithi138.stderr: stderr: 10+0 records in 2024-05-03T21:13:32.973 INFO:teuthology.orchestra.run.smithi138.stderr:10+0 records out 2024-05-03T21:13:32.973 INFO:teuthology.orchestra.run.smithi138.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0113695 s, 922 MB/s 2024-05-03T21:13:32.973 INFO:teuthology.orchestra.run.smithi138.stderr:--> Zapping successful for: 2024-05-03T21:13:33.290 INFO:journalctl@ceph.osd.1.smithi082.stdout:May 03 21:13:33 smithi082 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-1[37530]: 2024-05-03T21:13:33.212+0000 7ff96252e700 -1 osd.1 0 waiting for initial osdmap 2024-05-03T21:13:33.574 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:13:33 smithi138 ceph-mon[33110]: pgmap v60: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-05-03T21:13:33.574 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:13:33 smithi138 ceph-mon[33110]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi082", "root=default"]}]': finished 2024-05-03T21:13:33.575 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:13:33 smithi138 ceph-mon[33110]: osdmap e12: 2 total, 1 up, 2 in 2024-05-03T21:13:33.575 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:13:33 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-03T21:13:33.575 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:13:33 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-03T21:13:33.575 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:13:33 smithi138 ceph-mon[33110]: from='osd.1 ' entity='osd.1' 2024-05-03T21:13:33.643 DEBUG:teuthology.orchestra.run.smithi138:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9a047fe7599a549042d06a3fb5b6813a203eabea shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 713bc57a-0991-11ef-bc97-c7b262605968 -- ceph orch daemon add osd smithi138:vg_nvme/lv_4 2024-05-03T21:13:33.645 INFO:journalctl@ceph.osd.1.smithi082.stdout:May 03 21:13:33 smithi082 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-1[37530]: 2024-05-03T21:13:33.345+0000 7ff959b18700 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-05-03T21:13:33.645 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:13:33 smithi082 ceph-mon[32933]: pgmap v60: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-05-03T21:13:33.645 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:13:33 smithi082 ceph-mon[32933]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi082", "root=default"]}]': finished 2024-05-03T21:13:33.645 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:13:33 smithi082 ceph-mon[32933]: osdmap e12: 2 total, 1 up, 2 in 2024-05-03T21:13:33.645 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:13:33 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-03T21:13:33.645 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:13:33 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-03T21:13:33.645 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:13:33 smithi082 ceph-mon[32933]: from='osd.1 ' entity='osd.1' 2024-05-03T21:13:33.721 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:13:33 smithi067 ceph-mon[29730]: pgmap v60: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-05-03T21:13:33.721 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:13:33 smithi067 ceph-mon[29730]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi082", "root=default"]}]': finished 2024-05-03T21:13:33.721 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:13:33 smithi067 ceph-mon[29730]: osdmap e12: 2 total, 1 up, 2 in 2024-05-03T21:13:33.721 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:13:33 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-03T21:13:33.721 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:13:33 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-03T21:13:33.721 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:13:33 smithi067 ceph-mon[29730]: from='osd.1 ' entity='osd.1' 2024-05-03T21:13:33.883 INFO:teuthology.orchestra.run.smithi138.stderr:Inferring config /var/lib/ceph/713bc57a-0991-11ef-bc97-c7b262605968/mon.c/config 2024-05-03T21:13:34.536 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:13:34 smithi138 ceph-mon[33110]: purged_snaps scrub starts 2024-05-03T21:13:34.536 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:13:34 smithi138 ceph-mon[33110]: purged_snaps scrub ok 2024-05-03T21:13:34.536 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:13:34 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-03T21:13:34.536 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:13:34 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:13:34.536 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:13:34 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:13:34.536 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:13:34 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-05-03T21:13:34.537 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:13:34 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:13:34.537 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:13:34 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T21:13:34.537 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:13:34 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T21:13:34.537 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:13:34 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:13:34.721 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:13:34 smithi067 ceph-mon[29730]: purged_snaps scrub starts 2024-05-03T21:13:34.721 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:13:34 smithi067 ceph-mon[29730]: purged_snaps scrub ok 2024-05-03T21:13:34.721 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:13:34 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-03T21:13:34.721 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:13:34 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:13:34.721 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:13:34 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:13:34.721 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:13:34 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-05-03T21:13:34.722 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:13:34 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:13:34.722 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:13:34 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T21:13:34.722 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:13:34 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T21:13:34.722 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:13:34 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:13:34.745 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:13:34 smithi082 ceph-mon[32933]: purged_snaps scrub starts 2024-05-03T21:13:34.745 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:13:34 smithi082 ceph-mon[32933]: purged_snaps scrub ok 2024-05-03T21:13:34.745 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:13:34 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-03T21:13:34.745 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:13:34 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:13:34.745 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:13:34 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:13:34.745 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:13:34 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-05-03T21:13:34.745 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:13:34 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:13:34.745 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:13:34 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T21:13:34.746 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:13:34 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T21:13:34.746 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:13:34 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:13:35.277 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:13:35.274+0000 7f30f6053700 1 -- 172.21.15.138:0/4130180280 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f30f0103fa0 msgr2=0x7f30f010a850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:13:35.277 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:13:35.274+0000 7f30f6053700 1 --2- 172.21.15.138:0/4130180280 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f30f0103fa0 0x7f30f010a850 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f30e800b0a0 tx=0x7f30e800b3b0 comp rx=0 tx=0).stop 2024-05-03T21:13:35.277 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:13:35.274+0000 7f30f6053700 1 -- 172.21.15.138:0/4130180280 shutdown_connections 2024-05-03T21:13:35.277 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:13:35.274+0000 7f30f6053700 1 --2- 172.21.15.138:0/4130180280 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f30f0102400 0x7f30f0102820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:13:35.277 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:13:35.274+0000 7f30f6053700 1 --2- 172.21.15.138:0/4130180280 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f30f0103fa0 0x7f30f010a850 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:13:35.277 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:13:35.274+0000 7f30f6053700 1 --2- 172.21.15.138:0/4130180280 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f30f0103600 0x7f30f0103a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:13:35.277 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:13:35.274+0000 7f30f6053700 1 -- 172.21.15.138:0/4130180280 >> 172.21.15.138:0/4130180280 conn(0x7f30f00fd9a0 msgr2=0x7f30f00ffde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T21:13:35.277 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:13:35.274+0000 7f30f6053700 1 -- 172.21.15.138:0/4130180280 shutdown_connections 2024-05-03T21:13:35.277 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:13:35.274+0000 7f30f6053700 1 -- 172.21.15.138:0/4130180280 wait complete. 2024-05-03T21:13:35.279 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:13:35.274+0000 7f30f6053700 1 Processor -- start 2024-05-03T21:13:35.279 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:13:35.275+0000 7f30f6053700 1 -- start start 2024-05-03T21:13:35.279 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:13:35.275+0000 7f30f6053700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f30f0102400 0x7f30f0198600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:13:35.280 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:13:35.275+0000 7f30f6053700 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f30f0103600 0x7f30f0198b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:13:35.280 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:13:35.275+0000 7f30f6053700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f30f0103fa0 0x7f30f019fbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:13:35.280 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:13:35.275+0000 7f30f6053700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f30f010e110 con 0x7f30f0103fa0 2024-05-03T21:13:35.280 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:13:35.275+0000 7f30f6053700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f30f010df90 con 0x7f30f0102400 2024-05-03T21:13:35.280 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:13:35.275+0000 7f30f6053700 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f30f010e290 con 0x7f30f0103600 2024-05-03T21:13:35.280 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:13:35.275+0000 7f30eeffd700 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f30f0103600 0x7f30f0198b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:13:35.280 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:13:35.275+0000 7f30eeffd700 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f30f0103600 0x7f30f0198b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3300/0 says I am v2:172.21.15.138:35300/0 (socket says 172.21.15.138:35300) 2024-05-03T21:13:35.280 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:13:35.275+0000 7f30eeffd700 1 -- 172.21.15.138:0/1844458068 learned_addr learned my addr 172.21.15.138:0/1844458068 (peer_addr_for_me v2:172.21.15.138:0/0) 2024-05-03T21:13:35.280 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:13:35.275+0000 7f30effff700 1 --2- 172.21.15.138:0/1844458068 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f30f0103fa0 0x7f30f019fbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:13:35.280 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:13:35.275+0000 7f30eeffd700 1 -- 172.21.15.138:0/1844458068 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f30f0102400 msgr2=0x7f30f0198600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:13:35.280 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:13:35.275+0000 7f30ef7fe700 1 --2- 172.21.15.138:0/1844458068 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f30f0102400 0x7f30f0198600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:13:35.280 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:13:35.275+0000 7f30eeffd700 1 --2- 172.21.15.138:0/1844458068 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f30f0102400 0x7f30f0198600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:13:35.280 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:13:35.275+0000 7f30eeffd700 1 -- 172.21.15.138:0/1844458068 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f30f0103fa0 msgr2=0x7f30f019fbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:13:35.280 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:13:35.275+0000 7f30eeffd700 1 --2- 172.21.15.138:0/1844458068 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f30f0103fa0 0x7f30f019fbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:13:35.281 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:13:35.275+0000 7f30eeffd700 1 -- 172.21.15.138:0/1844458068 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f30f01a0260 con 0x7f30f0103600 2024-05-03T21:13:35.281 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:13:35.275+0000 7f30effff700 1 --2- 172.21.15.138:0/1844458068 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f30f0103fa0 0x7f30f019fbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-03T21:13:35.281 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:13:35.275+0000 7f30eeffd700 1 --2- 172.21.15.138:0/1844458068 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f30f0103600 0x7f30f0198b40 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f30e40099a0 tx=0x7f30e4009d60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T21:13:35.281 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:13:35.276+0000 7f30ecff9700 1 -- 172.21.15.138:0/1844458068 <== mon.1 v2:172.21.15.138:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30e401b070 con 0x7f30f0103600 2024-05-03T21:13:35.281 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:13:35.276+0000 7f30ecff9700 1 -- 172.21.15.138:0/1844458068 <== mon.1 v2:172.21.15.138:3300/0 2 ==== config(28 keys) ==== 1170+0+0 (secure 0 0 0) 0x7f30e400d260 con 0x7f30f0103600 2024-05-03T21:13:35.281 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:13:35.276+0000 7f30ecff9700 1 -- 172.21.15.138:0/1844458068 <== mon.1 v2:172.21.15.138:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30e400cc70 con 0x7f30f0103600 2024-05-03T21:13:35.281 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:13:35.276+0000 7f30f6053700 1 -- 172.21.15.138:0/1844458068 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f30f006bc30 con 0x7f30f0103600 2024-05-03T21:13:35.281 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:13:35.276+0000 7f30f6053700 1 -- 172.21.15.138:0/1844458068 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f30f006c170 con 0x7f30f0103600 2024-05-03T21:13:35.281 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:13:35.277+0000 7f30f6053700 1 -- 172.21.15.138:0/1844458068 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f30f006a190 con 0x7f30f0103600 2024-05-03T21:13:35.282 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:13:35.278+0000 7f30ecff9700 1 -- 172.21.15.138:0/1844458068 <== mon.1 v2:172.21.15.138:3300/0 4 ==== mgrmap(e 15) ==== 149596+0+0 (secure 0 0 0) 0x7f30e401e070 con 0x7f30f0103600 2024-05-03T21:13:35.282 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:13:35.279+0000 7f30ecff9700 1 --2- 172.21.15.138:0/1844458068 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f30d80b27f0 0x7f30d80b4cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:13:35.282 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:13:35.279+0000 7f30ecff9700 1 -- 172.21.15.138:0/1844458068 <== mon.1 v2:172.21.15.138:3300/0 5 ==== osd_map(13..13 src has 1..13) ==== 2309+0+0 (secure 0 0 0) 0x7f30e40e1840 con 0x7f30f0103600 2024-05-03T21:13:35.282 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:13:35.279+0000 7f30ef7fe700 1 --2- 172.21.15.138:0/1844458068 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f30d80b27f0 0x7f30d80b4cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:13:35.282 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:13:35.279+0000 7f30ef7fe700 1 --2- 172.21.15.138:0/1844458068 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f30d80b27f0 0x7f30d80b4cb0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f30e0005950 tx=0x7f30e00058e0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T21:13:35.285 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:13:35.282+0000 7f30ecff9700 1 -- 172.21.15.138:0/1844458068 <== mon.1 v2:172.21.15.138:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191839 (secure 0 0 0) 0x7f30e40aadb0 con 0x7f30f0103600 2024-05-03T21:13:35.499 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:13:35.497+0000 7f30f6053700 1 -- 172.21.15.138:0/1844458068 --> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi138:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) -- 0x7f30f006c490 con 0x7f30d80b27f0 2024-05-03T21:13:35.574 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:13:35 smithi138 ceph-mon[33110]: Detected new or changed devices on smithi082 2024-05-03T21:13:35.574 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:13:35 smithi138 ceph-mon[33110]: Adjusting osd_memory_target on smithi082 to 17088M 2024-05-03T21:13:35.574 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:13:35 smithi138 ceph-mon[33110]: pgmap v62: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-05-03T21:13:35.574 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:13:35 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-03T21:13:35.574 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:13:35 smithi138 ceph-mon[33110]: osd.1 [v2:172.21.15.82:6800/1139376397,v1:172.21.15.82:6801/1139376397] boot 2024-05-03T21:13:35.574 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:13:35 smithi138 ceph-mon[33110]: osdmap e13: 2 total, 2 up, 2 in 2024-05-03T21:13:35.574 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:13:35 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-03T21:13:35.721 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:13:35 smithi067 ceph-mon[29730]: Detected new or changed devices on smithi082 2024-05-03T21:13:35.721 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:13:35 smithi067 ceph-mon[29730]: Adjusting osd_memory_target on smithi082 to 17088M 2024-05-03T21:13:35.721 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:13:35 smithi067 ceph-mon[29730]: pgmap v62: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-05-03T21:13:35.721 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:13:35 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-03T21:13:35.721 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:13:35 smithi067 ceph-mon[29730]: osd.1 [v2:172.21.15.82:6800/1139376397,v1:172.21.15.82:6801/1139376397] boot 2024-05-03T21:13:35.721 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:13:35 smithi067 ceph-mon[29730]: osdmap e13: 2 total, 2 up, 2 in 2024-05-03T21:13:35.721 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:13:35 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-03T21:13:35.745 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:13:35 smithi082 ceph-mon[32933]: Detected new or changed devices on smithi082 2024-05-03T21:13:35.745 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:13:35 smithi082 ceph-mon[32933]: Adjusting osd_memory_target on smithi082 to 17088M 2024-05-03T21:13:35.745 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:13:35 smithi082 ceph-mon[32933]: pgmap v62: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-05-03T21:13:35.745 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:13:35 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-03T21:13:35.745 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:13:35 smithi082 ceph-mon[32933]: osd.1 [v2:172.21.15.82:6800/1139376397,v1:172.21.15.82:6801/1139376397] boot 2024-05-03T21:13:35.745 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:13:35 smithi082 ceph-mon[32933]: osdmap e13: 2 total, 2 up, 2 in 2024-05-03T21:13:35.745 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:13:35 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-03T21:13:36.351 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:13:36 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-03T21:13:36.351 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:13:36 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-03T21:13:36.351 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:13:36 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T21:13:36.351 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:13:36 smithi138 ceph-mon[33110]: osdmap e14: 2 total, 2 up, 2 in 2024-05-03T21:13:36.721 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:13:36 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-03T21:13:36.721 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:13:36 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-03T21:13:36.721 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:13:36 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T21:13:36.721 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:13:36 smithi067 ceph-mon[29730]: osdmap e14: 2 total, 2 up, 2 in 2024-05-03T21:13:36.745 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:13:36 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-03T21:13:36.745 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:13:36 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-03T21:13:36.745 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:13:36 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T21:13:36.745 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:13:36 smithi082 ceph-mon[32933]: osdmap e14: 2 total, 2 up, 2 in 2024-05-03T21:13:37.390 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:13:37 smithi138 ceph-mon[33110]: from='client.24163 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi138:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-05-03T21:13:37.397 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:13:37 smithi138 ceph-mon[33110]: pgmap v64: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-05-03T21:13:37.721 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:13:37 smithi067 ceph-mon[29730]: from='client.24163 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi138:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-05-03T21:13:37.722 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:13:37 smithi067 ceph-mon[29730]: pgmap v64: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-05-03T21:13:37.745 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:13:37 smithi082 ceph-mon[32933]: from='client.24163 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi138:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-05-03T21:13:37.745 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:13:37 smithi082 ceph-mon[32933]: pgmap v64: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-05-03T21:13:38.574 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:13:38 smithi138 ceph-mon[33110]: from='client.? 172.21.15.138:0/1582987425' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f9ece78d-6a19-4d46-9022-6ed184074745"}]: dispatch 2024-05-03T21:13:38.574 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:13:38 smithi138 ceph-mon[33110]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f9ece78d-6a19-4d46-9022-6ed184074745"}]: dispatch 2024-05-03T21:13:38.575 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:13:38 smithi138 ceph-mon[33110]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "f9ece78d-6a19-4d46-9022-6ed184074745"}]': finished 2024-05-03T21:13:38.575 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:13:38 smithi138 ceph-mon[33110]: osdmap e15: 3 total, 2 up, 3 in 2024-05-03T21:13:38.575 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:13:38 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-03T21:13:38.721 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:13:38 smithi067 ceph-mon[29730]: from='client.? 172.21.15.138:0/1582987425' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f9ece78d-6a19-4d46-9022-6ed184074745"}]: dispatch 2024-05-03T21:13:38.721 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:13:38 smithi067 ceph-mon[29730]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f9ece78d-6a19-4d46-9022-6ed184074745"}]: dispatch 2024-05-03T21:13:38.721 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:13:38 smithi067 ceph-mon[29730]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "f9ece78d-6a19-4d46-9022-6ed184074745"}]': finished 2024-05-03T21:13:38.721 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:13:38 smithi067 ceph-mon[29730]: osdmap e15: 3 total, 2 up, 3 in 2024-05-03T21:13:38.721 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:13:38 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-03T21:13:38.745 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:13:38 smithi082 ceph-mon[32933]: from='client.? 172.21.15.138:0/1582987425' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f9ece78d-6a19-4d46-9022-6ed184074745"}]: dispatch 2024-05-03T21:13:38.745 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:13:38 smithi082 ceph-mon[32933]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f9ece78d-6a19-4d46-9022-6ed184074745"}]: dispatch 2024-05-03T21:13:38.745 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:13:38 smithi082 ceph-mon[32933]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "f9ece78d-6a19-4d46-9022-6ed184074745"}]': finished 2024-05-03T21:13:38.745 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:13:38 smithi082 ceph-mon[32933]: osdmap e15: 3 total, 2 up, 3 in 2024-05-03T21:13:38.745 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:13:38 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-03T21:13:39.574 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:13:39 smithi138 ceph-mon[33110]: pgmap v66: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-05-03T21:13:39.574 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:13:39 smithi138 ceph-mon[33110]: from='client.? 172.21.15.138:0/870302820' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-03T21:13:39.721 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:13:39 smithi067 ceph-mon[29730]: pgmap v66: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-05-03T21:13:39.721 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:13:39 smithi067 ceph-mon[29730]: from='client.? 172.21.15.138:0/870302820' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-03T21:13:39.745 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:13:39 smithi082 ceph-mon[32933]: pgmap v66: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-05-03T21:13:39.745 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:13:39 smithi082 ceph-mon[32933]: from='client.? 172.21.15.138:0/870302820' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-03T21:13:41.574 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:13:41 smithi138 ceph-mon[33110]: pgmap v68: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-05-03T21:13:41.721 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:13:41 smithi067 ceph-mon[29730]: pgmap v68: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-05-03T21:13:41.745 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:13:41 smithi082 ceph-mon[32933]: pgmap v68: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-05-03T21:13:43.431 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:13:43 smithi138 ceph-mon[33110]: pgmap v69: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-05-03T21:13:43.721 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:13:43 smithi067 ceph-mon[29730]: pgmap v69: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-05-03T21:13:43.745 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:13:43 smithi082 ceph-mon[32933]: pgmap v69: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-05-03T21:13:45.544 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:13:45 smithi138 ceph-mon[33110]: pgmap v70: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-05-03T21:13:45.544 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:13:45 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-05-03T21:13:45.544 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:13:45 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T21:13:45.721 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:13:45 smithi067 ceph-mon[29730]: pgmap v70: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-05-03T21:13:45.721 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:13:45 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-05-03T21:13:45.721 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:13:45 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T21:13:45.745 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:13:45 smithi082 ceph-mon[32933]: pgmap v70: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-05-03T21:13:45.745 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:13:45 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-05-03T21:13:45.745 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:13:45 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T21:13:46.505 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:13:46 smithi138 ceph-mon[33110]: Deploying daemon osd.2 on smithi138 2024-05-03T21:13:46.720 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:13:46 smithi067 ceph-mon[29730]: Deploying daemon osd.2 on smithi138 2024-05-03T21:13:46.745 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:13:46 smithi082 ceph-mon[32933]: Deploying daemon osd.2 on smithi138 2024-05-03T21:13:47.465 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:13:47 smithi138 ceph-mon[33110]: pgmap v71: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-05-03T21:13:47.721 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:13:47 smithi067 ceph-mon[29730]: pgmap v71: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-05-03T21:13:47.734 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:13:47 smithi082 ceph-mon[32933]: pgmap v71: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-05-03T21:13:48.491 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:13:48 smithi138 ceph-mon[33110]: pgmap v72: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-05-03T21:13:48.721 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:13:48 smithi067 ceph-mon[29730]: pgmap v72: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-05-03T21:13:48.745 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:13:48 smithi082 ceph-mon[32933]: pgmap v72: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-05-03T21:13:51.074 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:13:50 smithi138 ceph-mon[33110]: pgmap v73: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-05-03T21:13:51.074 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:13:50 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:13:51.075 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:13:50 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:13:51.075 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:13:50 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T21:13:51.221 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:13:50 smithi067 ceph-mon[29730]: pgmap v73: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-05-03T21:13:51.221 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:13:50 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:13:51.221 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:13:50 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:13:51.221 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:13:50 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T21:13:51.245 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:13:50 smithi082 ceph-mon[32933]: pgmap v73: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-05-03T21:13:51.245 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:13:50 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:13:51.245 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:13:50 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:13:51.245 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:13:50 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T21:13:53.074 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:13:52 smithi138 ceph-mon[33110]: pgmap v74: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-05-03T21:13:53.221 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:13:52 smithi067 ceph-mon[29730]: pgmap v74: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-05-03T21:13:53.245 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:13:52 smithi082 ceph-mon[32933]: pgmap v74: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-05-03T21:13:53.513 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:13:53.511+0000 7f30ecff9700 1 -- 172.21.15.138:0/1844458068 <== mgr.14150 v2:172.21.15.67:6800/604451592 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f30f006c490 con 0x7f30d80b27f0 2024-05-03T21:13:53.513 INFO:teuthology.orchestra.run.smithi138.stdout:Created osd(s) 2 on host 'smithi138' 2024-05-03T21:13:53.515 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:13:53.513+0000 7f30f6053700 1 -- 172.21.15.138:0/1844458068 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f30d80b27f0 msgr2=0x7f30d80b4cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:13:53.515 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:13:53.513+0000 7f30f6053700 1 --2- 172.21.15.138:0/1844458068 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f30d80b27f0 0x7f30d80b4cb0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f30e0005950 tx=0x7f30e00058e0 comp rx=0 tx=0).stop 2024-05-03T21:13:53.515 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:13:53.513+0000 7f30f6053700 1 -- 172.21.15.138:0/1844458068 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f30f0103600 msgr2=0x7f30f0198b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:13:53.516 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:13:53.513+0000 7f30f6053700 1 --2- 172.21.15.138:0/1844458068 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f30f0103600 0x7f30f0198b40 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f30e40099a0 tx=0x7f30e4009d60 comp rx=0 tx=0).stop 2024-05-03T21:13:53.516 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:13:53.514+0000 7f30f6053700 1 -- 172.21.15.138:0/1844458068 shutdown_connections 2024-05-03T21:13:53.516 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:13:53.514+0000 7f30f6053700 1 --2- 172.21.15.138:0/1844458068 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f30d80b27f0 0x7f30d80b4cb0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:13:53.516 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:13:53.514+0000 7f30f6053700 1 --2- 172.21.15.138:0/1844458068 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f30f0103fa0 0x7f30f019fbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:13:53.516 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:13:53.514+0000 7f30f6053700 1 --2- 172.21.15.138:0/1844458068 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f30f0102400 0x7f30f0198600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:13:53.516 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:13:53.514+0000 7f30f6053700 1 --2- 172.21.15.138:0/1844458068 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f30f0103600 0x7f30f0198b40 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:13:53.516 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:13:53.514+0000 7f30f6053700 1 -- 172.21.15.138:0/1844458068 >> 172.21.15.138:0/1844458068 conn(0x7f30f00fd9a0 msgr2=0x7f30f0106830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T21:13:53.516 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:13:53.514+0000 7f30f6053700 1 -- 172.21.15.138:0/1844458068 shutdown_connections 2024-05-03T21:13:53.516 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:13:53.514+0000 7f30f6053700 1 -- 172.21.15.138:0/1844458068 wait complete. 2024-05-03T21:13:54.501 DEBUG:teuthology.orchestra.run.smithi138:osd.2> sudo journalctl -f -n 0 -u ceph-713bc57a-0991-11ef-bc97-c7b262605968@osd.2.service 2024-05-03T21:13:54.503 INFO:tasks.cephadm:Deploying osd.3 on smithi067 with /dev/vg_nvme/lv_3... 2024-05-03T21:13:54.504 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9a047fe7599a549042d06a3fb5b6813a203eabea shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 713bc57a-0991-11ef-bc97-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-05-03T21:13:54.744 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:13:54 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:13:54.744 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:13:54 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:13:54.744 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:13:54 smithi138 ceph-mon[33110]: pgmap v75: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-05-03T21:13:54.744 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:13:54 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:13:54.744 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:13:54 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:13:54.744 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:13:54 smithi138 ceph-mon[33110]: from='osd.2 [v2:172.21.15.138:6800/2774330123,v1:172.21.15.138:6801/2774330123]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-05-03T21:13:54.744 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:13:54 smithi138 ceph-mon[33110]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-05-03T21:13:54.745 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:13:54 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:13:54.745 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:13:54 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:13:54.745 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:13:54 smithi082 ceph-mon[32933]: pgmap v75: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-05-03T21:13:54.745 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:13:54 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:13:54.745 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:13:54 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:13:54.745 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:13:54 smithi082 ceph-mon[32933]: from='osd.2 [v2:172.21.15.138:6800/2774330123,v1:172.21.15.138:6801/2774330123]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-05-03T21:13:54.746 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:13:54 smithi082 ceph-mon[32933]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-05-03T21:13:54.760 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/713bc57a-0991-11ef-bc97-c7b262605968/mon.a/config 2024-05-03T21:13:54.870 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:13:54 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:13:54.870 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:13:54 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:13:54.870 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:13:54 smithi067 ceph-mon[29730]: pgmap v75: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-05-03T21:13:54.870 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:13:54 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:13:54.870 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:13:54 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:13:54.870 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:13:54 smithi067 ceph-mon[29730]: from='osd.2 [v2:172.21.15.138:6800/2774330123,v1:172.21.15.138:6801/2774330123]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-05-03T21:13:54.870 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:13:54 smithi067 ceph-mon[29730]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-05-03T21:13:56.324 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:13:56 smithi138 ceph-mon[33110]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-05-03T21:13:56.325 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:13:56 smithi138 ceph-mon[33110]: osdmap e16: 3 total, 2 up, 3 in 2024-05-03T21:13:56.325 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:13:56 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-03T21:13:56.325 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:13:56 smithi138 ceph-mon[33110]: from='osd.2 [v2:172.21.15.138:6800/2774330123,v1:172.21.15.138:6801/2774330123]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi138", "root=default"]}]: dispatch 2024-05-03T21:13:56.325 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:13:56 smithi138 ceph-mon[33110]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi138", "root=default"]}]: dispatch 2024-05-03T21:13:56.471 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:13:56 smithi067 ceph-mon[29730]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-05-03T21:13:56.471 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:13:56 smithi067 ceph-mon[29730]: osdmap e16: 3 total, 2 up, 3 in 2024-05-03T21:13:56.471 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:13:56 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-03T21:13:56.471 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:13:56 smithi067 ceph-mon[29730]: from='osd.2 [v2:172.21.15.138:6800/2774330123,v1:172.21.15.138:6801/2774330123]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi138", "root=default"]}]: dispatch 2024-05-03T21:13:56.471 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:13:56 smithi067 ceph-mon[29730]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi138", "root=default"]}]: dispatch 2024-05-03T21:13:56.495 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:13:56 smithi082 ceph-mon[32933]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-05-03T21:13:56.495 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:13:56 smithi082 ceph-mon[32933]: osdmap e16: 3 total, 2 up, 3 in 2024-05-03T21:13:56.495 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:13:56 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-03T21:13:56.495 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:13:56 smithi082 ceph-mon[32933]: from='osd.2 [v2:172.21.15.138:6800/2774330123,v1:172.21.15.138:6801/2774330123]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi138", "root=default"]}]: dispatch 2024-05-03T21:13:56.496 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:13:56 smithi082 ceph-mon[32933]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi138", "root=default"]}]: dispatch 2024-05-03T21:13:56.653 INFO:teuthology.orchestra.run.smithi067.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-05-03T21:13:56.653 INFO:teuthology.orchestra.run.smithi067.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-05-03T21:13:56.653 INFO:teuthology.orchestra.run.smithi067.stderr: stderr: 10+0 records in 2024-05-03T21:13:56.653 INFO:teuthology.orchestra.run.smithi067.stderr:10+0 records out 2024-05-03T21:13:56.653 INFO:teuthology.orchestra.run.smithi067.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0112651 s, 931 MB/s 2024-05-03T21:13:56.653 INFO:teuthology.orchestra.run.smithi067.stderr: stderr: 2024-05-03T21:13:56.653 INFO:teuthology.orchestra.run.smithi067.stderr:--> Zapping successful for: 2024-05-03T21:13:57.265 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:13:57 smithi067 ceph-mon[29730]: pgmap v77: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-05-03T21:13:57.265 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:13:57 smithi067 ceph-mon[29730]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi138", "root=default"]}]': finished 2024-05-03T21:13:57.265 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:13:57 smithi067 ceph-mon[29730]: osdmap e17: 3 total, 2 up, 3 in 2024-05-03T21:13:57.265 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:13:57 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-03T21:13:57.265 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:13:57 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-03T21:13:57.266 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:13:57 smithi067 ceph-mon[29730]: from='osd.2 ' entity='osd.2' 2024-05-03T21:13:57.271 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:13:57 smithi138 ceph-mon[33110]: pgmap v77: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-05-03T21:13:57.271 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:13:57 smithi138 ceph-mon[33110]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi138", "root=default"]}]': finished 2024-05-03T21:13:57.271 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:13:57 smithi138 ceph-mon[33110]: osdmap e17: 3 total, 2 up, 3 in 2024-05-03T21:13:57.271 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:13:57 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-03T21:13:57.271 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:13:57 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-03T21:13:57.271 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:13:57 smithi138 ceph-mon[33110]: from='osd.2 ' entity='osd.2' 2024-05-03T21:13:57.271 INFO:journalctl@ceph.osd.2.smithi138.stdout:May 03 21:13:56 smithi138 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-2[38260]: 2024-05-03T21:13:56.935+0000 7f86b9b02700 -1 osd.2 0 waiting for initial osdmap 2024-05-03T21:13:57.271 INFO:journalctl@ceph.osd.2.smithi138.stdout:May 03 21:13:57 smithi138 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-2[38260]: 2024-05-03T21:13:57.001+0000 7f86b18ed700 -1 osd.2 17 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-05-03T21:13:57.295 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9a047fe7599a549042d06a3fb5b6813a203eabea shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 713bc57a-0991-11ef-bc97-c7b262605968 -- ceph orch daemon add osd smithi067:vg_nvme/lv_3 2024-05-03T21:13:57.495 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:13:57 smithi082 ceph-mon[32933]: pgmap v77: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-05-03T21:13:57.495 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:13:57 smithi082 ceph-mon[32933]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi138", "root=default"]}]': finished 2024-05-03T21:13:57.495 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:13:57 smithi082 ceph-mon[32933]: osdmap e17: 3 total, 2 up, 3 in 2024-05-03T21:13:57.495 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:13:57 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-03T21:13:57.495 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:13:57 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-03T21:13:57.495 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:13:57 smithi082 ceph-mon[32933]: from='osd.2 ' entity='osd.2' 2024-05-03T21:13:57.554 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/713bc57a-0991-11ef-bc97-c7b262605968/mon.a/config 2024-05-03T21:13:58.221 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:13:58 smithi067 ceph-mon[29730]: purged_snaps scrub starts 2024-05-03T21:13:58.221 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:13:58 smithi067 ceph-mon[29730]: purged_snaps scrub ok 2024-05-03T21:13:58.221 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:13:58 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-03T21:13:58.221 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:13:58 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:13:58.221 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:13:58 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:13:58.221 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:13:58 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-05-03T21:13:58.221 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:13:58 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:13:58.222 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:13:58 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T21:13:58.222 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:13:58 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T21:13:58.222 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:13:58 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:13:58.222 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:13:58 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T21:13:58.222 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:13:58 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T21:13:58.222 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:13:58 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T21:13:58.222 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:13:58 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:13:58.222 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:13:58 smithi067 ceph-mon[29730]: osd.2 [v2:172.21.15.138:6800/2774330123,v1:172.21.15.138:6801/2774330123] boot 2024-05-03T21:13:58.222 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:13:58 smithi067 ceph-mon[29730]: osdmap e18: 3 total, 3 up, 3 in 2024-05-03T21:13:58.222 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:13:58 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-03T21:13:58.245 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:13:58 smithi082 ceph-mon[32933]: purged_snaps scrub starts 2024-05-03T21:13:58.246 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:13:58 smithi082 ceph-mon[32933]: purged_snaps scrub ok 2024-05-03T21:13:58.246 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:13:58 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-03T21:13:58.246 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:13:58 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:13:58.246 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:13:58 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:13:58.246 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:13:58 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-05-03T21:13:58.246 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:13:58 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:13:58.246 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:13:58 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T21:13:58.246 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:13:58 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T21:13:58.246 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:13:58 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:13:58.246 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:13:58 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T21:13:58.246 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:13:58 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T21:13:58.246 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:13:58 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T21:13:58.246 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:13:58 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:13:58.247 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:13:58 smithi082 ceph-mon[32933]: osd.2 [v2:172.21.15.138:6800/2774330123,v1:172.21.15.138:6801/2774330123] boot 2024-05-03T21:13:58.247 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:13:58 smithi082 ceph-mon[32933]: osdmap e18: 3 total, 3 up, 3 in 2024-05-03T21:13:58.247 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:13:58 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-03T21:13:58.324 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:13:58 smithi138 ceph-mon[33110]: purged_snaps scrub starts 2024-05-03T21:13:58.324 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:13:58 smithi138 ceph-mon[33110]: purged_snaps scrub ok 2024-05-03T21:13:58.324 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:13:58 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-03T21:13:58.325 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:13:58 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:13:58.325 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:13:58 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:13:58.325 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:13:58 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-05-03T21:13:58.325 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:13:58 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:13:58.325 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:13:58 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T21:13:58.325 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:13:58 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T21:13:58.325 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:13:58 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:13:58.325 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:13:58 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T21:13:58.325 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:13:58 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T21:13:58.325 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:13:58 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T21:13:58.325 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:13:58 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:13:58.325 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:13:58 smithi138 ceph-mon[33110]: osd.2 [v2:172.21.15.138:6800/2774330123,v1:172.21.15.138:6801/2774330123] boot 2024-05-03T21:13:58.325 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:13:58 smithi138 ceph-mon[33110]: osdmap e18: 3 total, 3 up, 3 in 2024-05-03T21:13:58.325 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:13:58 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-03T21:13:59.176 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:13:59.173+0000 7f7086013700 1 -- 172.21.15.67:0/2097292996 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7080106950 msgr2=0x7f7080108d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:13:59.176 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:13:59.173+0000 7f7086013700 1 --2- 172.21.15.67:0/2097292996 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7080106950 0x7f7080108d40 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f7070009990 tx=0x7f7070009ca0 comp rx=0 tx=0).stop 2024-05-03T21:13:59.176 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:13:59.173+0000 7f7086013700 1 -- 172.21.15.67:0/2097292996 shutdown_connections 2024-05-03T21:13:59.176 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:13:59.173+0000 7f7086013700 1 --2- 172.21.15.67:0/2097292996 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f7080109280 0x7f708010b670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:13:59.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:13:59.173+0000 7f7086013700 1 --2- 172.21.15.67:0/2097292996 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f7080104020 0x7f7080106410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:13:59.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:13:59.173+0000 7f7086013700 1 --2- 172.21.15.67:0/2097292996 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7080106950 0x7f7080108d40 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:13:59.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:13:59.173+0000 7f7086013700 1 -- 172.21.15.67:0/2097292996 >> 172.21.15.67:0/2097292996 conn(0x7f70800fd990 msgr2=0x7f70800ffdd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T21:13:59.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:13:59.173+0000 7f7086013700 1 -- 172.21.15.67:0/2097292996 shutdown_connections 2024-05-03T21:13:59.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:13:59.173+0000 7f7086013700 1 -- 172.21.15.67:0/2097292996 wait complete. 2024-05-03T21:13:59.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:13:59.173+0000 7f7086013700 1 Processor -- start 2024-05-03T21:13:59.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:13:59.174+0000 7f7086013700 1 -- start start 2024-05-03T21:13:59.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:13:59.174+0000 7f7086013700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f7080104020 0x7f7080103980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:13:59.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:13:59.174+0000 7f7086013700 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f7080106950 0x7f7080101fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:13:59.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:13:59.174+0000 7f7086013700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7080109280 0x7f7080102510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:13:59.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:13:59.174+0000 7f7086013700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f708010f1e0 con 0x7f7080109280 2024-05-03T21:13:59.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:13:59.174+0000 7f7086013700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f708010f060 con 0x7f7080104020 2024-05-03T21:13:59.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:13:59.174+0000 7f7086013700 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f708010f360 con 0x7f7080106950 2024-05-03T21:13:59.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:13:59.174+0000 7f707ffff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7080109280 0x7f7080102510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:13:59.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:13:59.174+0000 7f707f7fe700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f7080104020 0x7f7080103980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:13:59.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:13:59.174+0000 7f707ffff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7080109280 0x7f7080102510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:59450/0 (socket says 172.21.15.67:59450) 2024-05-03T21:13:59.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:13:59.174+0000 7f707f7fe700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f7080104020 0x7f7080103980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.67:50472/0 (socket says 172.21.15.67:50472) 2024-05-03T21:13:59.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:13:59.174+0000 7f707ffff700 1 -- 172.21.15.67:0/1483329835 learned_addr learned my addr 172.21.15.67:0/1483329835 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-03T21:13:59.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:13:59.174+0000 7f707effd700 1 --2- 172.21.15.67:0/1483329835 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f7080106950 0x7f7080101fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:13:59.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:13:59.174+0000 7f707ffff700 1 -- 172.21.15.67:0/1483329835 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f7080106950 msgr2=0x7f7080101fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:13:59.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:13:59.174+0000 7f707ffff700 1 --2- 172.21.15.67:0/1483329835 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f7080106950 0x7f7080101fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:13:59.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:13:59.174+0000 7f707ffff700 1 -- 172.21.15.67:0/1483329835 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f7080104020 msgr2=0x7f7080103980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:13:59.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:13:59.174+0000 7f707ffff700 1 --2- 172.21.15.67:0/1483329835 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f7080104020 0x7f7080103980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:13:59.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:13:59.174+0000 7f707ffff700 1 -- 172.21.15.67:0/1483329835 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7080102dc0 con 0x7f7080109280 2024-05-03T21:13:59.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:13:59.174+0000 7f707ffff700 1 --2- 172.21.15.67:0/1483329835 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7080109280 0x7f7080102510 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f70780094f0 tx=0x7f707800d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T21:13:59.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:13:59.175+0000 7f707cff9700 1 -- 172.21.15.67:0/1483329835 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7078004db0 con 0x7f7080109280 2024-05-03T21:13:59.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:13:59.175+0000 7f7086013700 1 -- 172.21.15.67:0/1483329835 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f70801a3950 con 0x7f7080109280 2024-05-03T21:13:59.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:13:59.175+0000 7f707cff9700 1 -- 172.21.15.67:0/1483329835 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(28 keys) ==== 1170+0+0 (secure 0 0 0) 0x7f7078009e20 con 0x7f7080109280 2024-05-03T21:13:59.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:13:59.175+0000 7f707cff9700 1 -- 172.21.15.67:0/1483329835 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7078015660 con 0x7f7080109280 2024-05-03T21:13:59.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:13:59.175+0000 7f7086013700 1 -- 172.21.15.67:0/1483329835 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f70801a3e90 con 0x7f7080109280 2024-05-03T21:13:59.180 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:13:59.176+0000 7f7086013700 1 -- 172.21.15.67:0/1483329835 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f708006a190 con 0x7f7080109280 2024-05-03T21:13:59.180 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:13:59.177+0000 7f707cff9700 1 -- 172.21.15.67:0/1483329835 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 15) ==== 149596+0+0 (secure 0 0 0) 0x7f707800db20 con 0x7f7080109280 2024-05-03T21:13:59.180 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:13:59.177+0000 7f707cff9700 1 --2- 172.21.15.67:0/1483329835 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f70680b27f0 0x7f70680b4cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:13:59.180 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:13:59.178+0000 7f707f7fe700 1 --2- 172.21.15.67:0/1483329835 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f70680b27f0 0x7f70680b4cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:13:59.180 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:13:59.178+0000 7f707cff9700 1 -- 172.21.15.67:0/1483329835 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(19..19 src has 1..19) ==== 2851+0+0 (secure 0 0 0) 0x7f70780e0580 con 0x7f7080109280 2024-05-03T21:13:59.180 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:13:59.178+0000 7f707f7fe700 1 --2- 172.21.15.67:0/1483329835 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f70680b27f0 0x7f70680b4cb0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f7074005950 tx=0x7f707400b410 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T21:13:59.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:13:59.182+0000 7f707cff9700 1 -- 172.21.15.67:0/1483329835 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191839 (secure 0 0 0) 0x7f7078010070 con 0x7f7080109280 2024-05-03T21:13:59.324 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:13:59 smithi138 ceph-mon[33110]: Detected new or changed devices on smithi138 2024-05-03T21:13:59.324 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:13:59 smithi138 ceph-mon[33110]: Adjusting osd_memory_target on smithi138 to 17088M 2024-05-03T21:13:59.324 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:13:59 smithi138 ceph-mon[33110]: pgmap v79: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-05-03T21:13:59.406 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:13:59.405+0000 7f7086013700 1 -- 172.21.15.67:0/1483329835 --> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi067:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) -- 0x7f70800638e0 con 0x7f70680b27f0 2024-05-03T21:13:59.406 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:13:59 smithi067 ceph-mon[29730]: Detected new or changed devices on smithi138 2024-05-03T21:13:59.407 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:13:59 smithi067 ceph-mon[29730]: Adjusting osd_memory_target on smithi138 to 17088M 2024-05-03T21:13:59.407 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:13:59 smithi067 ceph-mon[29730]: pgmap v79: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-05-03T21:13:59.495 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:13:59 smithi082 ceph-mon[32933]: Detected new or changed devices on smithi138 2024-05-03T21:13:59.495 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:13:59 smithi082 ceph-mon[32933]: Adjusting osd_memory_target on smithi138 to 17088M 2024-05-03T21:13:59.495 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:13:59 smithi082 ceph-mon[32933]: pgmap v79: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-05-03T21:14:00.324 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:14:00 smithi138 ceph-mon[33110]: osdmap e19: 3 total, 3 up, 3 in 2024-05-03T21:14:00.324 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:14:00 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-03T21:14:00.325 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:14:00 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-03T21:14:00.325 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:14:00 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T21:14:00.325 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:14:00 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-05-03T21:14:00.337 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:14:00 smithi067 ceph-mon[29730]: osdmap e19: 3 total, 3 up, 3 in 2024-05-03T21:14:00.337 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:14:00 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-03T21:14:00.337 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:14:00 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-03T21:14:00.337 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:14:00 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T21:14:00.337 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:14:00 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-05-03T21:14:00.495 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:14:00 smithi082 ceph-mon[32933]: osdmap e19: 3 total, 3 up, 3 in 2024-05-03T21:14:00.495 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:14:00 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-03T21:14:00.495 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:14:00 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-03T21:14:00.495 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:14:00 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T21:14:00.496 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:14:00 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-05-03T21:14:01.221 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:14:01 smithi067 ceph-mon[29730]: from='client.14295 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi067:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-05-03T21:14:01.221 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:14:01 smithi067 ceph-mon[29730]: pgmap v82: 0 pgs: ; 0 B data, 96 MiB used, 268 GiB / 268 GiB avail 2024-05-03T21:14:01.222 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:14:01 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-05-03T21:14:01.222 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:14:01 smithi067 ceph-mon[29730]: osdmap e20: 3 total, 3 up, 3 in 2024-05-03T21:14:01.222 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:14:01 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-05-03T21:14:01.222 INFO:journalctl@ceph.osd.0.smithi067.stdout:May 03 21:14:01 smithi067 sudo[46525]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-05-03T21:14:01.222 INFO:journalctl@ceph.osd.0.smithi067.stdout:May 03 21:14:01 smithi067 sudo[46525]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-05-03T21:14:01.222 INFO:journalctl@ceph.osd.0.smithi067.stdout:May 03 21:14:01 smithi067 sudo[46525]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-05-03T21:14:01.222 INFO:journalctl@ceph.osd.0.smithi067.stdout:May 03 21:14:01 smithi067 sudo[46525]: pam_unix(sudo:session): session closed for user root 2024-05-03T21:14:01.222 INFO:journalctl@ceph.osd.0.smithi067.stdout:May 03 21:14:01 smithi067 sudo[46529]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-05-03T21:14:01.222 INFO:journalctl@ceph.osd.0.smithi067.stdout:May 03 21:14:01 smithi067 sudo[46529]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-05-03T21:14:01.222 INFO:journalctl@ceph.osd.0.smithi067.stdout:May 03 21:14:01 smithi067 sudo[46529]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-05-03T21:14:01.222 INFO:journalctl@ceph.osd.0.smithi067.stdout:May 03 21:14:01 smithi067 sudo[46529]: pam_unix(sudo:session): session closed for user root 2024-05-03T21:14:01.324 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:14:01 smithi138 ceph-mon[33110]: from='client.14295 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi067:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-05-03T21:14:01.325 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:14:01 smithi138 ceph-mon[33110]: pgmap v82: 0 pgs: ; 0 B data, 96 MiB used, 268 GiB / 268 GiB avail 2024-05-03T21:14:01.325 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:14:01 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-05-03T21:14:01.325 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:14:01 smithi138 ceph-mon[33110]: osdmap e20: 3 total, 3 up, 3 in 2024-05-03T21:14:01.325 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:14:01 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-05-03T21:14:01.325 INFO:journalctl@ceph.osd.2.smithi138.stdout:May 03 21:14:01 smithi138 sudo[42323]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-05-03T21:14:01.325 INFO:journalctl@ceph.osd.2.smithi138.stdout:May 03 21:14:01 smithi138 sudo[42323]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-05-03T21:14:01.325 INFO:journalctl@ceph.osd.2.smithi138.stdout:May 03 21:14:01 smithi138 sudo[42323]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-05-03T21:14:01.325 INFO:journalctl@ceph.osd.2.smithi138.stdout:May 03 21:14:01 smithi138 sudo[42323]: pam_unix(sudo:session): session closed for user root 2024-05-03T21:14:01.325 INFO:journalctl@ceph.osd.2.smithi138.stdout:May 03 21:14:01 smithi138 sudo[42326]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-05-03T21:14:01.326 INFO:journalctl@ceph.osd.2.smithi138.stdout:May 03 21:14:01 smithi138 sudo[42326]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-05-03T21:14:01.326 INFO:journalctl@ceph.osd.2.smithi138.stdout:May 03 21:14:01 smithi138 sudo[42326]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-05-03T21:14:01.326 INFO:journalctl@ceph.osd.2.smithi138.stdout:May 03 21:14:01 smithi138 sudo[42326]: pam_unix(sudo:session): session closed for user root 2024-05-03T21:14:01.401 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:14:01 smithi082 ceph-mon[32933]: from='client.14295 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi067:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-05-03T21:14:01.401 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:14:01 smithi082 ceph-mon[32933]: pgmap v82: 0 pgs: ; 0 B data, 96 MiB used, 268 GiB / 268 GiB avail 2024-05-03T21:14:01.401 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:14:01 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-05-03T21:14:01.401 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:14:01 smithi082 ceph-mon[32933]: osdmap e20: 3 total, 3 up, 3 in 2024-05-03T21:14:01.402 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:14:01 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-05-03T21:14:01.402 INFO:journalctl@ceph.osd.1.smithi082.stdout:May 03 21:14:01 smithi082 sudo[41220]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-05-03T21:14:01.402 INFO:journalctl@ceph.osd.1.smithi082.stdout:May 03 21:14:01 smithi082 sudo[41220]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-05-03T21:14:01.402 INFO:journalctl@ceph.osd.1.smithi082.stdout:May 03 21:14:01 smithi082 sudo[41220]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-05-03T21:14:01.402 INFO:journalctl@ceph.osd.1.smithi082.stdout:May 03 21:14:01 smithi082 sudo[41220]: pam_unix(sudo:session): session closed for user root 2024-05-03T21:14:01.402 INFO:journalctl@ceph.osd.1.smithi082.stdout:May 03 21:14:01 smithi082 sudo[41223]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-05-03T21:14:01.402 INFO:journalctl@ceph.osd.1.smithi082.stdout:May 03 21:14:01 smithi082 sudo[41223]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-05-03T21:14:01.402 INFO:journalctl@ceph.osd.1.smithi082.stdout:May 03 21:14:01 smithi082 sudo[41223]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-05-03T21:14:01.402 INFO:journalctl@ceph.osd.1.smithi082.stdout:May 03 21:14:01 smithi082 sudo[41223]: pam_unix(sudo:session): session closed for user root 2024-05-03T21:14:01.494 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:14:01 smithi067 sudo[46536]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-05-03T21:14:01.494 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:14:01 smithi067 sudo[46536]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-05-03T21:14:01.494 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:14:01 smithi067 sudo[46536]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-05-03T21:14:01.494 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:14:01 smithi067 sudo[46536]: pam_unix(sudo:session): session closed for user root 2024-05-03T21:14:01.494 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:14:01 smithi067 sudo[46539]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-05-03T21:14:01.495 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:14:01 smithi067 sudo[46539]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-05-03T21:14:01.495 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:14:01 smithi067 sudo[46539]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-05-03T21:14:01.495 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:14:01 smithi067 sudo[46539]: pam_unix(sudo:session): session closed for user root 2024-05-03T21:14:01.745 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:14:01 smithi082 sudo[41226]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-05-03T21:14:01.745 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:14:01 smithi082 sudo[41226]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-05-03T21:14:01.745 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:14:01 smithi082 sudo[41226]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-05-03T21:14:01.745 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:14:01 smithi082 sudo[41226]: pam_unix(sudo:session): session closed for user root 2024-05-03T21:14:01.745 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:14:01 smithi082 sudo[41229]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-05-03T21:14:01.745 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:14:01 smithi082 sudo[41229]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-05-03T21:14:01.746 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:14:01 smithi082 sudo[41229]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-05-03T21:14:01.746 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:14:01 smithi082 sudo[41229]: pam_unix(sudo:session): session closed for user root 2024-05-03T21:14:01.824 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:14:01 smithi138 sudo[42329]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-05-03T21:14:01.824 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:14:01 smithi138 sudo[42329]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-05-03T21:14:01.824 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:14:01 smithi138 sudo[42329]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-05-03T21:14:01.825 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:14:01 smithi138 sudo[42329]: pam_unix(sudo:session): session closed for user root 2024-05-03T21:14:01.825 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:14:01 smithi138 sudo[42332]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-05-03T21:14:01.825 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:14:01 smithi138 sudo[42332]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-05-03T21:14:01.825 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:14:01 smithi138 sudo[42332]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-05-03T21:14:01.825 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:14:01 smithi138 sudo[42332]: pam_unix(sudo:session): session closed for user root 2024-05-03T21:14:02.329 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:14:02.326+0000 7f707cff9700 1 -- 172.21.15.67:0/1483329835 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mgrmap(e 16) ==== 149652+0+0 (secure 0 0 0) 0x7f70780a5460 con 0x7f7080109280 2024-05-03T21:14:02.471 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:14:02 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-05-03T21:14:02.471 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:14:02 smithi067 ceph-mon[29730]: osdmap e21: 3 total, 3 up, 3 in 2024-05-03T21:14:02.471 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:14:02 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:14:02.471 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:14:02 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:14:02.472 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:14:02 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:14:02.472 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:14:02 smithi067 ceph-mon[29730]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-05-03T21:14:02.472 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:14:02 smithi067 ceph-mon[29730]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-05-03T21:14:02.472 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:14:02 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-03T21:14:02.472 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:14:02 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-03T21:14:02.472 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:14:02 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-03T21:14:02.472 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:14:02 smithi067 ceph-mon[29730]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-05-03T21:14:02.472 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:14:02 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-03T21:14:02.472 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:14:02 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-03T21:14:02.472 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:14:02 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-03T21:14:02.472 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:14:02 smithi067 ceph-mon[29730]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-05-03T21:14:02.472 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:14:02 smithi067 ceph-mon[29730]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-05-03T21:14:02.472 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:14:02 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-03T21:14:02.472 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:14:02 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-03T21:14:02.473 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:14:02 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-03T21:14:02.473 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:14:02 smithi067 ceph-mon[29730]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-05-03T21:14:02.495 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:14:02 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-05-03T21:14:02.495 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:14:02 smithi082 ceph-mon[32933]: osdmap e21: 3 total, 3 up, 3 in 2024-05-03T21:14:02.496 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:14:02 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:14:02.496 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:14:02 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:14:02.496 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:14:02 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:14:02.496 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:14:02 smithi082 ceph-mon[32933]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-05-03T21:14:02.496 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:14:02 smithi082 ceph-mon[32933]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-05-03T21:14:02.496 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:14:02 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-03T21:14:02.496 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:14:02 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-03T21:14:02.496 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:14:02 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-03T21:14:02.496 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:14:02 smithi082 ceph-mon[32933]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-05-03T21:14:02.496 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:14:02 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-03T21:14:02.496 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:14:02 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-03T21:14:02.496 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:14:02 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-03T21:14:02.496 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:14:02 smithi082 ceph-mon[32933]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-05-03T21:14:02.496 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:14:02 smithi082 ceph-mon[32933]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-05-03T21:14:02.497 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:14:02 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-03T21:14:02.497 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:14:02 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-03T21:14:02.497 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:14:02 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-03T21:14:02.497 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:14:02 smithi082 ceph-mon[32933]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-05-03T21:14:02.575 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:14:02 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-05-03T21:14:02.575 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:14:02 smithi138 ceph-mon[33110]: osdmap e21: 3 total, 3 up, 3 in 2024-05-03T21:14:02.575 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:14:02 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:14:02.575 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:14:02 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:14:02.575 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:14:02 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:14:02.575 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:14:02 smithi138 ceph-mon[33110]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-05-03T21:14:02.575 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:14:02 smithi138 ceph-mon[33110]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-05-03T21:14:02.575 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:14:02 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-03T21:14:02.575 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:14:02 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-03T21:14:02.575 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:14:02 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-03T21:14:02.575 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:14:02 smithi138 ceph-mon[33110]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-05-03T21:14:02.575 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:14:02 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-03T21:14:02.576 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:14:02 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-03T21:14:02.576 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:14:02 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-03T21:14:02.576 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:14:02 smithi138 ceph-mon[33110]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-05-03T21:14:02.576 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:14:02 smithi138 ceph-mon[33110]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-05-03T21:14:02.576 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:14:02 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-03T21:14:02.576 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:14:02 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-03T21:14:02.576 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:14:02 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-03T21:14:02.576 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:14:02 smithi138 ceph-mon[33110]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-05-03T21:14:03.471 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:14:03 smithi067 ceph-mon[29730]: pgmap v85: 1 pgs: 1 creating+peering; 0 B data, 96 MiB used, 268 GiB / 268 GiB avail 2024-05-03T21:14:03.471 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:14:03 smithi067 ceph-mon[29730]: osdmap e22: 3 total, 3 up, 3 in 2024-05-03T21:14:03.471 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:14:03 smithi067 ceph-mon[29730]: from='client.? 172.21.15.67:0/2757892718' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3616df71-e7b3-4687-aaba-37b0b7766a3c"}]: dispatch 2024-05-03T21:14:03.471 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:14:03 smithi067 ceph-mon[29730]: from='client.? 172.21.15.67:0/2757892718' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "3616df71-e7b3-4687-aaba-37b0b7766a3c"}]': finished 2024-05-03T21:14:03.471 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:14:03 smithi067 ceph-mon[29730]: osdmap e23: 4 total, 3 up, 4 in 2024-05-03T21:14:03.472 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:14:03 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-03T21:14:03.472 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:14:03 smithi067 ceph-mon[29730]: mgrmap e16: x(active, since 2m), standbys: z, y 2024-05-03T21:14:03.472 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:14:03 smithi067 ceph-mon[29730]: from='client.? 172.21.15.67:0/4156570376' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-03T21:14:03.495 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:14:03 smithi082 ceph-mon[32933]: pgmap v85: 1 pgs: 1 creating+peering; 0 B data, 96 MiB used, 268 GiB / 268 GiB avail 2024-05-03T21:14:03.495 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:14:03 smithi082 ceph-mon[32933]: osdmap e22: 3 total, 3 up, 3 in 2024-05-03T21:14:03.495 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:14:03 smithi082 ceph-mon[32933]: from='client.? 172.21.15.67:0/2757892718' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3616df71-e7b3-4687-aaba-37b0b7766a3c"}]: dispatch 2024-05-03T21:14:03.495 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:14:03 smithi082 ceph-mon[32933]: from='client.? 172.21.15.67:0/2757892718' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "3616df71-e7b3-4687-aaba-37b0b7766a3c"}]': finished 2024-05-03T21:14:03.496 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:14:03 smithi082 ceph-mon[32933]: osdmap e23: 4 total, 3 up, 4 in 2024-05-03T21:14:03.496 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:14:03 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-03T21:14:03.496 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:14:03 smithi082 ceph-mon[32933]: mgrmap e16: x(active, since 2m), standbys: z, y 2024-05-03T21:14:03.496 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:14:03 smithi082 ceph-mon[32933]: from='client.? 172.21.15.67:0/4156570376' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-03T21:14:03.574 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:14:03 smithi138 ceph-mon[33110]: pgmap v85: 1 pgs: 1 creating+peering; 0 B data, 96 MiB used, 268 GiB / 268 GiB avail 2024-05-03T21:14:03.574 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:14:03 smithi138 ceph-mon[33110]: osdmap e22: 3 total, 3 up, 3 in 2024-05-03T21:14:03.575 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:14:03 smithi138 ceph-mon[33110]: from='client.? 172.21.15.67:0/2757892718' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3616df71-e7b3-4687-aaba-37b0b7766a3c"}]: dispatch 2024-05-03T21:14:03.575 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:14:03 smithi138 ceph-mon[33110]: from='client.? 172.21.15.67:0/2757892718' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "3616df71-e7b3-4687-aaba-37b0b7766a3c"}]': finished 2024-05-03T21:14:03.575 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:14:03 smithi138 ceph-mon[33110]: osdmap e23: 4 total, 3 up, 4 in 2024-05-03T21:14:03.575 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:14:03 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-03T21:14:03.575 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:14:03 smithi138 ceph-mon[33110]: mgrmap e16: x(active, since 2m), standbys: z, y 2024-05-03T21:14:03.575 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:14:03 smithi138 ceph-mon[33110]: from='client.? 172.21.15.67:0/4156570376' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-03T21:14:05.471 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:14:05 smithi067 ceph-mon[29730]: pgmap v88: 1 pgs: 1 creating+peering; 0 B data, 97 MiB used, 268 GiB / 268 GiB avail 2024-05-03T21:14:05.495 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:14:05 smithi082 ceph-mon[32933]: pgmap v88: 1 pgs: 1 creating+peering; 0 B data, 97 MiB used, 268 GiB / 268 GiB avail 2024-05-03T21:14:05.574 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:14:05 smithi138 ceph-mon[33110]: pgmap v88: 1 pgs: 1 creating+peering; 0 B data, 97 MiB used, 268 GiB / 268 GiB avail 2024-05-03T21:14:07.221 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:14:07 smithi067 ceph-mon[29730]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 97 MiB used, 268 GiB / 268 GiB avail 2024-05-03T21:14:07.495 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:14:07 smithi082 ceph-mon[32933]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 97 MiB used, 268 GiB / 268 GiB avail 2024-05-03T21:14:07.574 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:14:07 smithi138 ceph-mon[33110]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 97 MiB used, 268 GiB / 268 GiB avail 2024-05-03T21:14:09.471 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:14:09 smithi067 ceph-mon[29730]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 98 MiB used, 268 GiB / 268 GiB avail 2024-05-03T21:14:09.495 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:14:09 smithi082 ceph-mon[32933]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 98 MiB used, 268 GiB / 268 GiB avail 2024-05-03T21:14:09.574 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:14:09 smithi138 ceph-mon[33110]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 98 MiB used, 268 GiB / 268 GiB avail 2024-05-03T21:14:10.471 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:14:10 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-05-03T21:14:10.471 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:14:10 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T21:14:10.495 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:14:10 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-05-03T21:14:10.495 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:14:10 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T21:14:10.574 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:14:10 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-05-03T21:14:10.574 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:14:10 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T21:14:11.471 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:14:11 smithi067 ceph-mon[29730]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 98 MiB used, 268 GiB / 268 GiB avail 2024-05-03T21:14:11.471 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:14:11 smithi067 ceph-mon[29730]: Deploying daemon osd.3 on smithi067 2024-05-03T21:14:11.495 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:14:11 smithi082 ceph-mon[32933]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 98 MiB used, 268 GiB / 268 GiB avail 2024-05-03T21:14:11.495 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:14:11 smithi082 ceph-mon[32933]: Deploying daemon osd.3 on smithi067 2024-05-03T21:14:11.574 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:14:11 smithi138 ceph-mon[33110]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 98 MiB used, 268 GiB / 268 GiB avail 2024-05-03T21:14:11.574 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:14:11 smithi138 ceph-mon[33110]: Deploying daemon osd.3 on smithi067 2024-05-03T21:14:13.393 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:14:13 smithi138 ceph-mon[33110]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 98 MiB used, 268 GiB / 268 GiB avail 2024-05-03T21:14:13.471 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:14:13 smithi067 ceph-mon[29730]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 98 MiB used, 268 GiB / 268 GiB avail 2024-05-03T21:14:13.495 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:14:13 smithi082 ceph-mon[32933]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 98 MiB used, 268 GiB / 268 GiB avail 2024-05-03T21:14:15.364 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:14:15 smithi067 ceph-mon[29730]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 98 MiB used, 268 GiB / 268 GiB avail 2024-05-03T21:14:15.495 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:14:15 smithi082 ceph-mon[32933]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 98 MiB used, 268 GiB / 268 GiB avail 2024-05-03T21:14:15.574 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:14:15 smithi138 ceph-mon[33110]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 98 MiB used, 268 GiB / 268 GiB avail 2024-05-03T21:14:16.971 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:14:16 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:14:16.971 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:14:16 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T21:14:16.971 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:14:16 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:14:16.971 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:14:16 smithi067 ceph-mon[29730]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 98 MiB used, 268 GiB / 268 GiB avail 2024-05-03T21:14:16.995 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:14:16 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:14:16.995 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:14:16 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T21:14:16.995 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:14:16 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:14:16.995 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:14:16 smithi082 ceph-mon[32933]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 98 MiB used, 268 GiB / 268 GiB avail 2024-05-03T21:14:17.074 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:14:16 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:14:17.074 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:14:16 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T21:14:17.074 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:14:16 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:14:17.074 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:14:16 smithi138 ceph-mon[33110]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 98 MiB used, 268 GiB / 268 GiB avail 2024-05-03T21:14:18.920 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:14:18 smithi067 ceph-mon[29730]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 98 MiB used, 268 GiB / 268 GiB avail 2024-05-03T21:14:19.074 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:14:18 smithi138 ceph-mon[33110]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 98 MiB used, 268 GiB / 268 GiB avail 2024-05-03T21:14:19.218 INFO:teuthology.orchestra.run.smithi067.stdout:Created osd(s) 3 on host 'smithi067' 2024-05-03T21:14:19.219 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:14:19.216+0000 7f707cff9700 1 -- 172.21.15.67:0/1483329835 <== mgr.14150 v2:172.21.15.67:6800/604451592 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f70800638e0 con 0x7f70680b27f0 2024-05-03T21:14:19.221 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:14:19.218+0000 7f7086013700 1 -- 172.21.15.67:0/1483329835 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f70680b27f0 msgr2=0x7f70680b4cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:14:19.221 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:14:19.218+0000 7f7086013700 1 --2- 172.21.15.67:0/1483329835 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f70680b27f0 0x7f70680b4cb0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f7074005950 tx=0x7f707400b410 comp rx=0 tx=0).stop 2024-05-03T21:14:19.221 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:14:19.219+0000 7f7086013700 1 -- 172.21.15.67:0/1483329835 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7080109280 msgr2=0x7f7080102510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:14:19.221 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:14:19.219+0000 7f7086013700 1 --2- 172.21.15.67:0/1483329835 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7080109280 0x7f7080102510 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f70780094f0 tx=0x7f707800d3b0 comp rx=0 tx=0).stop 2024-05-03T21:14:19.221 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:14:19.219+0000 7f7086013700 1 -- 172.21.15.67:0/1483329835 shutdown_connections 2024-05-03T21:14:19.221 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:14:19.219+0000 7f7086013700 1 --2- 172.21.15.67:0/1483329835 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f70680b27f0 0x7f70680b4cb0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:14:19.222 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:14:19.219+0000 7f7086013700 1 --2- 172.21.15.67:0/1483329835 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7080109280 0x7f7080102510 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:14:19.222 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:14:19.219+0000 7f7086013700 1 --2- 172.21.15.67:0/1483329835 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f7080104020 0x7f7080103980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:14:19.222 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:14:19.219+0000 7f7086013700 1 --2- 172.21.15.67:0/1483329835 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f7080106950 0x7f7080101fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:14:19.222 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:14:19.219+0000 7f7086013700 1 -- 172.21.15.67:0/1483329835 >> 172.21.15.67:0/1483329835 conn(0x7f70800fd990 msgr2=0x7f70801048c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T21:14:19.222 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:14:19.219+0000 7f7086013700 1 -- 172.21.15.67:0/1483329835 shutdown_connections 2024-05-03T21:14:19.222 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:14:19.219+0000 7f7086013700 1 -- 172.21.15.67:0/1483329835 wait complete. 2024-05-03T21:14:19.245 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:14:18 smithi082 ceph-mon[32933]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 98 MiB used, 268 GiB / 268 GiB avail 2024-05-03T21:14:19.919 DEBUG:teuthology.orchestra.run.smithi067:osd.3> sudo journalctl -f -n 0 -u ceph-713bc57a-0991-11ef-bc97-c7b262605968@osd.3.service 2024-05-03T21:14:19.921 INFO:tasks.cephadm:Deploying osd.4 on smithi082 with /dev/vg_nvme/lv_3... 2024-05-03T21:14:19.921 DEBUG:teuthology.orchestra.run.smithi082:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9a047fe7599a549042d06a3fb5b6813a203eabea shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 713bc57a-0991-11ef-bc97-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-05-03T21:14:20.166 INFO:teuthology.orchestra.run.smithi082.stderr:Inferring config /var/lib/ceph/713bc57a-0991-11ef-bc97-c7b262605968/mon.b/config 2024-05-03T21:14:20.471 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:14:20 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:14:20.471 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:14:20 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:14:20.471 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:14:20 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:14:20.471 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:14:20 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:14:20.471 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:14:20 smithi067 ceph-mon[29730]: from='osd.3 [v2:172.21.15.67:6810/3915934652,v1:172.21.15.67:6811/3915934652]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-05-03T21:14:20.482 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:14:20 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:14:20.482 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:14:20 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:14:20.483 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:14:20 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:14:20.483 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:14:20 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:14:20.483 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:14:20 smithi082 ceph-mon[32933]: from='osd.3 [v2:172.21.15.67:6810/3915934652,v1:172.21.15.67:6811/3915934652]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-05-03T21:14:20.574 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:14:20 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:14:20.574 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:14:20 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:14:20.574 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:14:20 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:14:20.574 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:14:20 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:14:20.574 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:14:20 smithi138 ceph-mon[33110]: from='osd.3 [v2:172.21.15.67:6810/3915934652,v1:172.21.15.67:6811/3915934652]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-05-03T21:14:21.446 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:14:21 smithi082 ceph-mon[32933]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 98 MiB used, 268 GiB / 268 GiB avail 2024-05-03T21:14:21.446 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:14:21 smithi082 ceph-mon[32933]: from='osd.3 [v2:172.21.15.67:6810/3915934652,v1:172.21.15.67:6811/3915934652]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-05-03T21:14:21.446 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:14:21 smithi082 ceph-mon[32933]: osdmap e24: 4 total, 3 up, 4 in 2024-05-03T21:14:21.446 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:14:21 smithi082 ceph-mon[32933]: from='osd.3 [v2:172.21.15.67:6810/3915934652,v1:172.21.15.67:6811/3915934652]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi067", "root=default"]}]: dispatch 2024-05-03T21:14:21.446 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:14:21 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-03T21:14:21.574 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:14:21 smithi138 ceph-mon[33110]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 98 MiB used, 268 GiB / 268 GiB avail 2024-05-03T21:14:21.574 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:14:21 smithi138 ceph-mon[33110]: from='osd.3 [v2:172.21.15.67:6810/3915934652,v1:172.21.15.67:6811/3915934652]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-05-03T21:14:21.574 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:14:21 smithi138 ceph-mon[33110]: osdmap e24: 4 total, 3 up, 4 in 2024-05-03T21:14:21.574 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:14:21 smithi138 ceph-mon[33110]: from='osd.3 [v2:172.21.15.67:6810/3915934652,v1:172.21.15.67:6811/3915934652]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi067", "root=default"]}]: dispatch 2024-05-03T21:14:21.575 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:14:21 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-03T21:14:21.721 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:14:21 smithi067 ceph-mon[29730]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 98 MiB used, 268 GiB / 268 GiB avail 2024-05-03T21:14:21.721 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:14:21 smithi067 ceph-mon[29730]: from='osd.3 [v2:172.21.15.67:6810/3915934652,v1:172.21.15.67:6811/3915934652]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-05-03T21:14:21.721 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:14:21 smithi067 ceph-mon[29730]: osdmap e24: 4 total, 3 up, 4 in 2024-05-03T21:14:21.721 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:14:21 smithi067 ceph-mon[29730]: from='osd.3 [v2:172.21.15.67:6810/3915934652,v1:172.21.15.67:6811/3915934652]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi067", "root=default"]}]: dispatch 2024-05-03T21:14:21.722 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:14:21 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-03T21:14:21.999 INFO:teuthology.orchestra.run.smithi082.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-05-03T21:14:21.999 INFO:teuthology.orchestra.run.smithi082.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-05-03T21:14:21.999 INFO:teuthology.orchestra.run.smithi082.stderr: stderr: 10+0 records in 2024-05-03T21:14:21.999 INFO:teuthology.orchestra.run.smithi082.stderr:10+0 records out 2024-05-03T21:14:21.999 INFO:teuthology.orchestra.run.smithi082.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0116247 s, 902 MB/s 2024-05-03T21:14:21.999 INFO:teuthology.orchestra.run.smithi082.stderr:--> Zapping successful for: 2024-05-03T21:14:22.471 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:14:22 smithi067 ceph-mon[29730]: from='osd.3 [v2:172.21.15.67:6810/3915934652,v1:172.21.15.67:6811/3915934652]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi067", "root=default"]}]': finished 2024-05-03T21:14:22.471 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:14:22 smithi067 ceph-mon[29730]: osdmap e25: 4 total, 3 up, 4 in 2024-05-03T21:14:22.471 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:14:22 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-03T21:14:22.471 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:14:22 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-03T21:14:22.471 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:14:22 smithi067 ceph-mon[29730]: from='osd.3 [v2:172.21.15.67:6810/3915934652,v1:172.21.15.67:6811/3915934652]' entity='osd.3' 2024-05-03T21:14:22.472 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:14:22 smithi067 ceph-mon[29730]: osdmap e26: 4 total, 3 up, 4 in 2024-05-03T21:14:22.472 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:14:22 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-03T21:14:22.472 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:14:22 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-03T21:14:22.472 INFO:journalctl@ceph.osd.3.smithi067.stdout:May 03 21:14:22 smithi067 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-3[48631]: 2024-05-03T21:14:22.192+0000 7fa4b892b700 -1 osd.3 0 waiting for initial osdmap 2024-05-03T21:14:22.472 INFO:journalctl@ceph.osd.3.smithi067.stdout:May 03 21:14:22 smithi067 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-3[48631]: 2024-05-03T21:14:22.318+0000 7fa4aff15700 -1 osd.3 25 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-05-03T21:14:22.574 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:14:22 smithi138 ceph-mon[33110]: from='osd.3 [v2:172.21.15.67:6810/3915934652,v1:172.21.15.67:6811/3915934652]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi067", "root=default"]}]': finished 2024-05-03T21:14:22.574 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:14:22 smithi138 ceph-mon[33110]: osdmap e25: 4 total, 3 up, 4 in 2024-05-03T21:14:22.574 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:14:22 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-03T21:14:22.575 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:14:22 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-03T21:14:22.575 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:14:22 smithi138 ceph-mon[33110]: from='osd.3 [v2:172.21.15.67:6810/3915934652,v1:172.21.15.67:6811/3915934652]' entity='osd.3' 2024-05-03T21:14:22.575 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:14:22 smithi138 ceph-mon[33110]: osdmap e26: 4 total, 3 up, 4 in 2024-05-03T21:14:22.575 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:14:22 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-03T21:14:22.575 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:14:22 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-03T21:14:22.600 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:14:22 smithi082 ceph-mon[32933]: from='osd.3 [v2:172.21.15.67:6810/3915934652,v1:172.21.15.67:6811/3915934652]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi067", "root=default"]}]': finished 2024-05-03T21:14:22.601 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:14:22 smithi082 ceph-mon[32933]: osdmap e25: 4 total, 3 up, 4 in 2024-05-03T21:14:22.601 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:14:22 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-03T21:14:22.601 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:14:22 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-03T21:14:22.601 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:14:22 smithi082 ceph-mon[32933]: from='osd.3 [v2:172.21.15.67:6810/3915934652,v1:172.21.15.67:6811/3915934652]' entity='osd.3' 2024-05-03T21:14:22.601 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:14:22 smithi082 ceph-mon[32933]: osdmap e26: 4 total, 3 up, 4 in 2024-05-03T21:14:22.601 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:14:22 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-03T21:14:22.601 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:14:22 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-03T21:14:22.635 DEBUG:teuthology.orchestra.run.smithi082:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9a047fe7599a549042d06a3fb5b6813a203eabea shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 713bc57a-0991-11ef-bc97-c7b262605968 -- ceph orch daemon add osd smithi082:vg_nvme/lv_3 2024-05-03T21:14:22.874 INFO:teuthology.orchestra.run.smithi082.stderr:Inferring config /var/lib/ceph/713bc57a-0991-11ef-bc97-c7b262605968/mon.b/config 2024-05-03T21:14:23.575 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:14:23 smithi138 ceph-mon[33110]: purged_snaps scrub starts 2024-05-03T21:14:23.576 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:14:23 smithi138 ceph-mon[33110]: purged_snaps scrub ok 2024-05-03T21:14:23.576 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:14:23 smithi138 ceph-mon[33110]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 98 MiB used, 268 GiB / 268 GiB avail 2024-05-03T21:14:23.576 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:14:23 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:14:23.576 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:14:23 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:14:23.577 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:14:23 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-05-03T21:14:23.577 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:14:23 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-05-03T21:14:23.577 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:14:23 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:14:23.577 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:14:23 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T21:14:23.577 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:14:23 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T21:14:23.577 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:14:23 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:14:23.577 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:14:23 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T21:14:23.577 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:14:23 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T21:14:23.577 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:14:23 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T21:14:23.578 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:14:23 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:14:23.578 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:14:23 smithi138 ceph-mon[33110]: osd.3 [v2:172.21.15.67:6810/3915934652,v1:172.21.15.67:6811/3915934652] boot 2024-05-03T21:14:23.578 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:14:23 smithi138 ceph-mon[33110]: osdmap e27: 4 total, 4 up, 4 in 2024-05-03T21:14:23.578 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:14:23 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-03T21:14:23.673 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:14:23 smithi082 ceph-mon[32933]: purged_snaps scrub starts 2024-05-03T21:14:23.674 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:14:23 smithi082 ceph-mon[32933]: purged_snaps scrub ok 2024-05-03T21:14:23.674 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:14:23 smithi082 ceph-mon[32933]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 98 MiB used, 268 GiB / 268 GiB avail 2024-05-03T21:14:23.674 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:14:23 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:14:23.674 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:14:23 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:14:23.674 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:14:23 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-05-03T21:14:23.674 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:14:23 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-05-03T21:14:23.674 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:14:23 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:14:23.675 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:14:23 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T21:14:23.675 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:14:23 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T21:14:23.675 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:14:23 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:14:23.675 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:14:23 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T21:14:23.675 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:14:23 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T21:14:23.675 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:14:23 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T21:14:23.675 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:14:23 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:14:23.675 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:14:23 smithi082 ceph-mon[32933]: osd.3 [v2:172.21.15.67:6810/3915934652,v1:172.21.15.67:6811/3915934652] boot 2024-05-03T21:14:23.676 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:14:23 smithi082 ceph-mon[32933]: osdmap e27: 4 total, 4 up, 4 in 2024-05-03T21:14:23.676 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:14:23 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-03T21:14:23.721 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:14:23 smithi067 ceph-mon[29730]: purged_snaps scrub starts 2024-05-03T21:14:23.721 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:14:23 smithi067 ceph-mon[29730]: purged_snaps scrub ok 2024-05-03T21:14:23.722 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:14:23 smithi067 ceph-mon[29730]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 98 MiB used, 268 GiB / 268 GiB avail 2024-05-03T21:14:23.722 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:14:23 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:14:23.722 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:14:23 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:14:23.722 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:14:23 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-05-03T21:14:23.722 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:14:23 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-05-03T21:14:23.722 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:14:23 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:14:23.722 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:14:23 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T21:14:23.723 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:14:23 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T21:14:23.723 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:14:23 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:14:23.723 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:14:23 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T21:14:23.723 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:14:23 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T21:14:23.724 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:14:23 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T21:14:23.724 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:14:23 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:14:23.724 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:14:23 smithi067 ceph-mon[29730]: osd.3 [v2:172.21.15.67:6810/3915934652,v1:172.21.15.67:6811/3915934652] boot 2024-05-03T21:14:23.724 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:14:23 smithi067 ceph-mon[29730]: osdmap e27: 4 total, 4 up, 4 in 2024-05-03T21:14:23.724 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:14:23 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-03T21:14:24.391 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T21:14:24.387+0000 7fa45835c700 1 -- 172.21.15.82:0/4005312059 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fa450103500 msgr2=0x7fa450103920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:14:24.391 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T21:14:24.387+0000 7fa45835c700 1 --2- 172.21.15.82:0/4005312059 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fa450103500 0x7fa450103920 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fa4400099c0 tx=0x7fa440009cd0 comp rx=0 tx=0).stop 2024-05-03T21:14:24.391 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T21:14:24.387+0000 7fa45835c700 1 -- 172.21.15.82:0/4005312059 shutdown_connections 2024-05-03T21:14:24.391 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T21:14:24.387+0000 7fa45835c700 1 --2- 172.21.15.82:0/4005312059 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa4501050a0 0x7fa45010b950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:14:24.391 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T21:14:24.387+0000 7fa45835c700 1 --2- 172.21.15.82:0/4005312059 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fa450103500 0x7fa450103920 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:14:24.391 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T21:14:24.387+0000 7fa45835c700 1 --2- 172.21.15.82:0/4005312059 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fa450104700 0x7fa450104b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:14:24.391 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T21:14:24.387+0000 7fa45835c700 1 -- 172.21.15.82:0/4005312059 >> 172.21.15.82:0/4005312059 conn(0x7fa4500feaa0 msgr2=0x7fa450100ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T21:14:24.391 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T21:14:24.387+0000 7fa45835c700 1 -- 172.21.15.82:0/4005312059 shutdown_connections 2024-05-03T21:14:24.391 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T21:14:24.387+0000 7fa45835c700 1 -- 172.21.15.82:0/4005312059 wait complete. 2024-05-03T21:14:24.391 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T21:14:24.388+0000 7fa45835c700 1 Processor -- start 2024-05-03T21:14:24.391 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T21:14:24.388+0000 7fa45835c700 1 -- start start 2024-05-03T21:14:24.392 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T21:14:24.388+0000 7fa45835c700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fa450103500 0x7fa450077240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:14:24.392 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T21:14:24.388+0000 7fa45835c700 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fa450104700 0x7fa450075890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:14:24.392 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T21:14:24.388+0000 7fa45835c700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa4501050a0 0x7fa450075dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:14:24.392 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T21:14:24.388+0000 7fa45835c700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fa45010f280 con 0x7fa4501050a0 2024-05-03T21:14:24.392 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T21:14:24.388+0000 7fa45835c700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7fa45010f100 con 0x7fa450103500 2024-05-03T21:14:24.392 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T21:14:24.388+0000 7fa45835c700 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7fa45010f400 con 0x7fa450104700 2024-05-03T21:14:24.392 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T21:14:24.388+0000 7fa4560f8700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fa450103500 0x7fa450077240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:14:24.392 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T21:14:24.388+0000 7fa4560f8700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fa450103500 0x7fa450077240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.82:53180/0 (socket says 172.21.15.82:53180) 2024-05-03T21:14:24.392 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T21:14:24.388+0000 7fa4560f8700 1 -- 172.21.15.82:0/3361952164 learned_addr learned my addr 172.21.15.82:0/3361952164 (peer_addr_for_me v2:172.21.15.82:0/0) 2024-05-03T21:14:24.392 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T21:14:24.388+0000 7fa4568f9700 1 --2- 172.21.15.82:0/3361952164 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa4501050a0 0x7fa450075dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:14:24.392 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T21:14:24.388+0000 7fa4558f7700 1 --2- 172.21.15.82:0/3361952164 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fa450104700 0x7fa450075890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:14:24.392 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T21:14:24.388+0000 7fa4560f8700 1 -- 172.21.15.82:0/3361952164 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fa450104700 msgr2=0x7fa450075890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:14:24.392 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T21:14:24.388+0000 7fa4560f8700 1 --2- 172.21.15.82:0/3361952164 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fa450104700 0x7fa450075890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:14:24.392 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T21:14:24.388+0000 7fa4560f8700 1 -- 172.21.15.82:0/3361952164 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa4501050a0 msgr2=0x7fa450075dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:14:24.393 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T21:14:24.388+0000 7fa4560f8700 1 --2- 172.21.15.82:0/3361952164 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa4501050a0 0x7fa450075dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:14:24.393 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T21:14:24.388+0000 7fa4560f8700 1 -- 172.21.15.82:0/3361952164 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa450076680 con 0x7fa450103500 2024-05-03T21:14:24.393 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T21:14:24.388+0000 7fa4568f9700 1 --2- 172.21.15.82:0/3361952164 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa4501050a0 0x7fa450075dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-03T21:14:24.393 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T21:14:24.388+0000 7fa4560f8700 1 --2- 172.21.15.82:0/3361952164 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fa450103500 0x7fa450077240 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fa440005e80 tx=0x7fa440003680 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T21:14:24.393 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T21:14:24.389+0000 7fa4477fe700 1 -- 172.21.15.82:0/3361952164 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa440027070 con 0x7fa450103500 2024-05-03T21:14:24.393 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T21:14:24.389+0000 7fa45835c700 1 -- 172.21.15.82:0/3361952164 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa4501a3990 con 0x7fa450103500 2024-05-03T21:14:24.393 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T21:14:24.389+0000 7fa4477fe700 1 -- 172.21.15.82:0/3361952164 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(28 keys) ==== 1170+0+0 (secure 0 0 0) 0x7fa440022470 con 0x7fa450103500 2024-05-03T21:14:24.393 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T21:14:24.389+0000 7fa4477fe700 1 -- 172.21.15.82:0/3361952164 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa44000f6a0 con 0x7fa450103500 2024-05-03T21:14:24.393 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T21:14:24.389+0000 7fa45835c700 1 -- 172.21.15.82:0/3361952164 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa4501a3ec0 con 0x7fa450103500 2024-05-03T21:14:24.394 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T21:14:24.389+0000 7fa45835c700 1 -- 172.21.15.82:0/3361952164 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa45006a190 con 0x7fa450103500 2024-05-03T21:14:24.394 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T21:14:24.391+0000 7fa4477fe700 1 -- 172.21.15.82:0/3361952164 <== mon.2 v2:172.21.15.82:3300/0 4 ==== mgrmap(e 16) ==== 149652+0+0 (secure 0 0 0) 0x7fa44002a070 con 0x7fa450103500 2024-05-03T21:14:24.394 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T21:14:24.391+0000 7fa4477fe700 1 --2- 172.21.15.82:0/3361952164 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7fa43c0b2880 0x7fa43c0b4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:14:24.394 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T21:14:24.392+0000 7fa4477fe700 1 -- 172.21.15.82:0/3361952164 <== mon.2 v2:172.21.15.82:3300/0 5 ==== osd_map(28..28 src has 1..28) ==== 3669+0+0 (secure 0 0 0) 0x7fa4400e23e0 con 0x7fa450103500 2024-05-03T21:14:24.394 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T21:14:24.392+0000 7fa4558f7700 1 --2- 172.21.15.82:0/3361952164 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7fa43c0b2880 0x7fa43c0b4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:14:24.394 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T21:14:24.392+0000 7fa4558f7700 1 --2- 172.21.15.82:0/3361952164 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7fa43c0b2880 0x7fa43c0b4d40 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fa44c006fd0 tx=0x7fa44c008040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T21:14:24.397 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T21:14:24.395+0000 7fa4477fe700 1 -- 172.21.15.82:0/3361952164 <== mon.2 v2:172.21.15.82:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191839 (secure 0 0 0) 0x7fa440013070 con 0x7fa450103500 2024-05-03T21:14:24.495 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:14:24 smithi082 ceph-mon[32933]: Detected new or changed devices on smithi067 2024-05-03T21:14:24.495 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:14:24 smithi082 ceph-mon[32933]: Adjusting osd_memory_target on smithi067 to 8544M 2024-05-03T21:14:24.495 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:14:24 smithi082 ceph-mon[32933]: osdmap e28: 4 total, 4 up, 4 in 2024-05-03T21:14:24.574 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:14:24 smithi138 ceph-mon[33110]: Detected new or changed devices on smithi067 2024-05-03T21:14:24.574 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:14:24 smithi138 ceph-mon[33110]: Adjusting osd_memory_target on smithi067 to 8544M 2024-05-03T21:14:24.574 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:14:24 smithi138 ceph-mon[33110]: osdmap e28: 4 total, 4 up, 4 in 2024-05-03T21:14:24.607 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T21:14:24.606+0000 7fa45835c700 1 -- 172.21.15.82:0/3361952164 --> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi082:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) -- 0x7fa4500638e0 con 0x7fa43c0b2880 2024-05-03T21:14:24.721 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:14:24 smithi067 ceph-mon[29730]: Detected new or changed devices on smithi067 2024-05-03T21:14:24.721 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:14:24 smithi067 ceph-mon[29730]: Adjusting osd_memory_target on smithi067 to 8544M 2024-05-03T21:14:24.721 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:14:24 smithi067 ceph-mon[29730]: osdmap e28: 4 total, 4 up, 4 in 2024-05-03T21:14:25.495 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:14:25 smithi082 ceph-mon[32933]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 129 MiB used, 357 GiB / 358 GiB avail 2024-05-03T21:14:25.495 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:14:25 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-03T21:14:25.495 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:14:25 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-03T21:14:25.495 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:14:25 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T21:14:25.495 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:14:25 smithi082 ceph-mon[32933]: osdmap e29: 4 total, 4 up, 4 in 2024-05-03T21:14:25.574 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:14:25 smithi138 ceph-mon[33110]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 129 MiB used, 357 GiB / 358 GiB avail 2024-05-03T21:14:25.574 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:14:25 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-03T21:14:25.574 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:14:25 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-03T21:14:25.575 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:14:25 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T21:14:25.575 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:14:25 smithi138 ceph-mon[33110]: osdmap e29: 4 total, 4 up, 4 in 2024-05-03T21:14:25.622 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:14:25 smithi067 ceph-mon[29730]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 129 MiB used, 357 GiB / 358 GiB avail 2024-05-03T21:14:25.622 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:14:25 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-03T21:14:25.622 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:14:25 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-03T21:14:25.622 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:14:25 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T21:14:25.623 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:14:25 smithi067 ceph-mon[29730]: osdmap e29: 4 total, 4 up, 4 in 2024-05-03T21:14:26.365 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:14:26 smithi082 ceph-mon[32933]: from='client.24203 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi082:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-05-03T21:14:26.574 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:14:26 smithi138 ceph-mon[33110]: from='client.24203 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi082:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-05-03T21:14:26.721 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:14:26 smithi067 ceph-mon[29730]: from='client.24203 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi082:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-05-03T21:14:27.721 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:14:27 smithi067 ceph-mon[29730]: pgmap v105: 1 pgs: 1 activating+degraded; 577 KiB data, 128 MiB used, 357 GiB / 358 GiB avail; 2/6 objects degraded (33.333%) 2024-05-03T21:14:27.721 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:14:27 smithi067 ceph-mon[29730]: Health check failed: Degraded data redundancy: 2/6 objects degraded (33.333%), 1 pg degraded (PG_DEGRADED) 2024-05-03T21:14:27.721 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:14:27 smithi067 ceph-mon[29730]: from='client.? 172.21.15.82:0/2150546310' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c3fd0ffd-e23f-4279-b679-9d95a144cf32"}]: dispatch 2024-05-03T21:14:27.721 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:14:27 smithi067 ceph-mon[29730]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c3fd0ffd-e23f-4279-b679-9d95a144cf32"}]: dispatch 2024-05-03T21:14:27.721 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:14:27 smithi067 ceph-mon[29730]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c3fd0ffd-e23f-4279-b679-9d95a144cf32"}]': finished 2024-05-03T21:14:27.721 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:14:27 smithi067 ceph-mon[29730]: osdmap e30: 5 total, 4 up, 5 in 2024-05-03T21:14:27.721 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:14:27 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-03T21:14:27.745 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:14:27 smithi082 ceph-mon[32933]: pgmap v105: 1 pgs: 1 activating+degraded; 577 KiB data, 128 MiB used, 357 GiB / 358 GiB avail; 2/6 objects degraded (33.333%) 2024-05-03T21:14:27.745 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:14:27 smithi082 ceph-mon[32933]: Health check failed: Degraded data redundancy: 2/6 objects degraded (33.333%), 1 pg degraded (PG_DEGRADED) 2024-05-03T21:14:27.745 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:14:27 smithi082 ceph-mon[32933]: from='client.? 172.21.15.82:0/2150546310' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c3fd0ffd-e23f-4279-b679-9d95a144cf32"}]: dispatch 2024-05-03T21:14:27.746 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:14:27 smithi082 ceph-mon[32933]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c3fd0ffd-e23f-4279-b679-9d95a144cf32"}]: dispatch 2024-05-03T21:14:27.746 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:14:27 smithi082 ceph-mon[32933]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c3fd0ffd-e23f-4279-b679-9d95a144cf32"}]': finished 2024-05-03T21:14:27.746 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:14:27 smithi082 ceph-mon[32933]: osdmap e30: 5 total, 4 up, 5 in 2024-05-03T21:14:27.746 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:14:27 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-03T21:14:27.770 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:14:27 smithi138 ceph-mon[33110]: pgmap v105: 1 pgs: 1 activating+degraded; 577 KiB data, 128 MiB used, 357 GiB / 358 GiB avail; 2/6 objects degraded (33.333%) 2024-05-03T21:14:27.770 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:14:27 smithi138 ceph-mon[33110]: Health check failed: Degraded data redundancy: 2/6 objects degraded (33.333%), 1 pg degraded (PG_DEGRADED) 2024-05-03T21:14:27.770 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:14:27 smithi138 ceph-mon[33110]: from='client.? 172.21.15.82:0/2150546310' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c3fd0ffd-e23f-4279-b679-9d95a144cf32"}]: dispatch 2024-05-03T21:14:27.770 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:14:27 smithi138 ceph-mon[33110]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c3fd0ffd-e23f-4279-b679-9d95a144cf32"}]: dispatch 2024-05-03T21:14:27.770 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:14:27 smithi138 ceph-mon[33110]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c3fd0ffd-e23f-4279-b679-9d95a144cf32"}]': finished 2024-05-03T21:14:27.770 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:14:27 smithi138 ceph-mon[33110]: osdmap e30: 5 total, 4 up, 5 in 2024-05-03T21:14:27.770 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:14:27 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-03T21:14:28.721 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:14:28 smithi067 ceph-mon[29730]: from='client.? 172.21.15.82:0/4054226274' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-03T21:14:28.745 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:14:28 smithi082 ceph-mon[32933]: from='client.? 172.21.15.82:0/4054226274' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-03T21:14:28.824 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:14:28 smithi138 ceph-mon[33110]: from='client.? 172.21.15.82:0/4054226274' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-03T21:14:29.721 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:14:29 smithi067 ceph-mon[29730]: pgmap v107: 1 pgs: 1 activating+degraded; 577 KiB data, 128 MiB used, 357 GiB / 358 GiB avail; 2/6 objects degraded (33.333%) 2024-05-03T21:14:29.745 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:14:29 smithi082 ceph-mon[32933]: pgmap v107: 1 pgs: 1 activating+degraded; 577 KiB data, 128 MiB used, 357 GiB / 358 GiB avail; 2/6 objects degraded (33.333%) 2024-05-03T21:14:29.824 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:14:29 smithi138 ceph-mon[33110]: pgmap v107: 1 pgs: 1 activating+degraded; 577 KiB data, 128 MiB used, 357 GiB / 358 GiB avail; 2/6 objects degraded (33.333%) 2024-05-03T21:14:31.422 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:14:31 smithi082 ceph-mon[32933]: pgmap v108: 1 pgs: 1 activating+degraded; 577 KiB data, 129 MiB used, 357 GiB / 358 GiB avail; 2/6 objects degraded (33.333%) 2024-05-03T21:14:31.721 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:14:31 smithi067 ceph-mon[29730]: pgmap v108: 1 pgs: 1 activating+degraded; 577 KiB data, 129 MiB used, 357 GiB / 358 GiB avail; 2/6 objects degraded (33.333%) 2024-05-03T21:14:31.824 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:14:31 smithi138 ceph-mon[33110]: pgmap v108: 1 pgs: 1 activating+degraded; 577 KiB data, 129 MiB used, 357 GiB / 358 GiB avail; 2/6 objects degraded (33.333%) 2024-05-03T21:14:32.721 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:14:32 smithi067 ceph-mon[29730]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 129 MiB used, 357 GiB / 358 GiB avail; 75 KiB/s, 0 objects/s recovering 2024-05-03T21:14:32.721 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:14:32 smithi067 ceph-mon[29730]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 2/6 objects degraded (33.333%), 1 pg degraded) 2024-05-03T21:14:32.721 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:14:32 smithi067 ceph-mon[29730]: Cluster is now healthy 2024-05-03T21:14:32.729 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:14:32 smithi082 ceph-mon[32933]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 129 MiB used, 357 GiB / 358 GiB avail; 75 KiB/s, 0 objects/s recovering 2024-05-03T21:14:32.730 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:14:32 smithi082 ceph-mon[32933]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 2/6 objects degraded (33.333%), 1 pg degraded) 2024-05-03T21:14:32.730 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:14:32 smithi082 ceph-mon[32933]: Cluster is now healthy 2024-05-03T21:14:32.824 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:14:32 smithi138 ceph-mon[33110]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 129 MiB used, 357 GiB / 358 GiB avail; 75 KiB/s, 0 objects/s recovering 2024-05-03T21:14:32.824 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:14:32 smithi138 ceph-mon[33110]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 2/6 objects degraded (33.333%), 1 pg degraded) 2024-05-03T21:14:32.824 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:14:32 smithi138 ceph-mon[33110]: Cluster is now healthy 2024-05-03T21:14:35.069 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:14:34 smithi082 ceph-mon[32933]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 129 MiB used, 357 GiB / 358 GiB avail; 66 KiB/s, 0 objects/s recovering 2024-05-03T21:14:35.069 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:14:34 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-05-03T21:14:35.069 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:14:34 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T21:14:35.074 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:14:34 smithi138 ceph-mon[33110]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 129 MiB used, 357 GiB / 358 GiB avail; 66 KiB/s, 0 objects/s recovering 2024-05-03T21:14:35.074 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:14:34 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-05-03T21:14:35.074 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:14:34 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T21:14:35.221 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:14:34 smithi067 ceph-mon[29730]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 129 MiB used, 357 GiB / 358 GiB avail; 66 KiB/s, 0 objects/s recovering 2024-05-03T21:14:35.221 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:14:34 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-05-03T21:14:35.221 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:14:34 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T21:14:36.074 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:14:35 smithi138 ceph-mon[33110]: Deploying daemon osd.4 on smithi082 2024-05-03T21:14:36.093 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:14:35 smithi082 ceph-mon[32933]: Deploying daemon osd.4 on smithi082 2024-05-03T21:14:36.221 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:14:35 smithi067 ceph-mon[29730]: Deploying daemon osd.4 on smithi082 2024-05-03T21:14:36.905 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:14:36 smithi082 ceph-mon[32933]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 129 MiB used, 357 GiB / 358 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-05-03T21:14:37.074 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:14:36 smithi138 ceph-mon[33110]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 129 MiB used, 357 GiB / 358 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-05-03T21:14:37.221 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:14:36 smithi067 ceph-mon[29730]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 129 MiB used, 357 GiB / 358 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-05-03T21:14:38.921 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:14:38 smithi082 ceph-mon[32933]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 129 MiB used, 357 GiB / 358 GiB avail; 55 KiB/s, 0 objects/s recovering 2024-05-03T21:14:39.074 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:14:38 smithi138 ceph-mon[33110]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 129 MiB used, 357 GiB / 358 GiB avail; 55 KiB/s, 0 objects/s recovering 2024-05-03T21:14:39.221 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:14:38 smithi067 ceph-mon[29730]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 129 MiB used, 357 GiB / 358 GiB avail; 55 KiB/s, 0 objects/s recovering 2024-05-03T21:14:40.745 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:14:40 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:14:40.746 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:14:40 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:14:40.746 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:14:40 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T21:14:40.746 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:14:40 smithi082 ceph-mon[32933]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 129 MiB used, 357 GiB / 358 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-05-03T21:14:40.825 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:14:40 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:14:40.825 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:14:40 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:14:40.825 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:14:40 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T21:14:40.825 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:14:40 smithi138 ceph-mon[33110]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 129 MiB used, 357 GiB / 358 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-05-03T21:14:40.971 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:14:40 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:14:40.971 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:14:40 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:14:40.971 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:14:40 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T21:14:40.971 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:14:40 smithi067 ceph-mon[29730]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 129 MiB used, 357 GiB / 358 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-05-03T21:14:43.074 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T21:14:43.071+0000 7fa4477fe700 1 -- 172.21.15.82:0/3361952164 <== mgr.14150 v2:172.21.15.67:6800/604451592 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7fa4500638e0 con 0x7fa43c0b2880 2024-05-03T21:14:43.074 INFO:teuthology.orchestra.run.smithi082.stdout:Created osd(s) 4 on host 'smithi082' 2024-05-03T21:14:43.076 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T21:14:43.074+0000 7fa45835c700 1 -- 172.21.15.82:0/3361952164 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7fa43c0b2880 msgr2=0x7fa43c0b4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:14:43.076 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T21:14:43.074+0000 7fa45835c700 1 --2- 172.21.15.82:0/3361952164 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7fa43c0b2880 0x7fa43c0b4d40 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fa44c006fd0 tx=0x7fa44c008040 comp rx=0 tx=0).stop 2024-05-03T21:14:43.076 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T21:14:43.074+0000 7fa45835c700 1 -- 172.21.15.82:0/3361952164 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fa450103500 msgr2=0x7fa450077240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:14:43.076 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T21:14:43.074+0000 7fa45835c700 1 --2- 172.21.15.82:0/3361952164 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fa450103500 0x7fa450077240 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fa440005e80 tx=0x7fa440003680 comp rx=0 tx=0).stop 2024-05-03T21:14:43.076 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T21:14:43.074+0000 7fa45835c700 1 -- 172.21.15.82:0/3361952164 shutdown_connections 2024-05-03T21:14:43.076 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T21:14:43.074+0000 7fa45835c700 1 --2- 172.21.15.82:0/3361952164 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7fa43c0b2880 0x7fa43c0b4d40 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:14:43.076 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T21:14:43.074+0000 7fa45835c700 1 --2- 172.21.15.82:0/3361952164 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa4501050a0 0x7fa450075dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:14:43.077 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T21:14:43.074+0000 7fa45835c700 1 --2- 172.21.15.82:0/3361952164 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fa450103500 0x7fa450077240 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:14:43.077 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T21:14:43.074+0000 7fa45835c700 1 --2- 172.21.15.82:0/3361952164 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fa450104700 0x7fa450075890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:14:43.077 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T21:14:43.074+0000 7fa45835c700 1 -- 172.21.15.82:0/3361952164 >> 172.21.15.82:0/3361952164 conn(0x7fa4500feaa0 msgr2=0x7fa450107a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T21:14:43.077 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T21:14:43.074+0000 7fa45835c700 1 -- 172.21.15.82:0/3361952164 shutdown_connections 2024-05-03T21:14:43.077 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T21:14:43.074+0000 7fa45835c700 1 -- 172.21.15.82:0/3361952164 wait complete. 2024-05-03T21:14:43.245 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:14:43 smithi082 ceph-mon[32933]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 129 MiB used, 357 GiB / 358 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-05-03T21:14:43.245 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:14:43 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:14:43.245 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:14:43 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:14:43.245 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:14:43 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T21:14:43.246 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:14:43 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T21:14:43.246 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:14:43 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:14:43.246 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:14:43 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:14:43.246 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:14:43 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:14:43.246 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:14:43 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T21:14:43.471 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:14:43 smithi067 ceph-mon[29730]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 129 MiB used, 357 GiB / 358 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-05-03T21:14:43.471 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:14:43 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:14:43.472 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:14:43 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:14:43.472 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:14:43 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T21:14:43.472 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:14:43 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T21:14:43.472 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:14:43 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:14:43.472 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:14:43 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:14:43.472 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:14:43 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:14:43.472 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:14:43 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T21:14:43.528 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:14:43 smithi138 ceph-mon[33110]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 129 MiB used, 357 GiB / 358 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-05-03T21:14:43.528 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:14:43 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:14:43.528 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:14:43 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:14:43.528 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:14:43 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T21:14:43.528 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:14:43 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T21:14:43.528 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:14:43 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:14:43.528 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:14:43 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:14:43.529 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:14:43 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:14:43.529 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:14:43 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T21:14:43.636 DEBUG:teuthology.orchestra.run.smithi082:osd.4> sudo journalctl -f -n 0 -u ceph-713bc57a-0991-11ef-bc97-c7b262605968@osd.4.service 2024-05-03T21:14:43.639 INFO:tasks.cephadm:Deploying osd.5 on smithi138 with /dev/vg_nvme/lv_3... 2024-05-03T21:14:43.639 DEBUG:teuthology.orchestra.run.smithi138:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9a047fe7599a549042d06a3fb5b6813a203eabea shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 713bc57a-0991-11ef-bc97-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-05-03T21:14:43.881 INFO:teuthology.orchestra.run.smithi138.stderr:Inferring config /var/lib/ceph/713bc57a-0991-11ef-bc97-c7b262605968/mon.c/config 2024-05-03T21:14:44.303 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:14:44 smithi138 ceph-mon[33110]: from='osd.4 [v2:172.21.15.82:6808/4031818951,v1:172.21.15.82:6809/4031818951]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-05-03T21:14:44.304 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:14:44 smithi138 ceph-mon[33110]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-05-03T21:14:44.448 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:14:44 smithi082 ceph-mon[32933]: from='osd.4 [v2:172.21.15.82:6808/4031818951,v1:172.21.15.82:6809/4031818951]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-05-03T21:14:44.448 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:14:44 smithi082 ceph-mon[32933]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-05-03T21:14:44.471 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:14:44 smithi067 ceph-mon[29730]: from='osd.4 [v2:172.21.15.82:6808/4031818951,v1:172.21.15.82:6809/4031818951]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-05-03T21:14:44.471 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:14:44 smithi067 ceph-mon[29730]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-05-03T21:14:45.471 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:14:45 smithi067 ceph-mon[29730]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 129 MiB used, 357 GiB / 358 GiB avail 2024-05-03T21:14:45.471 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:14:45 smithi067 ceph-mon[29730]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-05-03T21:14:45.471 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:14:45 smithi067 ceph-mon[29730]: osdmap e31: 5 total, 4 up, 5 in 2024-05-03T21:14:45.471 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:14:45 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-03T21:14:45.471 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:14:45 smithi067 ceph-mon[29730]: from='osd.4 [v2:172.21.15.82:6808/4031818951,v1:172.21.15.82:6809/4031818951]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi082", "root=default"]}]: dispatch 2024-05-03T21:14:45.471 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:14:45 smithi067 ceph-mon[29730]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi082", "root=default"]}]: dispatch 2024-05-03T21:14:45.495 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:14:45 smithi082 ceph-mon[32933]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 129 MiB used, 357 GiB / 358 GiB avail 2024-05-03T21:14:45.495 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:14:45 smithi082 ceph-mon[32933]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-05-03T21:14:45.495 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:14:45 smithi082 ceph-mon[32933]: osdmap e31: 5 total, 4 up, 5 in 2024-05-03T21:14:45.496 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:14:45 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-03T21:14:45.496 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:14:45 smithi082 ceph-mon[32933]: from='osd.4 [v2:172.21.15.82:6808/4031818951,v1:172.21.15.82:6809/4031818951]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi082", "root=default"]}]: dispatch 2024-05-03T21:14:45.496 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:14:45 smithi082 ceph-mon[32933]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi082", "root=default"]}]: dispatch 2024-05-03T21:14:45.496 INFO:journalctl@ceph.osd.4.smithi082.stdout:May 03 21:14:45 smithi082 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-4[44044]: 2024-05-03T21:14:45.178+0000 7f16072f6700 -1 osd.4 0 waiting for initial osdmap 2024-05-03T21:14:45.496 INFO:journalctl@ceph.osd.4.smithi082.stdout:May 03 21:14:45 smithi082 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-4[44044]: 2024-05-03T21:14:45.184+0000 7f16030e9700 -1 osd.4 32 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-05-03T21:14:45.574 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:14:45 smithi138 ceph-mon[33110]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 129 MiB used, 357 GiB / 358 GiB avail 2024-05-03T21:14:45.574 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:14:45 smithi138 ceph-mon[33110]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-05-03T21:14:45.574 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:14:45 smithi138 ceph-mon[33110]: osdmap e31: 5 total, 4 up, 5 in 2024-05-03T21:14:45.574 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:14:45 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-03T21:14:45.575 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:14:45 smithi138 ceph-mon[33110]: from='osd.4 [v2:172.21.15.82:6808/4031818951,v1:172.21.15.82:6809/4031818951]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi082", "root=default"]}]: dispatch 2024-05-03T21:14:45.575 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:14:45 smithi138 ceph-mon[33110]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi082", "root=default"]}]: dispatch 2024-05-03T21:14:45.756 INFO:teuthology.orchestra.run.smithi138.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-05-03T21:14:45.756 INFO:teuthology.orchestra.run.smithi138.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-05-03T21:14:45.757 INFO:teuthology.orchestra.run.smithi138.stderr: stderr: 10+0 records in 2024-05-03T21:14:45.757 INFO:teuthology.orchestra.run.smithi138.stderr:10+0 records out 2024-05-03T21:14:45.757 INFO:teuthology.orchestra.run.smithi138.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0171836 s, 610 MB/s 2024-05-03T21:14:45.757 INFO:teuthology.orchestra.run.smithi138.stderr:--> Zapping successful for: 2024-05-03T21:14:46.355 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:14:46 smithi138 ceph-mon[33110]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi082", "root=default"]}]': finished 2024-05-03T21:14:46.355 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:14:46 smithi138 ceph-mon[33110]: osdmap e32: 5 total, 4 up, 5 in 2024-05-03T21:14:46.355 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:14:46 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-03T21:14:46.355 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:14:46 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-03T21:14:46.379 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:14:46 smithi082 ceph-mon[32933]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi082", "root=default"]}]': finished 2024-05-03T21:14:46.379 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:14:46 smithi082 ceph-mon[32933]: osdmap e32: 5 total, 4 up, 5 in 2024-05-03T21:14:46.379 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:14:46 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-03T21:14:46.379 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:14:46 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-03T21:14:46.383 DEBUG:teuthology.orchestra.run.smithi138:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9a047fe7599a549042d06a3fb5b6813a203eabea shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 713bc57a-0991-11ef-bc97-c7b262605968 -- ceph orch daemon add osd smithi138:vg_nvme/lv_3 2024-05-03T21:14:46.471 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:14:46 smithi067 ceph-mon[29730]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi082", "root=default"]}]': finished 2024-05-03T21:14:46.471 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:14:46 smithi067 ceph-mon[29730]: osdmap e32: 5 total, 4 up, 5 in 2024-05-03T21:14:46.471 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:14:46 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-03T21:14:46.471 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:14:46 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-03T21:14:46.624 INFO:teuthology.orchestra.run.smithi138.stderr:Inferring config /var/lib/ceph/713bc57a-0991-11ef-bc97-c7b262605968/mon.c/config 2024-05-03T21:14:47.324 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:14:47 smithi138 ceph-mon[33110]: purged_snaps scrub starts 2024-05-03T21:14:47.324 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:14:47 smithi138 ceph-mon[33110]: purged_snaps scrub ok 2024-05-03T21:14:47.325 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:14:47 smithi138 ceph-mon[33110]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 129 MiB used, 357 GiB / 358 GiB avail 2024-05-03T21:14:47.325 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:14:47 smithi138 ceph-mon[33110]: osd.4 [v2:172.21.15.82:6808/4031818951,v1:172.21.15.82:6809/4031818951] boot 2024-05-03T21:14:47.325 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:14:47 smithi138 ceph-mon[33110]: osdmap e33: 5 total, 5 up, 5 in 2024-05-03T21:14:47.325 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:14:47 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-03T21:14:47.325 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:14:47 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:14:47.325 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:14:47 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:14:47.325 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:14:47 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-05-03T21:14:47.325 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:14:47 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-05-03T21:14:47.325 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:14:47 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:14:47.325 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:14:47 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T21:14:47.325 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:14:47 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T21:14:47.325 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:14:47 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:14:47.471 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:14:47 smithi067 ceph-mon[29730]: purged_snaps scrub starts 2024-05-03T21:14:47.471 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:14:47 smithi067 ceph-mon[29730]: purged_snaps scrub ok 2024-05-03T21:14:47.471 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:14:47 smithi067 ceph-mon[29730]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 129 MiB used, 357 GiB / 358 GiB avail 2024-05-03T21:14:47.471 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:14:47 smithi067 ceph-mon[29730]: osd.4 [v2:172.21.15.82:6808/4031818951,v1:172.21.15.82:6809/4031818951] boot 2024-05-03T21:14:47.471 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:14:47 smithi067 ceph-mon[29730]: osdmap e33: 5 total, 5 up, 5 in 2024-05-03T21:14:47.471 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:14:47 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-03T21:14:47.472 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:14:47 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:14:47.472 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:14:47 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:14:47.472 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:14:47 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-05-03T21:14:47.472 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:14:47 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-05-03T21:14:47.472 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:14:47 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:14:47.472 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:14:47 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T21:14:47.472 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:14:47 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T21:14:47.472 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:14:47 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:14:47.495 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:14:47 smithi082 ceph-mon[32933]: purged_snaps scrub starts 2024-05-03T21:14:47.495 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:14:47 smithi082 ceph-mon[32933]: purged_snaps scrub ok 2024-05-03T21:14:47.495 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:14:47 smithi082 ceph-mon[32933]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 129 MiB used, 357 GiB / 358 GiB avail 2024-05-03T21:14:47.495 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:14:47 smithi082 ceph-mon[32933]: osd.4 [v2:172.21.15.82:6808/4031818951,v1:172.21.15.82:6809/4031818951] boot 2024-05-03T21:14:47.495 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:14:47 smithi082 ceph-mon[32933]: osdmap e33: 5 total, 5 up, 5 in 2024-05-03T21:14:47.496 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:14:47 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-03T21:14:47.496 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:14:47 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:14:47.496 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:14:47 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:14:47.496 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:14:47 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-05-03T21:14:47.496 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:14:47 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-05-03T21:14:47.496 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:14:47 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:14:47.496 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:14:47 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T21:14:47.496 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:14:47 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T21:14:47.496 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:14:47 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:14:48.196 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:14:48.191+0000 7fe8c3b3b700 1 -- 172.21.15.138:0/947442273 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fe8bc1035f0 msgr2=0x7fe8bc103a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:14:48.196 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:14:48.191+0000 7fe8c3b3b700 1 --2- 172.21.15.138:0/947442273 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fe8bc1035f0 0x7fe8bc103a50 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fe8ac009990 tx=0x7fe8ac009ca0 comp rx=0 tx=0).stop 2024-05-03T21:14:48.196 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:14:48.191+0000 7fe8c3b3b700 1 -- 172.21.15.138:0/947442273 shutdown_connections 2024-05-03T21:14:48.196 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:14:48.191+0000 7fe8c3b3b700 1 --2- 172.21.15.138:0/947442273 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe8bc103f90 0x7fe8bc10a840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:14:48.196 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:14:48.191+0000 7fe8c3b3b700 1 --2- 172.21.15.138:0/947442273 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fe8bc1023f0 0x7fe8bc102810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:14:48.196 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:14:48.191+0000 7fe8c3b3b700 1 --2- 172.21.15.138:0/947442273 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fe8bc1035f0 0x7fe8bc103a50 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:14:48.197 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:14:48.191+0000 7fe8c3b3b700 1 -- 172.21.15.138:0/947442273 >> 172.21.15.138:0/947442273 conn(0x7fe8bc0fd990 msgr2=0x7fe8bc0ffdd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T21:14:48.197 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:14:48.191+0000 7fe8c3b3b700 1 -- 172.21.15.138:0/947442273 shutdown_connections 2024-05-03T21:14:48.197 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:14:48.191+0000 7fe8c3b3b700 1 -- 172.21.15.138:0/947442273 wait complete. 2024-05-03T21:14:48.197 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:14:48.192+0000 7fe8c3b3b700 1 Processor -- start 2024-05-03T21:14:48.197 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:14:48.192+0000 7fe8c3b3b700 1 -- start start 2024-05-03T21:14:48.197 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:14:48.192+0000 7fe8c3b3b700 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fe8bc1023f0 0x7fe8bc06c570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:14:48.197 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:14:48.192+0000 7fe8c3b3b700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe8bc1035f0 0x7fe8bc19e510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:14:48.197 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:14:48.192+0000 7fe8c3b3b700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fe8bc103f90 0x7fe8bc1a0b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:14:48.197 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:14:48.192+0000 7fe8c3b3b700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fe8bc10e100 con 0x7fe8bc1035f0 2024-05-03T21:14:48.197 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:14:48.192+0000 7fe8c3b3b700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7fe8bc10df80 con 0x7fe8bc103f90 2024-05-03T21:14:48.197 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:14:48.192+0000 7fe8c3b3b700 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7fe8bc10e280 con 0x7fe8bc1023f0 2024-05-03T21:14:48.197 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:14:48.192+0000 7fe8c18d7700 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fe8bc1023f0 0x7fe8bc06c570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:14:48.198 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:14:48.192+0000 7fe8c18d7700 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fe8bc1023f0 0x7fe8bc06c570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3300/0 says I am v2:172.21.15.138:44988/0 (socket says 172.21.15.138:44988) 2024-05-03T21:14:48.198 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:14:48.192+0000 7fe8c18d7700 1 -- 172.21.15.138:0/423296562 learned_addr learned my addr 172.21.15.138:0/423296562 (peer_addr_for_me v2:172.21.15.138:0/0) 2024-05-03T21:14:48.198 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:14:48.192+0000 7fe8c20d8700 1 --2- 172.21.15.138:0/423296562 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fe8bc103f90 0x7fe8bc1a0b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:14:48.198 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:14:48.192+0000 7fe8c10d6700 1 --2- 172.21.15.138:0/423296562 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe8bc1035f0 0x7fe8bc19e510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:14:48.198 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:14:48.193+0000 7fe8c18d7700 1 -- 172.21.15.138:0/423296562 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fe8bc103f90 msgr2=0x7fe8bc1a0b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:14:48.198 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:14:48.193+0000 7fe8c18d7700 1 --2- 172.21.15.138:0/423296562 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fe8bc103f90 0x7fe8bc1a0b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:14:48.198 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:14:48.193+0000 7fe8c18d7700 1 -- 172.21.15.138:0/423296562 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe8bc1035f0 msgr2=0x7fe8bc19e510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:14:48.198 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:14:48.193+0000 7fe8c18d7700 1 --2- 172.21.15.138:0/423296562 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe8bc1035f0 0x7fe8bc19e510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:14:48.198 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:14:48.193+0000 7fe8c18d7700 1 -- 172.21.15.138:0/423296562 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe8bc1a1420 con 0x7fe8bc1023f0 2024-05-03T21:14:48.198 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:14:48.193+0000 7fe8c18d7700 1 --2- 172.21.15.138:0/423296562 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fe8bc1023f0 0x7fe8bc06c570 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fe8b800e970 tx=0x7fe8b800ec80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T21:14:48.199 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:14:48.197+0000 7fe8b2ffd700 1 -- 172.21.15.138:0/423296562 <== mon.1 v2:172.21.15.138:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe8b8004220 con 0x7fe8bc1023f0 2024-05-03T21:14:48.199 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:14:48.197+0000 7fe8b2ffd700 1 -- 172.21.15.138:0/423296562 <== mon.1 v2:172.21.15.138:3300/0 2 ==== config(28 keys) ==== 1170+0+0 (secure 0 0 0) 0x7fe8b800cce0 con 0x7fe8bc1023f0 2024-05-03T21:14:48.199 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:14:48.197+0000 7fe8b2ffd700 1 -- 172.21.15.138:0/423296562 <== mon.1 v2:172.21.15.138:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe8b8003e60 con 0x7fe8bc1023f0 2024-05-03T21:14:48.199 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:14:48.197+0000 7fe8c3b3b700 1 -- 172.21.15.138:0/423296562 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe8bc1a1710 con 0x7fe8bc1023f0 2024-05-03T21:14:48.199 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:14:48.197+0000 7fe8c3b3b700 1 -- 172.21.15.138:0/423296562 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe8bc1a1c20 con 0x7fe8bc1023f0 2024-05-03T21:14:48.199 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:14:48.198+0000 7fe8c3b3b700 1 -- 172.21.15.138:0/423296562 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe8bc06a190 con 0x7fe8bc1023f0 2024-05-03T21:14:48.200 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:14:48.198+0000 7fe8b2ffd700 1 -- 172.21.15.138:0/423296562 <== mon.1 v2:172.21.15.138:3300/0 4 ==== mgrmap(e 16) ==== 149652+0+0 (secure 0 0 0) 0x7fe8b800c7e0 con 0x7fe8bc1023f0 2024-05-03T21:14:48.201 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:14:48.199+0000 7fe8b2ffd700 1 --2- 172.21.15.138:0/423296562 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7fe8a80b2880 0x7fe8a80b4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:14:48.201 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:14:48.199+0000 7fe8b2ffd700 1 -- 172.21.15.138:0/423296562 <== mon.1 v2:172.21.15.138:3300/0 5 ==== osd_map(34..34 src has 1..34) ==== 4101+0+0 (secure 0 0 0) 0x7fe8b80e1360 con 0x7fe8bc1023f0 2024-05-03T21:14:48.201 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:14:48.199+0000 7fe8c10d6700 1 --2- 172.21.15.138:0/423296562 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7fe8a80b2880 0x7fe8a80b4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:14:48.201 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:14:48.200+0000 7fe8c10d6700 1 --2- 172.21.15.138:0/423296562 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7fe8a80b2880 0x7fe8a80b4d40 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fe8ac00b2e0 tx=0x7fe8ac005fd0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T21:14:48.205 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:14:48.204+0000 7fe8b2ffd700 1 -- 172.21.15.138:0/423296562 <== mon.1 v2:172.21.15.138:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191839 (secure 0 0 0) 0x7fe8b80aa0f0 con 0x7fe8bc1023f0 2024-05-03T21:14:48.206 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:14:48 smithi138 ceph-mon[33110]: Detected new or changed devices on smithi082 2024-05-03T21:14:48.206 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:14:48 smithi138 ceph-mon[33110]: Adjusting osd_memory_target on smithi082 to 8544M 2024-05-03T21:14:48.206 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:14:48 smithi138 ceph-mon[33110]: osdmap e34: 5 total, 5 up, 5 in 2024-05-03T21:14:48.422 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:14:48.420+0000 7fe8c3b3b700 1 -- 172.21.15.138:0/423296562 --> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi138:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) -- 0x7fe8bc0638e0 con 0x7fe8a80b2880 2024-05-03T21:14:48.471 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:14:48 smithi067 ceph-mon[29730]: Detected new or changed devices on smithi082 2024-05-03T21:14:48.471 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:14:48 smithi067 ceph-mon[29730]: Adjusting osd_memory_target on smithi082 to 8544M 2024-05-03T21:14:48.472 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:14:48 smithi067 ceph-mon[29730]: osdmap e34: 5 total, 5 up, 5 in 2024-05-03T21:14:48.495 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:14:48 smithi082 ceph-mon[32933]: Detected new or changed devices on smithi082 2024-05-03T21:14:48.495 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:14:48 smithi082 ceph-mon[32933]: Adjusting osd_memory_target on smithi082 to 8544M 2024-05-03T21:14:48.496 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:14:48 smithi082 ceph-mon[32933]: osdmap e34: 5 total, 5 up, 5 in 2024-05-03T21:14:49.300 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:14:49 smithi138 ceph-mon[33110]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 155 MiB used, 447 GiB / 447 GiB avail 2024-05-03T21:14:49.301 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:14:49 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-03T21:14:49.301 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:14:49 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-03T21:14:49.301 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:14:49 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T21:14:49.471 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:14:49 smithi067 ceph-mon[29730]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 155 MiB used, 447 GiB / 447 GiB avail 2024-05-03T21:14:49.471 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:14:49 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-03T21:14:49.471 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:14:49 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-03T21:14:49.471 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:14:49 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T21:14:49.495 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:14:49 smithi082 ceph-mon[32933]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 155 MiB used, 447 GiB / 447 GiB avail 2024-05-03T21:14:49.495 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:14:49 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-03T21:14:49.495 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:14:49 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-03T21:14:49.496 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:14:49 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T21:14:50.324 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:14:50 smithi138 ceph-mon[33110]: from='client.24217 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi138:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-05-03T21:14:50.471 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:14:50 smithi067 ceph-mon[29730]: from='client.24217 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi138:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-05-03T21:14:50.495 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:14:50 smithi082 ceph-mon[32933]: from='client.24217 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi138:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-05-03T21:14:51.471 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:14:51 smithi067 ceph-mon[29730]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 155 MiB used, 447 GiB / 447 GiB avail 2024-05-03T21:14:51.471 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:14:51 smithi067 ceph-mon[29730]: from='client.? 172.21.15.138:0/2212143710' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "540ef0d9-a6dc-43c6-b09e-e87340ba24cf"}]: dispatch 2024-05-03T21:14:51.471 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:14:51 smithi067 ceph-mon[29730]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "540ef0d9-a6dc-43c6-b09e-e87340ba24cf"}]: dispatch 2024-05-03T21:14:51.471 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:14:51 smithi067 ceph-mon[29730]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "540ef0d9-a6dc-43c6-b09e-e87340ba24cf"}]': finished 2024-05-03T21:14:51.471 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:14:51 smithi067 ceph-mon[29730]: osdmap e35: 6 total, 5 up, 6 in 2024-05-03T21:14:51.471 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:14:51 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-03T21:14:51.495 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:14:51 smithi082 ceph-mon[32933]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 155 MiB used, 447 GiB / 447 GiB avail 2024-05-03T21:14:51.495 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:14:51 smithi082 ceph-mon[32933]: from='client.? 172.21.15.138:0/2212143710' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "540ef0d9-a6dc-43c6-b09e-e87340ba24cf"}]: dispatch 2024-05-03T21:14:51.495 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:14:51 smithi082 ceph-mon[32933]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "540ef0d9-a6dc-43c6-b09e-e87340ba24cf"}]: dispatch 2024-05-03T21:14:51.495 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:14:51 smithi082 ceph-mon[32933]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "540ef0d9-a6dc-43c6-b09e-e87340ba24cf"}]': finished 2024-05-03T21:14:51.496 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:14:51 smithi082 ceph-mon[32933]: osdmap e35: 6 total, 5 up, 6 in 2024-05-03T21:14:51.496 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:14:51 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-03T21:14:51.574 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:14:51 smithi138 ceph-mon[33110]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 155 MiB used, 447 GiB / 447 GiB avail 2024-05-03T21:14:51.574 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:14:51 smithi138 ceph-mon[33110]: from='client.? 172.21.15.138:0/2212143710' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "540ef0d9-a6dc-43c6-b09e-e87340ba24cf"}]: dispatch 2024-05-03T21:14:51.574 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:14:51 smithi138 ceph-mon[33110]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "540ef0d9-a6dc-43c6-b09e-e87340ba24cf"}]: dispatch 2024-05-03T21:14:51.574 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:14:51 smithi138 ceph-mon[33110]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "540ef0d9-a6dc-43c6-b09e-e87340ba24cf"}]': finished 2024-05-03T21:14:51.575 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:14:51 smithi138 ceph-mon[33110]: osdmap e35: 6 total, 5 up, 6 in 2024-05-03T21:14:51.575 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:14:51 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-03T21:14:52.471 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:14:52 smithi067 ceph-mon[29730]: from='client.? 172.21.15.138:0/1736416540' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-03T21:14:52.495 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:14:52 smithi082 ceph-mon[32933]: from='client.? 172.21.15.138:0/1736416540' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-03T21:14:52.574 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:14:52 smithi138 ceph-mon[33110]: from='client.? 172.21.15.138:0/1736416540' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-03T21:14:53.471 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:14:53 smithi067 ceph-mon[29730]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 155 MiB used, 447 GiB / 447 GiB avail 2024-05-03T21:14:53.495 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:14:53 smithi082 ceph-mon[32933]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 155 MiB used, 447 GiB / 447 GiB avail 2024-05-03T21:14:53.574 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:14:53 smithi138 ceph-mon[33110]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 155 MiB used, 447 GiB / 447 GiB avail 2024-05-03T21:14:55.471 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:14:55 smithi067 ceph-mon[29730]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 155 MiB used, 447 GiB / 447 GiB avail 2024-05-03T21:14:55.495 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:14:55 smithi082 ceph-mon[32933]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 155 MiB used, 447 GiB / 447 GiB avail 2024-05-03T21:14:55.512 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:14:55 smithi138 ceph-mon[33110]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 155 MiB used, 447 GiB / 447 GiB avail 2024-05-03T21:14:57.229 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:14:57 smithi138 ceph-mon[33110]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 155 MiB used, 447 GiB / 447 GiB avail 2024-05-03T21:14:57.471 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:14:57 smithi067 ceph-mon[29730]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 155 MiB used, 447 GiB / 447 GiB avail 2024-05-03T21:14:57.495 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:14:57 smithi082 ceph-mon[32933]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 155 MiB used, 447 GiB / 447 GiB avail 2024-05-03T21:14:59.471 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:14:59 smithi067 ceph-mon[29730]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 155 MiB used, 447 GiB / 447 GiB avail 2024-05-03T21:14:59.471 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:14:59 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-05-03T21:14:59.471 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:14:59 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T21:14:59.471 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:14:59 smithi067 ceph-mon[29730]: Deploying daemon osd.5 on smithi138 2024-05-03T21:14:59.495 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:14:59 smithi082 ceph-mon[32933]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 155 MiB used, 447 GiB / 447 GiB avail 2024-05-03T21:14:59.495 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:14:59 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-05-03T21:14:59.495 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:14:59 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T21:14:59.495 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:14:59 smithi082 ceph-mon[32933]: Deploying daemon osd.5 on smithi138 2024-05-03T21:14:59.574 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:14:59 smithi138 ceph-mon[33110]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 155 MiB used, 447 GiB / 447 GiB avail 2024-05-03T21:14:59.575 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:14:59 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-05-03T21:14:59.575 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:14:59 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T21:14:59.575 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:14:59 smithi138 ceph-mon[33110]: Deploying daemon osd.5 on smithi138 2024-05-03T21:15:01.471 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:15:01 smithi067 ceph-mon[29730]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 155 MiB used, 447 GiB / 447 GiB avail 2024-05-03T21:15:01.495 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:15:01 smithi082 ceph-mon[32933]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 155 MiB used, 447 GiB / 447 GiB avail 2024-05-03T21:15:01.574 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:15:01 smithi138 ceph-mon[33110]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 155 MiB used, 447 GiB / 447 GiB avail 2024-05-03T21:15:03.476 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:15:03 smithi138 ceph-mon[33110]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 155 MiB used, 447 GiB / 447 GiB avail 2024-05-03T21:15:03.495 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:15:03 smithi082 ceph-mon[32933]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 155 MiB used, 447 GiB / 447 GiB avail 2024-05-03T21:15:03.721 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:15:03 smithi067 ceph-mon[29730]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 155 MiB used, 447 GiB / 447 GiB avail 2024-05-03T21:15:04.824 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:15:04 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:15:04.825 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:15:04 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:15:04.825 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:15:04 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T21:15:04.825 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:15:04 smithi138 ceph-mon[33110]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 155 MiB used, 447 GiB / 447 GiB avail 2024-05-03T21:15:04.971 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:15:04 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:15:04.971 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:15:04 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:15:04.971 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:15:04 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T21:15:04.971 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:15:04 smithi067 ceph-mon[29730]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 155 MiB used, 447 GiB / 447 GiB avail 2024-05-03T21:15:04.995 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:15:04 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:15:04.995 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:15:04 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:15:04.996 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:15:04 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T21:15:04.996 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:15:04 smithi082 ceph-mon[32933]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 155 MiB used, 447 GiB / 447 GiB avail 2024-05-03T21:15:07.221 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:15:06 smithi067 ceph-mon[29730]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 155 MiB used, 447 GiB / 447 GiB avail 2024-05-03T21:15:07.221 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:15:06 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:15:07.221 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:15:06 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:15:07.221 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:15:06 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T21:15:07.222 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:15:06 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T21:15:07.222 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:15:06 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:15:07.238 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:15:06 smithi138 ceph-mon[33110]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 155 MiB used, 447 GiB / 447 GiB avail 2024-05-03T21:15:07.239 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:15:06 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:15:07.239 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:15:06 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:15:07.239 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:15:06 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T21:15:07.239 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:15:06 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T21:15:07.239 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:15:06 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:15:07.245 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:15:06 smithi082 ceph-mon[32933]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 155 MiB used, 447 GiB / 447 GiB avail 2024-05-03T21:15:07.245 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:15:06 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:15:07.245 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:15:06 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:15:07.245 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:15:06 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T21:15:07.245 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:15:06 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T21:15:07.246 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:15:06 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:15:07.296 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:15:07.293+0000 7fe8b2ffd700 1 -- 172.21.15.138:0/423296562 <== mgr.14150 v2:172.21.15.67:6800/604451592 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7fe8bc0638e0 con 0x7fe8a80b2880 2024-05-03T21:15:07.296 INFO:teuthology.orchestra.run.smithi138.stdout:Created osd(s) 5 on host 'smithi138' 2024-05-03T21:15:07.299 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:15:07.295+0000 7fe8c3b3b700 1 -- 172.21.15.138:0/423296562 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7fe8a80b2880 msgr2=0x7fe8a80b4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:15:07.299 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:15:07.295+0000 7fe8c3b3b700 1 --2- 172.21.15.138:0/423296562 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7fe8a80b2880 0x7fe8a80b4d40 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fe8ac00b2e0 tx=0x7fe8ac005fd0 comp rx=0 tx=0).stop 2024-05-03T21:15:07.299 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:15:07.296+0000 7fe8c3b3b700 1 -- 172.21.15.138:0/423296562 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fe8bc1023f0 msgr2=0x7fe8bc06c570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:15:07.299 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:15:07.296+0000 7fe8c3b3b700 1 --2- 172.21.15.138:0/423296562 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fe8bc1023f0 0x7fe8bc06c570 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fe8b800e970 tx=0x7fe8b800ec80 comp rx=0 tx=0).stop 2024-05-03T21:15:07.299 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:15:07.296+0000 7fe8c3b3b700 1 -- 172.21.15.138:0/423296562 shutdown_connections 2024-05-03T21:15:07.299 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:15:07.296+0000 7fe8c3b3b700 1 --2- 172.21.15.138:0/423296562 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7fe8a80b2880 0x7fe8a80b4d40 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:15:07.299 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:15:07.296+0000 7fe8c3b3b700 1 --2- 172.21.15.138:0/423296562 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fe8bc103f90 0x7fe8bc1a0b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:15:07.299 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:15:07.296+0000 7fe8c3b3b700 1 --2- 172.21.15.138:0/423296562 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fe8bc1023f0 0x7fe8bc06c570 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:15:07.299 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:15:07.296+0000 7fe8c3b3b700 1 --2- 172.21.15.138:0/423296562 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe8bc1035f0 0x7fe8bc19e510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:15:07.299 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:15:07.296+0000 7fe8c3b3b700 1 -- 172.21.15.138:0/423296562 >> 172.21.15.138:0/423296562 conn(0x7fe8bc0fd990 msgr2=0x7fe8bc1068f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T21:15:07.299 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:15:07.296+0000 7fe8c3b3b700 1 -- 172.21.15.138:0/423296562 shutdown_connections 2024-05-03T21:15:07.299 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:15:07.296+0000 7fe8c3b3b700 1 -- 172.21.15.138:0/423296562 wait complete. 2024-05-03T21:15:07.922 DEBUG:teuthology.orchestra.run.smithi138:osd.5> sudo journalctl -f -n 0 -u ceph-713bc57a-0991-11ef-bc97-c7b262605968@osd.5.service 2024-05-03T21:15:07.924 INFO:tasks.cephadm:Deploying osd.6 on smithi067 with /dev/vg_nvme/lv_2... 2024-05-03T21:15:07.924 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9a047fe7599a549042d06a3fb5b6813a203eabea shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 713bc57a-0991-11ef-bc97-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-05-03T21:15:08.182 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/713bc57a-0991-11ef-bc97-c7b262605968/mon.a/config 2024-05-03T21:15:08.552 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:15:08 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:15:08.552 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:15:08 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T21:15:08.553 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:15:08 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:15:08.553 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:15:08 smithi067 ceph-mon[29730]: from='osd.5 [v2:172.21.15.138:6808/2521889499,v1:172.21.15.138:6809/2521889499]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-05-03T21:15:08.575 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:15:08 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:15:08.575 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:15:08 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T21:15:08.575 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:15:08 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:15:08.575 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:15:08 smithi138 ceph-mon[33110]: from='osd.5 [v2:172.21.15.138:6808/2521889499,v1:172.21.15.138:6809/2521889499]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-05-03T21:15:08.745 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:15:08 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:15:08.745 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:15:08 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T21:15:08.745 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:15:08 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:15:08.746 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:15:08 smithi082 ceph-mon[32933]: from='osd.5 [v2:172.21.15.138:6808/2521889499,v1:172.21.15.138:6809/2521889499]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-05-03T21:15:09.493 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:15:09 smithi067 ceph-mon[29730]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 155 MiB used, 447 GiB / 447 GiB avail 2024-05-03T21:15:09.493 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:15:09 smithi067 ceph-mon[29730]: from='osd.5 [v2:172.21.15.138:6808/2521889499,v1:172.21.15.138:6809/2521889499]' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-05-03T21:15:09.493 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:15:09 smithi067 ceph-mon[29730]: osdmap e36: 6 total, 5 up, 6 in 2024-05-03T21:15:09.494 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:15:09 smithi067 ceph-mon[29730]: from='osd.5 [v2:172.21.15.138:6808/2521889499,v1:172.21.15.138:6809/2521889499]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi138", "root=default"]}]: dispatch 2024-05-03T21:15:09.494 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:15:09 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-03T21:15:09.574 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:15:09 smithi138 ceph-mon[33110]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 155 MiB used, 447 GiB / 447 GiB avail 2024-05-03T21:15:09.574 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:15:09 smithi138 ceph-mon[33110]: from='osd.5 [v2:172.21.15.138:6808/2521889499,v1:172.21.15.138:6809/2521889499]' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-05-03T21:15:09.574 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:15:09 smithi138 ceph-mon[33110]: osdmap e36: 6 total, 5 up, 6 in 2024-05-03T21:15:09.575 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:15:09 smithi138 ceph-mon[33110]: from='osd.5 [v2:172.21.15.138:6808/2521889499,v1:172.21.15.138:6809/2521889499]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi138", "root=default"]}]: dispatch 2024-05-03T21:15:09.575 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:15:09 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-03T21:15:09.745 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:15:09 smithi082 ceph-mon[32933]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 155 MiB used, 447 GiB / 447 GiB avail 2024-05-03T21:15:09.745 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:15:09 smithi082 ceph-mon[32933]: from='osd.5 [v2:172.21.15.138:6808/2521889499,v1:172.21.15.138:6809/2521889499]' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-05-03T21:15:09.746 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:15:09 smithi082 ceph-mon[32933]: osdmap e36: 6 total, 5 up, 6 in 2024-05-03T21:15:09.746 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:15:09 smithi082 ceph-mon[32933]: from='osd.5 [v2:172.21.15.138:6808/2521889499,v1:172.21.15.138:6809/2521889499]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi138", "root=default"]}]: dispatch 2024-05-03T21:15:09.746 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:15:09 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-03T21:15:10.087 INFO:teuthology.orchestra.run.smithi067.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-05-03T21:15:10.087 INFO:teuthology.orchestra.run.smithi067.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-05-03T21:15:10.087 INFO:teuthology.orchestra.run.smithi067.stderr: stderr: 10+0 records in 2024-05-03T21:15:10.088 INFO:teuthology.orchestra.run.smithi067.stderr:10+0 records out 2024-05-03T21:15:10.088 INFO:teuthology.orchestra.run.smithi067.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0166862 s, 628 MB/s 2024-05-03T21:15:10.088 INFO:teuthology.orchestra.run.smithi067.stderr: stderr: 2024-05-03T21:15:10.088 INFO:teuthology.orchestra.run.smithi067.stderr:--> Zapping successful for: 2024-05-03T21:15:10.341 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:15:10 smithi138 ceph-mon[33110]: from='osd.5 [v2:172.21.15.138:6808/2521889499,v1:172.21.15.138:6809/2521889499]' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi138", "root=default"]}]': finished 2024-05-03T21:15:10.341 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:15:10 smithi138 ceph-mon[33110]: osdmap e37: 6 total, 5 up, 6 in 2024-05-03T21:15:10.341 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:15:10 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-03T21:15:10.341 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:15:10 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-03T21:15:10.341 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:15:10 smithi138 ceph-mon[33110]: from='osd.5 [v2:172.21.15.138:6808/2521889499,v1:172.21.15.138:6809/2521889499]' entity='osd.5' 2024-05-03T21:15:10.342 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:15:10 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-03T21:15:10.342 INFO:journalctl@ceph.osd.5.smithi138.stdout:May 03 21:15:10 smithi138 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-5[45227]: 2024-05-03T21:15:10.178+0000 7fe1d65b1700 -1 osd.5 0 waiting for initial osdmap 2024-05-03T21:15:10.342 INFO:journalctl@ceph.osd.5.smithi138.stdout:May 03 21:15:10 smithi138 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-5[45227]: 2024-05-03T21:15:10.338+0000 7fe1ce39c700 -1 osd.5 37 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-05-03T21:15:10.579 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:15:10 smithi067 ceph-mon[29730]: from='osd.5 [v2:172.21.15.138:6808/2521889499,v1:172.21.15.138:6809/2521889499]' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi138", "root=default"]}]': finished 2024-05-03T21:15:10.579 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:15:10 smithi067 ceph-mon[29730]: osdmap e37: 6 total, 5 up, 6 in 2024-05-03T21:15:10.579 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:15:10 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-03T21:15:10.579 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:15:10 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-03T21:15:10.579 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:15:10 smithi067 ceph-mon[29730]: from='osd.5 [v2:172.21.15.138:6808/2521889499,v1:172.21.15.138:6809/2521889499]' entity='osd.5' 2024-05-03T21:15:10.580 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:15:10 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-03T21:15:10.745 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:15:10 smithi082 ceph-mon[32933]: from='osd.5 [v2:172.21.15.138:6808/2521889499,v1:172.21.15.138:6809/2521889499]' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi138", "root=default"]}]': finished 2024-05-03T21:15:10.745 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:15:10 smithi082 ceph-mon[32933]: osdmap e37: 6 total, 5 up, 6 in 2024-05-03T21:15:10.745 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:15:10 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-03T21:15:10.746 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:15:10 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-03T21:15:10.746 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:15:10 smithi082 ceph-mon[32933]: from='osd.5 [v2:172.21.15.138:6808/2521889499,v1:172.21.15.138:6809/2521889499]' entity='osd.5' 2024-05-03T21:15:10.746 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:15:10 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-03T21:15:10.752 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9a047fe7599a549042d06a3fb5b6813a203eabea shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 713bc57a-0991-11ef-bc97-c7b262605968 -- ceph orch daemon add osd smithi067:vg_nvme/lv_2 2024-05-03T21:15:11.021 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/713bc57a-0991-11ef-bc97-c7b262605968/mon.a/config 2024-05-03T21:15:11.411 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:15:11 smithi067 ceph-mon[29730]: purged_snaps scrub starts 2024-05-03T21:15:11.411 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:15:11 smithi067 ceph-mon[29730]: purged_snaps scrub ok 2024-05-03T21:15:11.411 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:15:11 smithi067 ceph-mon[29730]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 155 MiB used, 447 GiB / 447 GiB avail 2024-05-03T21:15:11.411 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:15:11 smithi067 ceph-mon[29730]: osdmap e38: 6 total, 5 up, 6 in 2024-05-03T21:15:11.411 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:15:11 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-03T21:15:11.412 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:15:11 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:15:11.412 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:15:11 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:15:11.412 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:15:11 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-05-03T21:15:11.412 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:15:11 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-05-03T21:15:11.412 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:15:11 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:15:11.412 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:15:11 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T21:15:11.412 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:15:11 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T21:15:11.412 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:15:11 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:15:11.412 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:15:11 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-03T21:15:11.574 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:15:11 smithi138 ceph-mon[33110]: purged_snaps scrub starts 2024-05-03T21:15:11.575 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:15:11 smithi138 ceph-mon[33110]: purged_snaps scrub ok 2024-05-03T21:15:11.575 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:15:11 smithi138 ceph-mon[33110]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 155 MiB used, 447 GiB / 447 GiB avail 2024-05-03T21:15:11.575 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:15:11 smithi138 ceph-mon[33110]: osdmap e38: 6 total, 5 up, 6 in 2024-05-03T21:15:11.575 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:15:11 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-03T21:15:11.575 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:15:11 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:15:11.575 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:15:11 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:15:11.575 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:15:11 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-05-03T21:15:11.575 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:15:11 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-05-03T21:15:11.575 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:15:11 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:15:11.575 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:15:11 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T21:15:11.575 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:15:11 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T21:15:11.575 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:15:11 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:15:11.575 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:15:11 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-03T21:15:11.745 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:15:11 smithi082 ceph-mon[32933]: purged_snaps scrub starts 2024-05-03T21:15:11.745 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:15:11 smithi082 ceph-mon[32933]: purged_snaps scrub ok 2024-05-03T21:15:11.746 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:15:11 smithi082 ceph-mon[32933]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 155 MiB used, 447 GiB / 447 GiB avail 2024-05-03T21:15:11.746 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:15:11 smithi082 ceph-mon[32933]: osdmap e38: 6 total, 5 up, 6 in 2024-05-03T21:15:11.746 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:15:11 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-03T21:15:11.746 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:15:11 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:15:11.746 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:15:11 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:15:11.746 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:15:11 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-05-03T21:15:11.746 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:15:11 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-05-03T21:15:11.746 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:15:11 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:15:11.746 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:15:11 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T21:15:11.746 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:15:11 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T21:15:11.747 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:15:11 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:15:11.747 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:15:11 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-03T21:15:12.430 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:15:12 smithi067 ceph-mon[29730]: Detected new or changed devices on smithi138 2024-05-03T21:15:12.430 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:15:12 smithi067 ceph-mon[29730]: Adjusting osd_memory_target on smithi138 to 8544M 2024-05-03T21:15:12.430 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:15:12 smithi067 ceph-mon[29730]: osd.5 [v2:172.21.15.138:6808/2521889499,v1:172.21.15.138:6809/2521889499] boot 2024-05-03T21:15:12.430 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:15:12 smithi067 ceph-mon[29730]: osdmap e39: 6 total, 6 up, 6 in 2024-05-03T21:15:12.430 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:15:12 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-03T21:15:12.430 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:15:12 smithi067 ceph-mon[29730]: osdmap e40: 6 total, 6 up, 6 in 2024-05-03T21:15:12.574 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:15:12 smithi138 ceph-mon[33110]: Detected new or changed devices on smithi138 2024-05-03T21:15:12.575 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:15:12 smithi138 ceph-mon[33110]: Adjusting osd_memory_target on smithi138 to 8544M 2024-05-03T21:15:12.575 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:15:12 smithi138 ceph-mon[33110]: osd.5 [v2:172.21.15.138:6808/2521889499,v1:172.21.15.138:6809/2521889499] boot 2024-05-03T21:15:12.575 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:15:12 smithi138 ceph-mon[33110]: osdmap e39: 6 total, 6 up, 6 in 2024-05-03T21:15:12.575 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:15:12 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-03T21:15:12.575 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:15:12 smithi138 ceph-mon[33110]: osdmap e40: 6 total, 6 up, 6 in 2024-05-03T21:15:12.639 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:15:12.634+0000 7fbf34a9f700 1 -- 172.21.15.67:0/1108343874 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbf30102400 msgr2=0x7fbf30102820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:15:12.639 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:15:12.634+0000 7fbf34a9f700 1 --2- 172.21.15.67:0/1108343874 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbf30102400 0x7fbf30102820 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fbf200099c0 tx=0x7fbf20009cd0 comp rx=0 tx=0).stop 2024-05-03T21:15:12.639 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:15:12.634+0000 7fbf34a9f700 1 -- 172.21.15.67:0/1108343874 shutdown_connections 2024-05-03T21:15:12.639 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:15:12.634+0000 7fbf34a9f700 1 --2- 172.21.15.67:0/1108343874 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbf30102400 0x7fbf30102820 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:15:12.639 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:15:12.634+0000 7fbf34a9f700 1 --2- 172.21.15.67:0/1108343874 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fbf30103fa0 0x7fbf3010a850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:15:12.639 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:15:12.634+0000 7fbf34a9f700 1 --2- 172.21.15.67:0/1108343874 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fbf30103600 0x7fbf30103a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:15:12.639 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:15:12.634+0000 7fbf34a9f700 1 -- 172.21.15.67:0/1108343874 >> 172.21.15.67:0/1108343874 conn(0x7fbf300fd9a0 msgr2=0x7fbf300ffde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T21:15:12.639 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:15:12.634+0000 7fbf34a9f700 1 -- 172.21.15.67:0/1108343874 shutdown_connections 2024-05-03T21:15:12.639 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:15:12.634+0000 7fbf34a9f700 1 -- 172.21.15.67:0/1108343874 wait complete. 2024-05-03T21:15:12.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:15:12.635+0000 7fbf34a9f700 1 Processor -- start 2024-05-03T21:15:12.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:15:12.635+0000 7fbf34a9f700 1 -- start start 2024-05-03T21:15:12.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:15:12.635+0000 7fbf34a9f700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbf30102400 0x7fbf3019bac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:15:12.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:15:12.635+0000 7fbf34a9f700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fbf30103600 0x7fbf3019c000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:15:12.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:15:12.635+0000 7fbf34a9f700 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fbf30103fa0 0x7fbf301a3080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:15:12.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:15:12.635+0000 7fbf34a9f700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fbf3010e180 con 0x7fbf30102400 2024-05-03T21:15:12.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:15:12.635+0000 7fbf34a9f700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7fbf3010e000 con 0x7fbf30103600 2024-05-03T21:15:12.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:15:12.635+0000 7fbf34a9f700 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7fbf3010e300 con 0x7fbf30103fa0 2024-05-03T21:15:12.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:15:12.635+0000 7fbf2e59c700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbf30102400 0x7fbf3019bac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:15:12.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:15:12.635+0000 7fbf2e59c700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbf30102400 0x7fbf3019bac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:36236/0 (socket says 172.21.15.67:36236) 2024-05-03T21:15:12.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:15:12.635+0000 7fbf2e59c700 1 -- 172.21.15.67:0/3232930693 learned_addr learned my addr 172.21.15.67:0/3232930693 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-03T21:15:12.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:15:12.635+0000 7fbf2ed9d700 1 --2- 172.21.15.67:0/3232930693 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fbf30103fa0 0x7fbf301a3080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:15:12.643 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:15:12.635+0000 7fbf2e59c700 1 -- 172.21.15.67:0/3232930693 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fbf30103fa0 msgr2=0x7fbf301a3080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:15:12.643 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:15:12.635+0000 7fbf2e59c700 1 --2- 172.21.15.67:0/3232930693 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fbf30103fa0 0x7fbf301a3080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:15:12.643 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:15:12.635+0000 7fbf2dd9b700 1 --2- 172.21.15.67:0/3232930693 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fbf30103600 0x7fbf3019c000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:15:12.643 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:15:12.635+0000 7fbf2e59c700 1 -- 172.21.15.67:0/3232930693 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fbf30103600 msgr2=0x7fbf3019c000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:15:12.643 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:15:12.635+0000 7fbf2e59c700 1 --2- 172.21.15.67:0/3232930693 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fbf30103600 0x7fbf3019c000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:15:12.643 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:15:12.635+0000 7fbf2e59c700 1 -- 172.21.15.67:0/3232930693 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbf301a3720 con 0x7fbf30102400 2024-05-03T21:15:12.643 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:15:12.635+0000 7fbf2ed9d700 1 --2- 172.21.15.67:0/3232930693 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fbf30103fa0 0x7fbf301a3080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-03T21:15:12.643 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:15:12.635+0000 7fbf2dd9b700 1 --2- 172.21.15.67:0/3232930693 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fbf30103600 0x7fbf3019c000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-03T21:15:12.643 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:15:12.636+0000 7fbf2e59c700 1 --2- 172.21.15.67:0/3232930693 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbf30102400 0x7fbf3019bac0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fbf200047b0 tx=0x7fbf20009580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T21:15:12.643 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:15:12.636+0000 7fbf1f7fe700 1 -- 172.21.15.67:0/3232930693 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf20026070 con 0x7fbf30102400 2024-05-03T21:15:12.643 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:15:12.636+0000 7fbf34a9f700 1 -- 172.21.15.67:0/3232930693 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbf301a39b0 con 0x7fbf30102400 2024-05-03T21:15:12.643 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:15:12.636+0000 7fbf1f7fe700 1 -- 172.21.15.67:0/3232930693 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(28 keys) ==== 1170+0+0 (secure 0 0 0) 0x7fbf2000f460 con 0x7fbf30102400 2024-05-03T21:15:12.643 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:15:12.636+0000 7fbf1f7fe700 1 -- 172.21.15.67:0/3232930693 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf2000bea0 con 0x7fbf30102400 2024-05-03T21:15:12.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:15:12.636+0000 7fbf34a9f700 1 -- 172.21.15.67:0/3232930693 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbf301a3e60 con 0x7fbf30102400 2024-05-03T21:15:12.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:15:12.637+0000 7fbf34a9f700 1 -- 172.21.15.67:0/3232930693 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbf3006a190 con 0x7fbf30102400 2024-05-03T21:15:12.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:15:12.638+0000 7fbf1f7fe700 1 -- 172.21.15.67:0/3232930693 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 16) ==== 149652+0+0 (secure 0 0 0) 0x7fbf20029070 con 0x7fbf30102400 2024-05-03T21:15:12.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:15:12.639+0000 7fbf1f7fe700 1 --2- 172.21.15.67:0/3232930693 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7fbf180b2900 0x7fbf180b4dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:15:12.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:15:12.639+0000 7fbf2dd9b700 1 --2- 172.21.15.67:0/3232930693 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7fbf180b2900 0x7fbf180b4dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:15:12.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:15:12.639+0000 7fbf1f7fe700 1 -- 172.21.15.67:0/3232930693 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(40..40 src has 1..40) ==== 4533+0+0 (secure 0 0 0) 0x7fbf200e2210 con 0x7fbf30102400 2024-05-03T21:15:12.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:15:12.639+0000 7fbf2dd9b700 1 --2- 172.21.15.67:0/3232930693 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7fbf180b2900 0x7fbf180b4dc0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fbf24005fd0 tx=0x7fbf24005ee0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T21:15:12.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:15:12.642+0000 7fbf1f7fe700 1 -- 172.21.15.67:0/3232930693 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191839 (secure 0 0 0) 0x7fbf20030020 con 0x7fbf30102400 2024-05-03T21:15:12.745 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:15:12 smithi082 ceph-mon[32933]: Detected new or changed devices on smithi138 2024-05-03T21:15:12.746 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:15:12 smithi082 ceph-mon[32933]: Adjusting osd_memory_target on smithi138 to 8544M 2024-05-03T21:15:12.746 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:15:12 smithi082 ceph-mon[32933]: osd.5 [v2:172.21.15.138:6808/2521889499,v1:172.21.15.138:6809/2521889499] boot 2024-05-03T21:15:12.746 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:15:12 smithi082 ceph-mon[32933]: osdmap e39: 6 total, 6 up, 6 in 2024-05-03T21:15:12.746 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:15:12 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-03T21:15:12.746 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:15:12 smithi082 ceph-mon[32933]: osdmap e40: 6 total, 6 up, 6 in 2024-05-03T21:15:12.868 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:15:12.866+0000 7fbf34a9f700 1 -- 172.21.15.67:0/3232930693 --> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi067:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) -- 0x7fbf300ffb50 con 0x7fbf180b2900 2024-05-03T21:15:13.574 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:15:13 smithi138 ceph-mon[33110]: pgmap v138: 1 pgs: 1 peering; 577 KiB data, 155 MiB used, 447 GiB / 447 GiB avail 2024-05-03T21:15:13.575 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:15:13 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-03T21:15:13.575 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:15:13 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-03T21:15:13.575 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:15:13 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T21:15:13.575 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:15:13 smithi138 ceph-mon[33110]: osdmap e41: 6 total, 6 up, 6 in 2024-05-03T21:15:13.629 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:15:13 smithi067 ceph-mon[29730]: pgmap v138: 1 pgs: 1 peering; 577 KiB data, 155 MiB used, 447 GiB / 447 GiB avail 2024-05-03T21:15:13.629 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:15:13 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-03T21:15:13.629 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:15:13 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-03T21:15:13.629 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:15:13 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T21:15:13.629 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:15:13 smithi067 ceph-mon[29730]: osdmap e41: 6 total, 6 up, 6 in 2024-05-03T21:15:13.745 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:15:13 smithi082 ceph-mon[32933]: pgmap v138: 1 pgs: 1 peering; 577 KiB data, 155 MiB used, 447 GiB / 447 GiB avail 2024-05-03T21:15:13.745 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:15:13 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-03T21:15:13.745 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:15:13 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-03T21:15:13.745 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:15:13 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T21:15:13.746 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:15:13 smithi082 ceph-mon[32933]: osdmap e41: 6 total, 6 up, 6 in 2024-05-03T21:15:14.574 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:15:14 smithi138 ceph-mon[33110]: from='client.14367 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi067:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-05-03T21:15:14.696 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:15:14 smithi067 ceph-mon[29730]: from='client.14367 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi067:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-05-03T21:15:14.745 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:15:14 smithi082 ceph-mon[32933]: from='client.14367 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi067:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-05-03T21:15:15.574 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:15:15 smithi138 ceph-mon[33110]: pgmap v141: 1 pgs: 1 peering; 577 KiB data, 186 MiB used, 536 GiB / 536 GiB avail 2024-05-03T21:15:15.646 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:15:15 smithi082 ceph-mon[32933]: pgmap v141: 1 pgs: 1 peering; 577 KiB data, 186 MiB used, 536 GiB / 536 GiB avail 2024-05-03T21:15:15.721 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:15:15 smithi067 ceph-mon[29730]: pgmap v141: 1 pgs: 1 peering; 577 KiB data, 186 MiB used, 536 GiB / 536 GiB avail 2024-05-03T21:15:16.471 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:15:16 smithi067 ceph-mon[29730]: from='client.? 172.21.15.67:0/3263316169' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "49a89fd0-958a-4be9-a8db-6022f946153e"}]: dispatch 2024-05-03T21:15:16.471 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:15:16 smithi067 ceph-mon[29730]: from='client.? 172.21.15.67:0/3263316169' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "49a89fd0-958a-4be9-a8db-6022f946153e"}]': finished 2024-05-03T21:15:16.471 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:15:16 smithi067 ceph-mon[29730]: osdmap e42: 7 total, 6 up, 7 in 2024-05-03T21:15:16.471 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:15:16 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-03T21:15:16.746 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:15:16 smithi082 ceph-mon[32933]: from='client.? 172.21.15.67:0/3263316169' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "49a89fd0-958a-4be9-a8db-6022f946153e"}]: dispatch 2024-05-03T21:15:16.746 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:15:16 smithi082 ceph-mon[32933]: from='client.? 172.21.15.67:0/3263316169' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "49a89fd0-958a-4be9-a8db-6022f946153e"}]': finished 2024-05-03T21:15:16.746 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:15:16 smithi082 ceph-mon[32933]: osdmap e42: 7 total, 6 up, 7 in 2024-05-03T21:15:16.746 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:15:16 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-03T21:15:16.824 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:15:16 smithi138 ceph-mon[33110]: from='client.? 172.21.15.67:0/3263316169' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "49a89fd0-958a-4be9-a8db-6022f946153e"}]: dispatch 2024-05-03T21:15:16.824 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:15:16 smithi138 ceph-mon[33110]: from='client.? 172.21.15.67:0/3263316169' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "49a89fd0-958a-4be9-a8db-6022f946153e"}]': finished 2024-05-03T21:15:16.824 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:15:16 smithi138 ceph-mon[33110]: osdmap e42: 7 total, 6 up, 7 in 2024-05-03T21:15:16.825 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:15:16 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-03T21:15:17.721 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:15:17 smithi067 ceph-mon[29730]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 186 MiB used, 536 GiB / 536 GiB avail; 106 KiB/s, 0 objects/s recovering 2024-05-03T21:15:17.721 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:15:17 smithi067 ceph-mon[29730]: from='client.? 172.21.15.67:0/528665118' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-03T21:15:17.745 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:15:17 smithi082 ceph-mon[32933]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 186 MiB used, 536 GiB / 536 GiB avail; 106 KiB/s, 0 objects/s recovering 2024-05-03T21:15:17.745 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:15:17 smithi082 ceph-mon[32933]: from='client.? 172.21.15.67:0/528665118' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-03T21:15:17.824 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:15:17 smithi138 ceph-mon[33110]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 186 MiB used, 536 GiB / 536 GiB avail; 106 KiB/s, 0 objects/s recovering 2024-05-03T21:15:17.824 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:15:17 smithi138 ceph-mon[33110]: from='client.? 172.21.15.67:0/528665118' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-03T21:15:19.721 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:15:19 smithi067 ceph-mon[29730]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 187 MiB used, 536 GiB / 536 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-05-03T21:15:19.745 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:15:19 smithi082 ceph-mon[32933]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 187 MiB used, 536 GiB / 536 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-05-03T21:15:19.824 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:15:19 smithi138 ceph-mon[33110]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 187 MiB used, 536 GiB / 536 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-05-03T21:15:21.474 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:15:21 smithi067 ceph-mon[29730]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 187 MiB used, 536 GiB / 536 GiB avail; 77 KiB/s, 0 objects/s recovering 2024-05-03T21:15:21.745 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:15:21 smithi082 ceph-mon[32933]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 187 MiB used, 536 GiB / 536 GiB avail; 77 KiB/s, 0 objects/s recovering 2024-05-03T21:15:21.824 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:15:21 smithi138 ceph-mon[33110]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 187 MiB used, 536 GiB / 536 GiB avail; 77 KiB/s, 0 objects/s recovering 2024-05-03T21:15:23.471 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:15:23 smithi067 ceph-mon[29730]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 187 MiB used, 536 GiB / 536 GiB avail; 68 KiB/s, 0 objects/s recovering 2024-05-03T21:15:23.746 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:15:23 smithi082 ceph-mon[32933]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 187 MiB used, 536 GiB / 536 GiB avail; 68 KiB/s, 0 objects/s recovering 2024-05-03T21:15:23.824 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:15:23 smithi138 ceph-mon[33110]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 187 MiB used, 536 GiB / 536 GiB avail; 68 KiB/s, 0 objects/s recovering 2024-05-03T21:15:24.574 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:15:24 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-05-03T21:15:24.575 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:15:24 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T21:15:24.705 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:15:24 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-05-03T21:15:24.706 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:15:24 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T21:15:24.745 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:15:24 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-05-03T21:15:24.745 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:15:24 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T21:15:25.721 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:15:25 smithi067 ceph-mon[29730]: Deploying daemon osd.6 on smithi067 2024-05-03T21:15:25.721 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:15:25 smithi067 ceph-mon[29730]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 187 MiB used, 536 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-05-03T21:15:25.745 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:15:25 smithi082 ceph-mon[32933]: Deploying daemon osd.6 on smithi067 2024-05-03T21:15:25.745 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:15:25 smithi082 ceph-mon[32933]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 187 MiB used, 536 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-05-03T21:15:25.824 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:15:25 smithi138 ceph-mon[33110]: Deploying daemon osd.6 on smithi067 2024-05-03T21:15:25.824 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:15:25 smithi138 ceph-mon[33110]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 187 MiB used, 536 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-05-03T21:15:26.546 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:15:26 smithi067 ceph-mon[29730]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 187 MiB used, 536 GiB / 536 GiB avail 2024-05-03T21:15:26.746 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:15:26 smithi082 ceph-mon[32933]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 187 MiB used, 536 GiB / 536 GiB avail 2024-05-03T21:15:26.824 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:15:26 smithi138 ceph-mon[33110]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 187 MiB used, 536 GiB / 536 GiB avail 2024-05-03T21:15:28.971 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:15:28 smithi067 ceph-mon[29730]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 187 MiB used, 536 GiB / 536 GiB avail 2024-05-03T21:15:29.074 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:15:28 smithi138 ceph-mon[33110]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 187 MiB used, 536 GiB / 536 GiB avail 2024-05-03T21:15:29.245 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:15:28 smithi082 ceph-mon[32933]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 187 MiB used, 536 GiB / 536 GiB avail 2024-05-03T21:15:30.471 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:15:30 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:15:30.472 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:15:30 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:15:30.472 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:15:30 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T21:15:30.495 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:15:30 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:15:30.496 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:15:30 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:15:30.496 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:15:30 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T21:15:30.574 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:15:30 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:15:30.574 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:15:30 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:15:30.574 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:15:30 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T21:15:31.436 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:15:31 smithi067 ceph-mon[29730]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 187 MiB used, 536 GiB / 536 GiB avail 2024-05-03T21:15:31.495 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:15:31 smithi082 ceph-mon[32933]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 187 MiB used, 536 GiB / 536 GiB avail 2024-05-03T21:15:31.574 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:15:31 smithi138 ceph-mon[33110]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 187 MiB used, 536 GiB / 536 GiB avail 2024-05-03T21:15:33.079 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:15:33.076+0000 7fbf1f7fe700 1 -- 172.21.15.67:0/3232930693 <== mgr.14150 v2:172.21.15.67:6800/604451592 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7fbf300ffb50 con 0x7fbf180b2900 2024-05-03T21:15:33.079 INFO:teuthology.orchestra.run.smithi067.stdout:Created osd(s) 6 on host 'smithi067' 2024-05-03T21:15:33.081 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:15:33.079+0000 7fbf34a9f700 1 -- 172.21.15.67:0/3232930693 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7fbf180b2900 msgr2=0x7fbf180b4dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:15:33.081 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:15:33.079+0000 7fbf34a9f700 1 --2- 172.21.15.67:0/3232930693 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7fbf180b2900 0x7fbf180b4dc0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fbf24005fd0 tx=0x7fbf24005ee0 comp rx=0 tx=0).stop 2024-05-03T21:15:33.081 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:15:33.079+0000 7fbf34a9f700 1 -- 172.21.15.67:0/3232930693 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbf30102400 msgr2=0x7fbf3019bac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:15:33.081 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:15:33.079+0000 7fbf34a9f700 1 --2- 172.21.15.67:0/3232930693 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbf30102400 0x7fbf3019bac0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fbf200047b0 tx=0x7fbf20009580 comp rx=0 tx=0).stop 2024-05-03T21:15:33.082 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:15:33.079+0000 7fbf34a9f700 1 -- 172.21.15.67:0/3232930693 shutdown_connections 2024-05-03T21:15:33.082 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:15:33.079+0000 7fbf34a9f700 1 --2- 172.21.15.67:0/3232930693 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7fbf180b2900 0x7fbf180b4dc0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:15:33.082 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:15:33.079+0000 7fbf34a9f700 1 --2- 172.21.15.67:0/3232930693 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbf30102400 0x7fbf3019bac0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:15:33.082 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:15:33.079+0000 7fbf34a9f700 1 --2- 172.21.15.67:0/3232930693 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fbf30103fa0 0x7fbf301a3080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:15:33.082 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:15:33.079+0000 7fbf34a9f700 1 --2- 172.21.15.67:0/3232930693 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fbf30103600 0x7fbf3019c000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:15:33.082 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:15:33.079+0000 7fbf34a9f700 1 -- 172.21.15.67:0/3232930693 >> 172.21.15.67:0/3232930693 conn(0x7fbf300fd9a0 msgr2=0x7fbf300ff440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T21:15:33.082 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:15:33.079+0000 7fbf34a9f700 1 -- 172.21.15.67:0/3232930693 shutdown_connections 2024-05-03T21:15:33.082 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:15:33.079+0000 7fbf34a9f700 1 -- 172.21.15.67:0/3232930693 wait complete. 2024-05-03T21:15:33.388 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:15:33 smithi067 ceph-mon[29730]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 187 MiB used, 536 GiB / 536 GiB avail 2024-05-03T21:15:33.389 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:15:33 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:15:33.389 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:15:33 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:15:33.389 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:15:33 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T21:15:33.389 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:15:33 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T21:15:33.389 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:15:33 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:15:33.389 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:15:33 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:15:33.389 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:15:33 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:15:33.389 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:15:33 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T21:15:33.389 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:15:33 smithi067 ceph-mon[29730]: from='osd.6 [v2:172.21.15.67:6818/3269682731,v1:172.21.15.67:6819/3269682731]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-05-03T21:15:33.498 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:15:33 smithi082 ceph-mon[32933]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 187 MiB used, 536 GiB / 536 GiB avail 2024-05-03T21:15:33.499 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:15:33 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:15:33.499 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:15:33 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:15:33.499 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:15:33 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T21:15:33.499 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:15:33 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T21:15:33.499 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:15:33 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:15:33.499 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:15:33 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:15:33.499 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:15:33 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:15:33.499 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:15:33 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T21:15:33.500 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:15:33 smithi082 ceph-mon[32933]: from='osd.6 [v2:172.21.15.67:6818/3269682731,v1:172.21.15.67:6819/3269682731]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-05-03T21:15:33.574 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:15:33 smithi138 ceph-mon[33110]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 187 MiB used, 536 GiB / 536 GiB avail 2024-05-03T21:15:33.575 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:15:33 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:15:33.575 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:15:33 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:15:33.575 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:15:33 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T21:15:33.575 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:15:33 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T21:15:33.575 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:15:33 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:15:33.575 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:15:33 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:15:33.575 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:15:33 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:15:33.575 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:15:33 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T21:15:33.575 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:15:33 smithi138 ceph-mon[33110]: from='osd.6 [v2:172.21.15.67:6818/3269682731,v1:172.21.15.67:6819/3269682731]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-05-03T21:15:33.743 DEBUG:teuthology.orchestra.run.smithi067:osd.6> sudo journalctl -f -n 0 -u ceph-713bc57a-0991-11ef-bc97-c7b262605968@osd.6.service 2024-05-03T21:15:33.746 INFO:tasks.cephadm:Deploying osd.7 on smithi082 with /dev/vg_nvme/lv_2... 2024-05-03T21:15:33.746 DEBUG:teuthology.orchestra.run.smithi082:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9a047fe7599a549042d06a3fb5b6813a203eabea shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 713bc57a-0991-11ef-bc97-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-05-03T21:15:33.991 INFO:teuthology.orchestra.run.smithi082.stderr:Inferring config /var/lib/ceph/713bc57a-0991-11ef-bc97-c7b262605968/mon.b/config 2024-05-03T21:15:35.222 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:15:35 smithi067 ceph-mon[29730]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 187 MiB used, 536 GiB / 536 GiB avail 2024-05-03T21:15:35.222 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:15:35 smithi067 ceph-mon[29730]: from='osd.6 [v2:172.21.15.67:6818/3269682731,v1:172.21.15.67:6819/3269682731]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-05-03T21:15:35.222 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:15:35 smithi067 ceph-mon[29730]: osdmap e43: 7 total, 6 up, 7 in 2024-05-03T21:15:35.222 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:15:35 smithi067 ceph-mon[29730]: from='osd.6 [v2:172.21.15.67:6818/3269682731,v1:172.21.15.67:6819/3269682731]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi067", "root=default"]}]: dispatch 2024-05-03T21:15:35.222 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:15:35 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-03T21:15:35.222 INFO:journalctl@ceph.osd.6.smithi067.stdout:May 03 21:15:35 smithi067 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-6[55180]: 2024-05-03T21:15:35.052+0000 7f3894da4700 -1 osd.6 0 waiting for initial osdmap 2024-05-03T21:15:35.222 INFO:journalctl@ceph.osd.6.smithi067.stdout:May 03 21:15:35 smithi067 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-6[55180]: 2024-05-03T21:15:35.064+0000 7f3890b97700 -1 osd.6 44 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-05-03T21:15:35.313 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:15:35 smithi082 ceph-mon[32933]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 187 MiB used, 536 GiB / 536 GiB avail 2024-05-03T21:15:35.313 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:15:35 smithi082 ceph-mon[32933]: from='osd.6 [v2:172.21.15.67:6818/3269682731,v1:172.21.15.67:6819/3269682731]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-05-03T21:15:35.314 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:15:35 smithi082 ceph-mon[32933]: osdmap e43: 7 total, 6 up, 7 in 2024-05-03T21:15:35.314 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:15:35 smithi082 ceph-mon[32933]: from='osd.6 [v2:172.21.15.67:6818/3269682731,v1:172.21.15.67:6819/3269682731]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi067", "root=default"]}]: dispatch 2024-05-03T21:15:35.314 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:15:35 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-03T21:15:35.324 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:15:35 smithi138 ceph-mon[33110]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 187 MiB used, 536 GiB / 536 GiB avail 2024-05-03T21:15:35.324 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:15:35 smithi138 ceph-mon[33110]: from='osd.6 [v2:172.21.15.67:6818/3269682731,v1:172.21.15.67:6819/3269682731]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-05-03T21:15:35.325 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:15:35 smithi138 ceph-mon[33110]: osdmap e43: 7 total, 6 up, 7 in 2024-05-03T21:15:35.325 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:15:35 smithi138 ceph-mon[33110]: from='osd.6 [v2:172.21.15.67:6818/3269682731,v1:172.21.15.67:6819/3269682731]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi067", "root=default"]}]: dispatch 2024-05-03T21:15:35.325 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:15:35 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-03T21:15:35.968 INFO:teuthology.orchestra.run.smithi082.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-05-03T21:15:35.968 INFO:teuthology.orchestra.run.smithi082.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-05-03T21:15:35.968 INFO:teuthology.orchestra.run.smithi082.stderr: stderr: 10+0 records in 2024-05-03T21:15:35.968 INFO:teuthology.orchestra.run.smithi082.stderr:10+0 records out 2024-05-03T21:15:35.968 INFO:teuthology.orchestra.run.smithi082.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0116321 s, 901 MB/s 2024-05-03T21:15:35.968 INFO:teuthology.orchestra.run.smithi082.stderr:--> Zapping successful for: 2024-05-03T21:15:36.245 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:15:36 smithi082 ceph-mon[32933]: from='osd.6 [v2:172.21.15.67:6818/3269682731,v1:172.21.15.67:6819/3269682731]' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi067", "root=default"]}]': finished 2024-05-03T21:15:36.245 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:15:36 smithi082 ceph-mon[32933]: osdmap e44: 7 total, 6 up, 7 in 2024-05-03T21:15:36.246 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:15:36 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-03T21:15:36.246 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:15:36 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-03T21:15:36.324 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:15:36 smithi138 ceph-mon[33110]: from='osd.6 [v2:172.21.15.67:6818/3269682731,v1:172.21.15.67:6819/3269682731]' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi067", "root=default"]}]': finished 2024-05-03T21:15:36.324 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:15:36 smithi138 ceph-mon[33110]: osdmap e44: 7 total, 6 up, 7 in 2024-05-03T21:15:36.324 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:15:36 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-03T21:15:36.324 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:15:36 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-03T21:15:36.471 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:15:36 smithi067 ceph-mon[29730]: from='osd.6 [v2:172.21.15.67:6818/3269682731,v1:172.21.15.67:6819/3269682731]' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi067", "root=default"]}]': finished 2024-05-03T21:15:36.471 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:15:36 smithi067 ceph-mon[29730]: osdmap e44: 7 total, 6 up, 7 in 2024-05-03T21:15:36.471 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:15:36 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-03T21:15:36.471 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:15:36 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-03T21:15:36.605 DEBUG:teuthology.orchestra.run.smithi082:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9a047fe7599a549042d06a3fb5b6813a203eabea shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 713bc57a-0991-11ef-bc97-c7b262605968 -- ceph orch daemon add osd smithi082:vg_nvme/lv_2 2024-05-03T21:15:36.849 INFO:teuthology.orchestra.run.smithi082.stderr:Inferring config /var/lib/ceph/713bc57a-0991-11ef-bc97-c7b262605968/mon.b/config 2024-05-03T21:15:37.221 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:15:37 smithi067 ceph-mon[29730]: purged_snaps scrub starts 2024-05-03T21:15:37.221 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:15:37 smithi067 ceph-mon[29730]: purged_snaps scrub ok 2024-05-03T21:15:37.222 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:15:37 smithi067 ceph-mon[29730]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 187 MiB used, 536 GiB / 536 GiB avail 2024-05-03T21:15:37.222 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:15:37 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-03T21:15:37.222 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:15:37 smithi067 ceph-mon[29730]: osd.6 [v2:172.21.15.67:6818/3269682731,v1:172.21.15.67:6819/3269682731] boot 2024-05-03T21:15:37.222 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:15:37 smithi067 ceph-mon[29730]: osdmap e45: 7 total, 7 up, 7 in 2024-05-03T21:15:37.222 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:15:37 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-03T21:15:37.222 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:15:37 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:15:37.222 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:15:37 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:15:37.222 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:15:37 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-05-03T21:15:37.222 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:15:37 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-05-03T21:15:37.222 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:15:37 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-05-03T21:15:37.222 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:15:37 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:15:37.222 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:15:37 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T21:15:37.222 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:15:37 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T21:15:37.223 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:15:37 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:15:37.235 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:15:37 smithi082 ceph-mon[32933]: purged_snaps scrub starts 2024-05-03T21:15:37.236 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:15:37 smithi082 ceph-mon[32933]: purged_snaps scrub ok 2024-05-03T21:15:37.236 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:15:37 smithi082 ceph-mon[32933]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 187 MiB used, 536 GiB / 536 GiB avail 2024-05-03T21:15:37.236 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:15:37 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-03T21:15:37.236 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:15:37 smithi082 ceph-mon[32933]: osd.6 [v2:172.21.15.67:6818/3269682731,v1:172.21.15.67:6819/3269682731] boot 2024-05-03T21:15:37.236 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:15:37 smithi082 ceph-mon[32933]: osdmap e45: 7 total, 7 up, 7 in 2024-05-03T21:15:37.236 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:15:37 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-03T21:15:37.236 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:15:37 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:15:37.236 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:15:37 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:15:37.236 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:15:37 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-05-03T21:15:37.236 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:15:37 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-05-03T21:15:37.236 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:15:37 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-05-03T21:15:37.236 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:15:37 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:15:37.237 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:15:37 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T21:15:37.237 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:15:37 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T21:15:37.237 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:15:37 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:15:37.325 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:15:37 smithi138 ceph-mon[33110]: purged_snaps scrub starts 2024-05-03T21:15:37.325 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:15:37 smithi138 ceph-mon[33110]: purged_snaps scrub ok 2024-05-03T21:15:37.325 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:15:37 smithi138 ceph-mon[33110]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 187 MiB used, 536 GiB / 536 GiB avail 2024-05-03T21:15:37.325 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:15:37 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-03T21:15:37.325 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:15:37 smithi138 ceph-mon[33110]: osd.6 [v2:172.21.15.67:6818/3269682731,v1:172.21.15.67:6819/3269682731] boot 2024-05-03T21:15:37.325 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:15:37 smithi138 ceph-mon[33110]: osdmap e45: 7 total, 7 up, 7 in 2024-05-03T21:15:37.325 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:15:37 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-03T21:15:37.325 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:15:37 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:15:37.325 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:15:37 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:15:37.325 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:15:37 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-05-03T21:15:37.325 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:15:37 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-05-03T21:15:37.325 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:15:37 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-05-03T21:15:37.326 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:15:37 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:15:37.326 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:15:37 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T21:15:37.326 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:15:37 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T21:15:37.326 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:15:37 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:15:38.294 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:15:38 smithi082 ceph-mon[32933]: Detected new or changed devices on smithi067 2024-05-03T21:15:38.294 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:15:38 smithi082 ceph-mon[32933]: Adjusting osd_memory_target on smithi067 to 5696M 2024-05-03T21:15:38.294 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:15:38 smithi082 ceph-mon[32933]: osdmap e46: 7 total, 7 up, 7 in 2024-05-03T21:15:38.334 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T21:15:38.331+0000 7f3da29a2700 1 -- 172.21.15.82:0/71941237 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f3d9c103640 msgr2=0x7f3d9c103aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:15:38.335 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T21:15:38.331+0000 7f3da29a2700 1 --2- 172.21.15.82:0/71941237 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f3d9c103640 0x7f3d9c103aa0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f3d900099c0 tx=0x7f3d90009cd0 comp rx=0 tx=0).stop 2024-05-03T21:15:38.335 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T21:15:38.331+0000 7f3da29a2700 1 -- 172.21.15.82:0/71941237 shutdown_connections 2024-05-03T21:15:38.335 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T21:15:38.331+0000 7f3da29a2700 1 --2- 172.21.15.82:0/71941237 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3d9c103fe0 0x7f3d9c10a890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:15:38.335 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T21:15:38.331+0000 7f3da29a2700 1 --2- 172.21.15.82:0/71941237 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f3d9c102440 0x7f3d9c102860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:15:38.335 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T21:15:38.331+0000 7f3da29a2700 1 --2- 172.21.15.82:0/71941237 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f3d9c103640 0x7f3d9c103aa0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:15:38.335 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T21:15:38.331+0000 7f3da29a2700 1 -- 172.21.15.82:0/71941237 >> 172.21.15.82:0/71941237 conn(0x7f3d9c0fda00 msgr2=0x7f3d9c0ffe20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T21:15:38.335 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T21:15:38.331+0000 7f3da29a2700 1 -- 172.21.15.82:0/71941237 shutdown_connections 2024-05-03T21:15:38.335 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T21:15:38.331+0000 7f3da29a2700 1 -- 172.21.15.82:0/71941237 wait complete. 2024-05-03T21:15:38.335 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T21:15:38.331+0000 7f3da29a2700 1 Processor -- start 2024-05-03T21:15:38.335 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T21:15:38.331+0000 7f3da29a2700 1 -- start start 2024-05-03T21:15:38.335 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T21:15:38.332+0000 7f3da29a2700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f3d9c102440 0x7f3d9c06c750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:15:38.335 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T21:15:38.332+0000 7f3da29a2700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3d9c103640 0x7f3d9c19e5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:15:38.336 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T21:15:38.332+0000 7f3da29a2700 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f3d9c103fe0 0x7f3d9c1a0c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:15:38.336 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T21:15:38.332+0000 7f3da29a2700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f3d9c10e170 con 0x7f3d9c103640 2024-05-03T21:15:38.336 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T21:15:38.332+0000 7f3da29a2700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f3d9c10dff0 con 0x7f3d9c102440 2024-05-03T21:15:38.336 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T21:15:38.332+0000 7f3da29a2700 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f3d9c10e2f0 con 0x7f3d9c103fe0 2024-05-03T21:15:38.336 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T21:15:38.332+0000 7f3d9bfff700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f3d9c102440 0x7f3d9c06c750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:15:38.336 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T21:15:38.332+0000 7f3d9bfff700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f3d9c102440 0x7f3d9c06c750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.82:44530/0 (socket says 172.21.15.82:44530) 2024-05-03T21:15:38.336 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T21:15:38.332+0000 7f3d9bfff700 1 -- 172.21.15.82:0/110130704 learned_addr learned my addr 172.21.15.82:0/110130704 (peer_addr_for_me v2:172.21.15.82:0/0) 2024-05-03T21:15:38.336 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T21:15:38.332+0000 7f3d9b7fe700 1 --2- 172.21.15.82:0/110130704 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3d9c103640 0x7f3d9c19e5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:15:38.336 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T21:15:38.332+0000 7f3da0f3f700 1 --2- 172.21.15.82:0/110130704 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f3d9c103fe0 0x7f3d9c1a0c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:15:38.336 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T21:15:38.332+0000 7f3d9bfff700 1 -- 172.21.15.82:0/110130704 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f3d9c103fe0 msgr2=0x7f3d9c1a0c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:15:38.336 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T21:15:38.332+0000 7f3d9bfff700 1 --2- 172.21.15.82:0/110130704 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f3d9c103fe0 0x7f3d9c1a0c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:15:38.336 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T21:15:38.332+0000 7f3d9bfff700 1 -- 172.21.15.82:0/110130704 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3d9c103640 msgr2=0x7f3d9c19e5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:15:38.336 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T21:15:38.332+0000 7f3d9bfff700 1 --2- 172.21.15.82:0/110130704 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3d9c103640 0x7f3d9c19e5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:15:38.336 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T21:15:38.332+0000 7f3d9bfff700 1 -- 172.21.15.82:0/110130704 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3d9c1a14d0 con 0x7f3d9c102440 2024-05-03T21:15:38.337 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T21:15:38.332+0000 7f3d9bfff700 1 --2- 172.21.15.82:0/110130704 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f3d9c102440 0x7f3d9c06c750 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f3d8c00c8f0 tx=0x7f3d8c00cc00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T21:15:38.337 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T21:15:38.332+0000 7f3d997fa700 1 -- 172.21.15.82:0/110130704 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d8c004220 con 0x7f3d9c102440 2024-05-03T21:15:38.337 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T21:15:38.333+0000 7f3da29a2700 1 -- 172.21.15.82:0/110130704 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3d9c1a17c0 con 0x7f3d9c102440 2024-05-03T21:15:38.337 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T21:15:38.333+0000 7f3d997fa700 1 -- 172.21.15.82:0/110130704 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(28 keys) ==== 1170+0+0 (secure 0 0 0) 0x7f3d8c0043c0 con 0x7f3d9c102440 2024-05-03T21:15:38.337 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T21:15:38.333+0000 7f3d997fa700 1 -- 172.21.15.82:0/110130704 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d8c003de0 con 0x7f3d9c102440 2024-05-03T21:15:38.337 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T21:15:38.333+0000 7f3da29a2700 1 -- 172.21.15.82:0/110130704 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3d9c1a1d00 con 0x7f3d9c102440 2024-05-03T21:15:38.338 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T21:15:38.333+0000 7f3da29a2700 1 -- 172.21.15.82:0/110130704 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3d9c06a190 con 0x7f3d9c102440 2024-05-03T21:15:38.338 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T21:15:38.335+0000 7f3d997fa700 1 -- 172.21.15.82:0/110130704 <== mon.2 v2:172.21.15.82:3300/0 4 ==== mgrmap(e 16) ==== 149652+0+0 (secure 0 0 0) 0x7f3d8c007740 con 0x7f3d9c102440 2024-05-03T21:15:38.338 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T21:15:38.335+0000 7f3d997fa700 1 --2- 172.21.15.82:0/110130704 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f3d840b2880 0x7f3d840b4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:15:38.338 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T21:15:38.335+0000 7f3d997fa700 1 -- 172.21.15.82:0/110130704 <== mon.2 v2:172.21.15.82:3300/0 5 ==== osd_map(46..46 src has 1..46) ==== 4965+0+0 (secure 0 0 0) 0x7f3d8c0e1520 con 0x7f3d9c102440 2024-05-03T21:15:38.338 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T21:15:38.336+0000 7f3d9b7fe700 1 --2- 172.21.15.82:0/110130704 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f3d840b2880 0x7f3d840b4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:15:38.338 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T21:15:38.336+0000 7f3d9b7fe700 1 --2- 172.21.15.82:0/110130704 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f3d840b2880 0x7f3d840b4d40 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f3d9c06ccd0 tx=0x7f3d90005fb0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T21:15:38.341 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T21:15:38.339+0000 7f3d997fa700 1 -- 172.21.15.82:0/110130704 <== mon.2 v2:172.21.15.82:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191839 (secure 0 0 0) 0x7f3d8c014030 con 0x7f3d9c102440 2024-05-03T21:15:38.471 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:15:38 smithi067 ceph-mon[29730]: Detected new or changed devices on smithi067 2024-05-03T21:15:38.472 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:15:38 smithi067 ceph-mon[29730]: Adjusting osd_memory_target on smithi067 to 5696M 2024-05-03T21:15:38.472 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:15:38 smithi067 ceph-mon[29730]: osdmap e46: 7 total, 7 up, 7 in 2024-05-03T21:15:38.553 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T21:15:38.552+0000 7f3da29a2700 1 -- 172.21.15.82:0/110130704 --> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi082:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) -- 0x7f3d9c0638e0 con 0x7f3d840b2880 2024-05-03T21:15:38.574 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:15:38 smithi138 ceph-mon[33110]: Detected new or changed devices on smithi067 2024-05-03T21:15:38.574 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:15:38 smithi138 ceph-mon[33110]: Adjusting osd_memory_target on smithi067 to 5696M 2024-05-03T21:15:38.574 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:15:38 smithi138 ceph-mon[33110]: osdmap e46: 7 total, 7 up, 7 in 2024-05-03T21:15:39.363 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:15:39 smithi082 ceph-mon[32933]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 213 MiB used, 626 GiB / 626 GiB avail 2024-05-03T21:15:39.363 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:15:39 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-03T21:15:39.363 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:15:39 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-03T21:15:39.363 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:15:39 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T21:15:39.471 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:15:39 smithi067 ceph-mon[29730]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 213 MiB used, 626 GiB / 626 GiB avail 2024-05-03T21:15:39.471 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:15:39 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-03T21:15:39.471 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:15:39 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-03T21:15:39.471 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:15:39 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T21:15:39.574 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:15:39 smithi138 ceph-mon[33110]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 213 MiB used, 626 GiB / 626 GiB avail 2024-05-03T21:15:39.574 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:15:39 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-03T21:15:39.574 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:15:39 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-03T21:15:39.575 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:15:39 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T21:15:40.245 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:15:40 smithi082 ceph-mon[32933]: from='client.24266 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi082:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-05-03T21:15:40.471 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:15:40 smithi067 ceph-mon[29730]: from='client.24266 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi082:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-05-03T21:15:40.574 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:15:40 smithi138 ceph-mon[33110]: from='client.24266 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi082:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-05-03T21:15:41.424 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:15:41 smithi082 ceph-mon[32933]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 214 MiB used, 626 GiB / 626 GiB avail 2024-05-03T21:15:41.424 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:15:41 smithi082 ceph-mon[32933]: from='client.? 172.21.15.82:0/1324439371' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8fa4cd64-ca6b-48ae-8db0-d369ce6102ac"}]: dispatch 2024-05-03T21:15:41.424 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:15:41 smithi082 ceph-mon[32933]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8fa4cd64-ca6b-48ae-8db0-d369ce6102ac"}]: dispatch 2024-05-03T21:15:41.424 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:15:41 smithi082 ceph-mon[32933]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "8fa4cd64-ca6b-48ae-8db0-d369ce6102ac"}]': finished 2024-05-03T21:15:41.424 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:15:41 smithi082 ceph-mon[32933]: osdmap e47: 8 total, 7 up, 8 in 2024-05-03T21:15:41.424 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:15:41 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-05-03T21:15:41.471 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:15:41 smithi067 ceph-mon[29730]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 214 MiB used, 626 GiB / 626 GiB avail 2024-05-03T21:15:41.471 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:15:41 smithi067 ceph-mon[29730]: from='client.? 172.21.15.82:0/1324439371' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8fa4cd64-ca6b-48ae-8db0-d369ce6102ac"}]: dispatch 2024-05-03T21:15:41.471 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:15:41 smithi067 ceph-mon[29730]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8fa4cd64-ca6b-48ae-8db0-d369ce6102ac"}]: dispatch 2024-05-03T21:15:41.471 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:15:41 smithi067 ceph-mon[29730]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "8fa4cd64-ca6b-48ae-8db0-d369ce6102ac"}]': finished 2024-05-03T21:15:41.471 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:15:41 smithi067 ceph-mon[29730]: osdmap e47: 8 total, 7 up, 8 in 2024-05-03T21:15:41.472 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:15:41 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-05-03T21:15:41.574 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:15:41 smithi138 ceph-mon[33110]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 214 MiB used, 626 GiB / 626 GiB avail 2024-05-03T21:15:41.574 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:15:41 smithi138 ceph-mon[33110]: from='client.? 172.21.15.82:0/1324439371' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8fa4cd64-ca6b-48ae-8db0-d369ce6102ac"}]: dispatch 2024-05-03T21:15:41.574 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:15:41 smithi138 ceph-mon[33110]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8fa4cd64-ca6b-48ae-8db0-d369ce6102ac"}]: dispatch 2024-05-03T21:15:41.575 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:15:41 smithi138 ceph-mon[33110]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "8fa4cd64-ca6b-48ae-8db0-d369ce6102ac"}]': finished 2024-05-03T21:15:41.575 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:15:41 smithi138 ceph-mon[33110]: osdmap e47: 8 total, 7 up, 8 in 2024-05-03T21:15:41.575 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:15:41 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-05-03T21:15:42.471 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:15:42 smithi067 ceph-mon[29730]: from='client.? 172.21.15.82:0/3071264754' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-03T21:15:42.495 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:15:42 smithi082 ceph-mon[32933]: from='client.? 172.21.15.82:0/3071264754' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-03T21:15:42.574 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:15:42 smithi138 ceph-mon[33110]: from='client.? 172.21.15.82:0/3071264754' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-03T21:15:43.471 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:15:43 smithi067 ceph-mon[29730]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 214 MiB used, 626 GiB / 626 GiB avail 2024-05-03T21:15:43.475 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:15:43 smithi138 ceph-mon[33110]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 214 MiB used, 626 GiB / 626 GiB avail 2024-05-03T21:15:43.495 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:15:43 smithi082 ceph-mon[32933]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 214 MiB used, 626 GiB / 626 GiB avail 2024-05-03T21:15:45.245 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:15:45 smithi082 ceph-mon[32933]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 214 MiB used, 626 GiB / 626 GiB avail 2024-05-03T21:15:45.471 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:15:45 smithi067 ceph-mon[29730]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 214 MiB used, 626 GiB / 626 GiB avail 2024-05-03T21:15:45.574 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:15:45 smithi138 ceph-mon[33110]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 214 MiB used, 626 GiB / 626 GiB avail 2024-05-03T21:15:47.295 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:15:47 smithi082 ceph-mon[32933]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 214 MiB used, 626 GiB / 626 GiB avail 2024-05-03T21:15:47.471 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:15:47 smithi067 ceph-mon[29730]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 214 MiB used, 626 GiB / 626 GiB avail 2024-05-03T21:15:47.574 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:15:47 smithi138 ceph-mon[33110]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 214 MiB used, 626 GiB / 626 GiB avail 2024-05-03T21:15:49.356 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:15:49 smithi082 ceph-mon[32933]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 214 MiB used, 626 GiB / 626 GiB avail 2024-05-03T21:15:49.357 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:15:49 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-05-03T21:15:49.357 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:15:49 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T21:15:49.471 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:15:49 smithi067 ceph-mon[29730]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 214 MiB used, 626 GiB / 626 GiB avail 2024-05-03T21:15:49.471 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:15:49 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-05-03T21:15:49.471 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:15:49 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T21:15:49.574 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:15:49 smithi138 ceph-mon[33110]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 214 MiB used, 626 GiB / 626 GiB avail 2024-05-03T21:15:49.575 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:15:49 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-05-03T21:15:49.575 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:15:49 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T21:15:50.245 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:15:50 smithi082 ceph-mon[32933]: Deploying daemon osd.7 on smithi082 2024-05-03T21:15:50.471 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:15:50 smithi067 ceph-mon[29730]: Deploying daemon osd.7 on smithi082 2024-05-03T21:15:50.574 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:15:50 smithi138 ceph-mon[33110]: Deploying daemon osd.7 on smithi082 2024-05-03T21:15:51.245 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:15:51 smithi082 ceph-mon[32933]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 214 MiB used, 626 GiB / 626 GiB avail 2024-05-03T21:15:51.471 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:15:51 smithi067 ceph-mon[29730]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 214 MiB used, 626 GiB / 626 GiB avail 2024-05-03T21:15:51.574 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:15:51 smithi138 ceph-mon[33110]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 214 MiB used, 626 GiB / 626 GiB avail 2024-05-03T21:15:53.245 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:15:53 smithi082 ceph-mon[32933]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 214 MiB used, 626 GiB / 626 GiB avail 2024-05-03T21:15:53.471 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:15:53 smithi067 ceph-mon[29730]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 214 MiB used, 626 GiB / 626 GiB avail 2024-05-03T21:15:53.574 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:15:53 smithi138 ceph-mon[33110]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 214 MiB used, 626 GiB / 626 GiB avail 2024-05-03T21:15:55.074 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:15:54 smithi138 ceph-mon[33110]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 214 MiB used, 626 GiB / 626 GiB avail 2024-05-03T21:15:55.074 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:15:54 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:15:55.075 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:15:54 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:15:55.075 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:15:54 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T21:15:55.125 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:15:54 smithi082 ceph-mon[32933]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 214 MiB used, 626 GiB / 626 GiB avail 2024-05-03T21:15:55.125 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:15:54 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:15:55.125 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:15:54 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:15:55.125 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:15:54 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T21:15:55.221 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:15:54 smithi067 ceph-mon[29730]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 214 MiB used, 626 GiB / 626 GiB avail 2024-05-03T21:15:55.221 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:15:54 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:15:55.221 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:15:54 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:15:55.221 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:15:54 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T21:15:57.026 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:15:56 smithi082 ceph-mon[32933]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 214 MiB used, 626 GiB / 626 GiB avail 2024-05-03T21:15:57.067 INFO:teuthology.orchestra.run.smithi082.stdout:Created osd(s) 7 on host 'smithi082' 2024-05-03T21:15:57.067 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T21:15:57.064+0000 7f3d997fa700 1 -- 172.21.15.82:0/110130704 <== mgr.14150 v2:172.21.15.67:6800/604451592 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f3d9c0638e0 con 0x7f3d840b2880 2024-05-03T21:15:57.070 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T21:15:57.067+0000 7f3da29a2700 1 -- 172.21.15.82:0/110130704 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f3d840b2880 msgr2=0x7f3d840b4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:15:57.070 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T21:15:57.067+0000 7f3da29a2700 1 --2- 172.21.15.82:0/110130704 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f3d840b2880 0x7f3d840b4d40 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f3d9c06ccd0 tx=0x7f3d90005fb0 comp rx=0 tx=0).stop 2024-05-03T21:15:57.071 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T21:15:57.067+0000 7f3da29a2700 1 -- 172.21.15.82:0/110130704 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f3d9c102440 msgr2=0x7f3d9c06c750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:15:57.071 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T21:15:57.067+0000 7f3da29a2700 1 --2- 172.21.15.82:0/110130704 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f3d9c102440 0x7f3d9c06c750 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f3d8c00c8f0 tx=0x7f3d8c00cc00 comp rx=0 tx=0).stop 2024-05-03T21:15:57.071 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T21:15:57.068+0000 7f3da29a2700 1 -- 172.21.15.82:0/110130704 shutdown_connections 2024-05-03T21:15:57.071 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T21:15:57.068+0000 7f3da29a2700 1 --2- 172.21.15.82:0/110130704 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f3d840b2880 0x7f3d840b4d40 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:15:57.071 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T21:15:57.068+0000 7f3da29a2700 1 --2- 172.21.15.82:0/110130704 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f3d9c103fe0 0x7f3d9c1a0c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:15:57.071 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T21:15:57.068+0000 7f3da29a2700 1 --2- 172.21.15.82:0/110130704 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f3d9c102440 0x7f3d9c06c750 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:15:57.071 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T21:15:57.068+0000 7f3da29a2700 1 --2- 172.21.15.82:0/110130704 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3d9c103640 0x7f3d9c19e5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:15:57.071 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T21:15:57.068+0000 7f3da29a2700 1 -- 172.21.15.82:0/110130704 >> 172.21.15.82:0/110130704 conn(0x7f3d9c0fda00 msgr2=0x7f3d9c106940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T21:15:57.071 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T21:15:57.068+0000 7f3da29a2700 1 -- 172.21.15.82:0/110130704 shutdown_connections 2024-05-03T21:15:57.071 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T21:15:57.068+0000 7f3da29a2700 1 -- 172.21.15.82:0/110130704 wait complete. 2024-05-03T21:15:57.074 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:15:56 smithi138 ceph-mon[33110]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 214 MiB used, 626 GiB / 626 GiB avail 2024-05-03T21:15:57.221 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:15:56 smithi067 ceph-mon[29730]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 214 MiB used, 626 GiB / 626 GiB avail 2024-05-03T21:15:57.666 DEBUG:teuthology.orchestra.run.smithi082:osd.7> sudo journalctl -f -n 0 -u ceph-713bc57a-0991-11ef-bc97-c7b262605968@osd.7.service 2024-05-03T21:15:57.668 INFO:tasks.cephadm:Deploying osd.8 on smithi138 with /dev/vg_nvme/lv_2... 2024-05-03T21:15:57.668 DEBUG:teuthology.orchestra.run.smithi138:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9a047fe7599a549042d06a3fb5b6813a203eabea shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 713bc57a-0991-11ef-bc97-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-05-03T21:15:57.933 INFO:teuthology.orchestra.run.smithi138.stderr:Inferring config /var/lib/ceph/713bc57a-0991-11ef-bc97-c7b262605968/mon.c/config 2024-05-03T21:15:57.995 INFO:journalctl@ceph.osd.7.smithi082.stdout:May 03 21:15:57 smithi082 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-7[50761]: 2024-05-03T21:15:57.719+0000 7f44c285b7c0 -1 osd.7 0 log_to_monitors true 2024-05-03T21:15:58.324 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:15:58 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:15:58.325 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:15:58 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:15:58.325 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:15:58 smithi138 ceph-mon[33110]: from='osd.7 [v2:172.21.15.82:6816/3008589976,v1:172.21.15.82:6817/3008589976]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-05-03T21:15:58.325 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:15:58 smithi138 ceph-mon[33110]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-05-03T21:15:58.325 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:15:58 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:15:58.325 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:15:58 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:15:58.471 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:15:58 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:15:58.471 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:15:58 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:15:58.471 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:15:58 smithi067 ceph-mon[29730]: from='osd.7 [v2:172.21.15.82:6816/3008589976,v1:172.21.15.82:6817/3008589976]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-05-03T21:15:58.471 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:15:58 smithi067 ceph-mon[29730]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-05-03T21:15:58.471 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:15:58 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:15:58.471 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:15:58 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:15:58.472 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:15:58 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:15:58.472 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:15:58 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:15:58.472 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:15:58 smithi082 ceph-mon[32933]: from='osd.7 [v2:172.21.15.82:6816/3008589976,v1:172.21.15.82:6817/3008589976]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-05-03T21:15:58.472 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:15:58 smithi082 ceph-mon[32933]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-05-03T21:15:58.472 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:15:58 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:15:58.472 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:15:58 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:15:59.231 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:15:59 smithi138 ceph-mon[33110]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 214 MiB used, 626 GiB / 626 GiB avail 2024-05-03T21:15:59.231 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:15:59 smithi138 ceph-mon[33110]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-05-03T21:15:59.231 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:15:59 smithi138 ceph-mon[33110]: from='osd.7 [v2:172.21.15.82:6816/3008589976,v1:172.21.15.82:6817/3008589976]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi082", "root=default"]}]: dispatch 2024-05-03T21:15:59.231 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:15:59 smithi138 ceph-mon[33110]: osdmap e48: 8 total, 7 up, 8 in 2024-05-03T21:15:59.231 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:15:59 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-05-03T21:15:59.231 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:15:59 smithi138 ceph-mon[33110]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi082", "root=default"]}]: dispatch 2024-05-03T21:15:59.379 INFO:journalctl@ceph.osd.7.smithi082.stdout:May 03 21:15:59 smithi082 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-7[50761]: 2024-05-03T21:15:59.080+0000 7f44b7ebf700 -1 osd.7 0 waiting for initial osdmap 2024-05-03T21:15:59.379 INFO:journalctl@ceph.osd.7.smithi082.stdout:May 03 21:15:59 smithi082 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-7[50761]: 2024-05-03T21:15:59.092+0000 7f44b3cb2700 -1 osd.7 49 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-05-03T21:15:59.379 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:15:59 smithi082 ceph-mon[32933]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 214 MiB used, 626 GiB / 626 GiB avail 2024-05-03T21:15:59.379 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:15:59 smithi082 ceph-mon[32933]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-05-03T21:15:59.379 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:15:59 smithi082 ceph-mon[32933]: from='osd.7 [v2:172.21.15.82:6816/3008589976,v1:172.21.15.82:6817/3008589976]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi082", "root=default"]}]: dispatch 2024-05-03T21:15:59.379 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:15:59 smithi082 ceph-mon[32933]: osdmap e48: 8 total, 7 up, 8 in 2024-05-03T21:15:59.379 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:15:59 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-05-03T21:15:59.379 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:15:59 smithi082 ceph-mon[32933]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi082", "root=default"]}]: dispatch 2024-05-03T21:15:59.471 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:15:59 smithi067 ceph-mon[29730]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 214 MiB used, 626 GiB / 626 GiB avail 2024-05-03T21:15:59.471 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:15:59 smithi067 ceph-mon[29730]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-05-03T21:15:59.471 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:15:59 smithi067 ceph-mon[29730]: from='osd.7 [v2:172.21.15.82:6816/3008589976,v1:172.21.15.82:6817/3008589976]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi082", "root=default"]}]: dispatch 2024-05-03T21:15:59.471 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:15:59 smithi067 ceph-mon[29730]: osdmap e48: 8 total, 7 up, 8 in 2024-05-03T21:15:59.471 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:15:59 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-05-03T21:15:59.471 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:15:59 smithi067 ceph-mon[29730]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi082", "root=default"]}]: dispatch 2024-05-03T21:15:59.782 INFO:teuthology.orchestra.run.smithi138.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-05-03T21:15:59.782 INFO:teuthology.orchestra.run.smithi138.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-05-03T21:15:59.783 INFO:teuthology.orchestra.run.smithi138.stderr: stderr: 10+0 records in 2024-05-03T21:15:59.783 INFO:teuthology.orchestra.run.smithi138.stderr:10+0 records out 2024-05-03T21:15:59.783 INFO:teuthology.orchestra.run.smithi138.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0111252 s, 943 MB/s 2024-05-03T21:15:59.783 INFO:teuthology.orchestra.run.smithi138.stderr:--> Zapping successful for: 2024-05-03T21:16:00.372 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:16:00 smithi138 ceph-mon[33110]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi082", "root=default"]}]': finished 2024-05-03T21:16:00.373 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:16:00 smithi138 ceph-mon[33110]: osdmap e49: 8 total, 7 up, 8 in 2024-05-03T21:16:00.373 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:16:00 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-05-03T21:16:00.373 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:16:00 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-05-03T21:16:00.401 DEBUG:teuthology.orchestra.run.smithi138:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9a047fe7599a549042d06a3fb5b6813a203eabea shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 713bc57a-0991-11ef-bc97-c7b262605968 -- ceph orch daemon add osd smithi138:vg_nvme/lv_2 2024-05-03T21:16:00.471 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:16:00 smithi067 ceph-mon[29730]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi082", "root=default"]}]': finished 2024-05-03T21:16:00.471 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:16:00 smithi067 ceph-mon[29730]: osdmap e49: 8 total, 7 up, 8 in 2024-05-03T21:16:00.471 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:16:00 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-05-03T21:16:00.471 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:16:00 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-05-03T21:16:00.495 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:16:00 smithi082 ceph-mon[32933]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi082", "root=default"]}]': finished 2024-05-03T21:16:00.495 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:16:00 smithi082 ceph-mon[32933]: osdmap e49: 8 total, 7 up, 8 in 2024-05-03T21:16:00.496 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:16:00 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-05-03T21:16:00.496 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:16:00 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-05-03T21:16:00.648 INFO:teuthology.orchestra.run.smithi138.stderr:Inferring config /var/lib/ceph/713bc57a-0991-11ef-bc97-c7b262605968/mon.c/config 2024-05-03T21:16:01.245 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:16:01 smithi082 ceph-mon[32933]: purged_snaps scrub starts 2024-05-03T21:16:01.245 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:16:01 smithi082 ceph-mon[32933]: purged_snaps scrub ok 2024-05-03T21:16:01.246 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:16:01 smithi082 ceph-mon[32933]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 214 MiB used, 626 GiB / 626 GiB avail 2024-05-03T21:16:01.246 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:16:01 smithi082 ceph-mon[32933]: osd.7 [v2:172.21.15.82:6816/3008589976,v1:172.21.15.82:6817/3008589976] boot 2024-05-03T21:16:01.246 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:16:01 smithi082 ceph-mon[32933]: osdmap e50: 8 total, 8 up, 8 in 2024-05-03T21:16:01.246 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:16:01 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-05-03T21:16:01.246 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:16:01 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:16:01.246 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:16:01 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:16:01.246 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:16:01 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-05-03T21:16:01.246 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:16:01 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-05-03T21:16:01.246 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:16:01 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-05-03T21:16:01.246 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:16:01 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:16:01.246 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:16:01 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T21:16:01.246 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:16:01 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T21:16:01.247 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:16:01 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:16:01.247 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:16:01 smithi082 ceph-mon[32933]: osdmap e51: 8 total, 8 up, 8 in 2024-05-03T21:16:01.324 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:16:01 smithi138 ceph-mon[33110]: purged_snaps scrub starts 2024-05-03T21:16:01.324 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:16:01 smithi138 ceph-mon[33110]: purged_snaps scrub ok 2024-05-03T21:16:01.324 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:16:01 smithi138 ceph-mon[33110]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 214 MiB used, 626 GiB / 626 GiB avail 2024-05-03T21:16:01.325 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:16:01 smithi138 ceph-mon[33110]: osd.7 [v2:172.21.15.82:6816/3008589976,v1:172.21.15.82:6817/3008589976] boot 2024-05-03T21:16:01.325 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:16:01 smithi138 ceph-mon[33110]: osdmap e50: 8 total, 8 up, 8 in 2024-05-03T21:16:01.325 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:16:01 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-05-03T21:16:01.325 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:16:01 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:16:01.325 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:16:01 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:16:01.325 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:16:01 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-05-03T21:16:01.325 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:16:01 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-05-03T21:16:01.325 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:16:01 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-05-03T21:16:01.325 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:16:01 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:16:01.325 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:16:01 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T21:16:01.325 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:16:01 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T21:16:01.325 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:16:01 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:16:01.325 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:16:01 smithi138 ceph-mon[33110]: osdmap e51: 8 total, 8 up, 8 in 2024-05-03T21:16:01.471 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:16:01 smithi067 ceph-mon[29730]: purged_snaps scrub starts 2024-05-03T21:16:01.471 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:16:01 smithi067 ceph-mon[29730]: purged_snaps scrub ok 2024-05-03T21:16:01.471 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:16:01 smithi067 ceph-mon[29730]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 214 MiB used, 626 GiB / 626 GiB avail 2024-05-03T21:16:01.471 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:16:01 smithi067 ceph-mon[29730]: osd.7 [v2:172.21.15.82:6816/3008589976,v1:172.21.15.82:6817/3008589976] boot 2024-05-03T21:16:01.471 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:16:01 smithi067 ceph-mon[29730]: osdmap e50: 8 total, 8 up, 8 in 2024-05-03T21:16:01.471 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:16:01 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-05-03T21:16:01.471 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:16:01 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:16:01.472 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:16:01 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:16:01.472 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:16:01 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-05-03T21:16:01.472 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:16:01 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-05-03T21:16:01.472 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:16:01 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-05-03T21:16:01.472 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:16:01 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:16:01.472 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:16:01 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T21:16:01.472 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:16:01 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T21:16:01.472 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:16:01 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:16:01.472 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:16:01 smithi067 ceph-mon[29730]: osdmap e51: 8 total, 8 up, 8 in 2024-05-03T21:16:02.258 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:16:02 smithi138 ceph-mon[33110]: Detected new or changed devices on smithi082 2024-05-03T21:16:02.258 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:16:02 smithi138 ceph-mon[33110]: Adjusting osd_memory_target on smithi082 to 5696M 2024-05-03T21:16:02.258 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:16:02 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T21:16:02.258 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:16:02 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T21:16:02.258 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:16:02 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T21:16:02.258 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:16:02 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:16:02.258 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:16:02 smithi138 ceph-mon[33110]: osdmap e52: 8 total, 8 up, 8 in 2024-05-03T21:16:02.286 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:16:02.283+0000 7f89a8a5c700 1 -- 172.21.15.138:0/255804273 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f89a4103f90 msgr2=0x7f89a410a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:16:02.287 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:16:02.283+0000 7f89a8a5c700 1 --2- 172.21.15.138:0/255804273 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f89a4103f90 0x7f89a410a840 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f899c00b0a0 tx=0x7f899c00b3b0 comp rx=0 tx=0).stop 2024-05-03T21:16:02.287 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:16:02.284+0000 7f89a8a5c700 1 -- 172.21.15.138:0/255804273 shutdown_connections 2024-05-03T21:16:02.287 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:16:02.284+0000 7f89a8a5c700 1 --2- 172.21.15.138:0/255804273 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f89a41023f0 0x7f89a4102810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:16:02.287 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:16:02.284+0000 7f89a8a5c700 1 --2- 172.21.15.138:0/255804273 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f89a4103f90 0x7f89a410a840 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:16:02.287 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:16:02.284+0000 7f89a8a5c700 1 --2- 172.21.15.138:0/255804273 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f89a41035f0 0x7f89a4103a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:16:02.287 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:16:02.284+0000 7f89a8a5c700 1 -- 172.21.15.138:0/255804273 >> 172.21.15.138:0/255804273 conn(0x7f89a40fd9d0 msgr2=0x7f89a40ffdf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T21:16:02.287 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:16:02.284+0000 7f89a8a5c700 1 -- 172.21.15.138:0/255804273 shutdown_connections 2024-05-03T21:16:02.287 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:16:02.284+0000 7f89a8a5c700 1 -- 172.21.15.138:0/255804273 wait complete. 2024-05-03T21:16:02.289 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:16:02.285+0000 7f89a8a5c700 1 Processor -- start 2024-05-03T21:16:02.289 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:16:02.285+0000 7f89a8a5c700 1 -- start start 2024-05-03T21:16:02.289 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:16:02.285+0000 7f89a8a5c700 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f89a41023f0 0x7f89a41986e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:16:02.289 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:16:02.285+0000 7f89a8a5c700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f89a41035f0 0x7f89a4198c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:16:02.290 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:16:02.285+0000 7f89a8a5c700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f89a4103f90 0x7f89a419fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:16:02.290 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:16:02.285+0000 7f89a8a5c700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f89a410e1b0 con 0x7f89a41035f0 2024-05-03T21:16:02.290 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:16:02.285+0000 7f89a8a5c700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f89a410e030 con 0x7f89a4103f90 2024-05-03T21:16:02.290 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:16:02.285+0000 7f89a8a5c700 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f89a410e330 con 0x7f89a41023f0 2024-05-03T21:16:02.290 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:16:02.285+0000 7f89a259c700 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f89a41023f0 0x7f89a41986e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:16:02.290 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:16:02.285+0000 7f89a259c700 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f89a41023f0 0x7f89a41986e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3300/0 says I am v2:172.21.15.138:59812/0 (socket says 172.21.15.138:59812) 2024-05-03T21:16:02.290 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:16:02.285+0000 7f89a259c700 1 -- 172.21.15.138:0/2927934871 learned_addr learned my addr 172.21.15.138:0/2927934871 (peer_addr_for_me v2:172.21.15.138:0/0) 2024-05-03T21:16:02.290 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:16:02.285+0000 7f89a2d9d700 1 --2- 172.21.15.138:0/2927934871 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f89a4103f90 0x7f89a419fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:16:02.290 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:16:02.285+0000 7f89a1d9b700 1 --2- 172.21.15.138:0/2927934871 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f89a41035f0 0x7f89a4198c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:16:02.290 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:16:02.285+0000 7f89a259c700 1 -- 172.21.15.138:0/2927934871 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f89a4103f90 msgr2=0x7f89a419fca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:16:02.290 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:16:02.285+0000 7f89a259c700 1 --2- 172.21.15.138:0/2927934871 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f89a4103f90 0x7f89a419fca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:16:02.290 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:16:02.285+0000 7f89a259c700 1 -- 172.21.15.138:0/2927934871 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f89a41035f0 msgr2=0x7f89a4198c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:16:02.291 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:16:02.285+0000 7f89a259c700 1 --2- 172.21.15.138:0/2927934871 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f89a41035f0 0x7f89a4198c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:16:02.291 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:16:02.285+0000 7f89a259c700 1 -- 172.21.15.138:0/2927934871 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f89a41a0340 con 0x7f89a41023f0 2024-05-03T21:16:02.291 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:16:02.285+0000 7f89a259c700 1 --2- 172.21.15.138:0/2927934871 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f89a41023f0 0x7f89a41986e0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f899400b530 tx=0x7f899400b840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T21:16:02.291 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:16:02.286+0000 7f89937fe700 1 -- 172.21.15.138:0/2927934871 <== mon.1 v2:172.21.15.138:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8994015020 con 0x7f89a41023f0 2024-05-03T21:16:02.291 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:16:02.286+0000 7f89937fe700 1 -- 172.21.15.138:0/2927934871 <== mon.1 v2:172.21.15.138:3300/0 2 ==== config(28 keys) ==== 1170+0+0 (secure 0 0 0) 0x7f8994011d30 con 0x7f89a41023f0 2024-05-03T21:16:02.291 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:16:02.286+0000 7f89a8a5c700 1 -- 172.21.15.138:0/2927934871 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f89a41a0630 con 0x7f89a41023f0 2024-05-03T21:16:02.291 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:16:02.286+0000 7f89937fe700 1 -- 172.21.15.138:0/2927934871 <== mon.1 v2:172.21.15.138:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f899400f2e0 con 0x7f89a41023f0 2024-05-03T21:16:02.291 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:16:02.286+0000 7f89a8a5c700 1 -- 172.21.15.138:0/2927934871 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f89a41a0bf0 con 0x7f89a41023f0 2024-05-03T21:16:02.291 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:16:02.287+0000 7f89a8a5c700 1 -- 172.21.15.138:0/2927934871 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f89a406a190 con 0x7f89a41023f0 2024-05-03T21:16:02.292 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:16:02.288+0000 7f89937fe700 1 -- 172.21.15.138:0/2927934871 <== mon.1 v2:172.21.15.138:3300/0 4 ==== mgrmap(e 16) ==== 149652+0+0 (secure 0 0 0) 0x7f8994018020 con 0x7f89a41023f0 2024-05-03T21:16:02.292 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:16:02.288+0000 7f89937fe700 1 --2- 172.21.15.138:0/2927934871 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f898c0b2880 0x7f898c0b4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:16:02.292 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:16:02.289+0000 7f89937fe700 1 -- 172.21.15.138:0/2927934871 <== mon.1 v2:172.21.15.138:3300/0 5 ==== osd_map(52..52 src has 1..52) ==== 5397+0+0 (secure 0 0 0) 0x7f89940e0e00 con 0x7f89a41023f0 2024-05-03T21:16:02.292 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:16:02.289+0000 7f89a1d9b700 1 --2- 172.21.15.138:0/2927934871 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f898c0b2880 0x7f898c0b4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:16:02.292 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:16:02.289+0000 7f89a1d9b700 1 --2- 172.21.15.138:0/2927934871 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f898c0b2880 0x7f898c0b4d40 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f8998005950 tx=0x7f89980058e0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T21:16:02.296 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:16:02.293+0000 7f89937fe700 1 -- 172.21.15.138:0/2927934871 <== mon.1 v2:172.21.15.138:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191839 (secure 0 0 0) 0x7f899401f020 con 0x7f89a41023f0 2024-05-03T21:16:02.471 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:16:02 smithi067 ceph-mon[29730]: Detected new or changed devices on smithi082 2024-05-03T21:16:02.471 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:16:02 smithi067 ceph-mon[29730]: Adjusting osd_memory_target on smithi082 to 5696M 2024-05-03T21:16:02.471 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:16:02 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T21:16:02.471 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:16:02 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T21:16:02.471 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:16:02 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T21:16:02.471 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:16:02 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:16:02.471 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:16:02 smithi067 ceph-mon[29730]: osdmap e52: 8 total, 8 up, 8 in 2024-05-03T21:16:02.495 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:16:02 smithi082 ceph-mon[32933]: Detected new or changed devices on smithi082 2024-05-03T21:16:02.495 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:16:02 smithi082 ceph-mon[32933]: Adjusting osd_memory_target on smithi082 to 5696M 2024-05-03T21:16:02.496 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:16:02 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T21:16:02.496 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:16:02 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T21:16:02.496 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:16:02 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T21:16:02.496 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:16:02 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:16:02.496 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:16:02 smithi082 ceph-mon[32933]: osdmap e52: 8 total, 8 up, 8 in 2024-05-03T21:16:02.508 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:16:02.505+0000 7f89a8a5c700 1 -- 172.21.15.138:0/2927934871 --> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi138:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) -- 0x7f89a40638e0 con 0x7f898c0b2880 2024-05-03T21:16:03.411 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:16:03 smithi138 ceph-mon[33110]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 241 MiB used, 715 GiB / 715 GiB avail 2024-05-03T21:16:03.411 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:16:03 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-03T21:16:03.411 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:16:03 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-03T21:16:03.411 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:16:03 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T21:16:03.471 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:16:03 smithi067 ceph-mon[29730]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 241 MiB used, 715 GiB / 715 GiB avail 2024-05-03T21:16:03.471 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:16:03 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-03T21:16:03.471 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:16:03 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-03T21:16:03.471 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:16:03 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T21:16:03.495 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:16:03 smithi082 ceph-mon[32933]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 241 MiB used, 715 GiB / 715 GiB avail 2024-05-03T21:16:03.495 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:16:03 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-03T21:16:03.495 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:16:03 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-03T21:16:03.495 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:16:03 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T21:16:04.301 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:16:04 smithi138 ceph-mon[33110]: from='client.24265 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi138:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-05-03T21:16:04.471 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:16:04 smithi067 ceph-mon[29730]: from='client.24265 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi138:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-05-03T21:16:04.495 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:16:04 smithi082 ceph-mon[32933]: from='client.24265 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi138:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-05-03T21:16:05.471 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:16:05 smithi067 ceph-mon[29730]: pgmap v177: 1 pgs: 1 active+clean; 577 KiB data, 241 MiB used, 715 GiB / 715 GiB avail 2024-05-03T21:16:05.471 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:16:05 smithi067 ceph-mon[29730]: from='client.? 172.21.15.138:0/529428037' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "fae39087-6c38-4771-a2b9-3f4fecb08edd"}]: dispatch 2024-05-03T21:16:05.471 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:16:05 smithi067 ceph-mon[29730]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "fae39087-6c38-4771-a2b9-3f4fecb08edd"}]: dispatch 2024-05-03T21:16:05.471 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:16:05 smithi067 ceph-mon[29730]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "fae39087-6c38-4771-a2b9-3f4fecb08edd"}]': finished 2024-05-03T21:16:05.471 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:16:05 smithi067 ceph-mon[29730]: osdmap e53: 9 total, 8 up, 9 in 2024-05-03T21:16:05.472 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:16:05 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-05-03T21:16:05.491 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:16:05 smithi138 ceph-mon[33110]: pgmap v177: 1 pgs: 1 active+clean; 577 KiB data, 241 MiB used, 715 GiB / 715 GiB avail 2024-05-03T21:16:05.491 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:16:05 smithi138 ceph-mon[33110]: from='client.? 172.21.15.138:0/529428037' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "fae39087-6c38-4771-a2b9-3f4fecb08edd"}]: dispatch 2024-05-03T21:16:05.491 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:16:05 smithi138 ceph-mon[33110]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "fae39087-6c38-4771-a2b9-3f4fecb08edd"}]: dispatch 2024-05-03T21:16:05.491 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:16:05 smithi138 ceph-mon[33110]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "fae39087-6c38-4771-a2b9-3f4fecb08edd"}]': finished 2024-05-03T21:16:05.491 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:16:05 smithi138 ceph-mon[33110]: osdmap e53: 9 total, 8 up, 9 in 2024-05-03T21:16:05.492 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:16:05 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-05-03T21:16:05.495 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:16:05 smithi082 ceph-mon[32933]: pgmap v177: 1 pgs: 1 active+clean; 577 KiB data, 241 MiB used, 715 GiB / 715 GiB avail 2024-05-03T21:16:05.495 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:16:05 smithi082 ceph-mon[32933]: from='client.? 172.21.15.138:0/529428037' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "fae39087-6c38-4771-a2b9-3f4fecb08edd"}]: dispatch 2024-05-03T21:16:05.495 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:16:05 smithi082 ceph-mon[32933]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "fae39087-6c38-4771-a2b9-3f4fecb08edd"}]: dispatch 2024-05-03T21:16:05.495 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:16:05 smithi082 ceph-mon[32933]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "fae39087-6c38-4771-a2b9-3f4fecb08edd"}]': finished 2024-05-03T21:16:05.495 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:16:05 smithi082 ceph-mon[32933]: osdmap e53: 9 total, 8 up, 9 in 2024-05-03T21:16:05.495 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:16:05 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-05-03T21:16:06.471 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:16:06 smithi067 ceph-mon[29730]: from='client.? 172.21.15.138:0/3247866547' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-03T21:16:06.495 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:16:06 smithi082 ceph-mon[32933]: from='client.? 172.21.15.138:0/3247866547' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-03T21:16:06.574 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:16:06 smithi138 ceph-mon[33110]: from='client.? 172.21.15.138:0/3247866547' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-03T21:16:07.474 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:16:07 smithi067 ceph-mon[29730]: pgmap v179: 1 pgs: 1 active+clean; 577 KiB data, 241 MiB used, 715 GiB / 715 GiB avail; 101 KiB/s, 0 objects/s recovering 2024-05-03T21:16:07.495 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:16:07 smithi082 ceph-mon[32933]: pgmap v179: 1 pgs: 1 active+clean; 577 KiB data, 241 MiB used, 715 GiB / 715 GiB avail; 101 KiB/s, 0 objects/s recovering 2024-05-03T21:16:07.574 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:16:07 smithi138 ceph-mon[33110]: pgmap v179: 1 pgs: 1 active+clean; 577 KiB data, 241 MiB used, 715 GiB / 715 GiB avail; 101 KiB/s, 0 objects/s recovering 2024-05-03T21:16:09.324 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:16:09 smithi138 ceph-mon[33110]: pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 241 MiB used, 715 GiB / 715 GiB avail; 86 KiB/s, 0 objects/s recovering 2024-05-03T21:16:09.471 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:16:09 smithi067 ceph-mon[29730]: pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 241 MiB used, 715 GiB / 715 GiB avail; 86 KiB/s, 0 objects/s recovering 2024-05-03T21:16:09.495 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:16:09 smithi082 ceph-mon[32933]: pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 241 MiB used, 715 GiB / 715 GiB avail; 86 KiB/s, 0 objects/s recovering 2024-05-03T21:16:11.421 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:16:11 smithi138 ceph-mon[33110]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 241 MiB used, 715 GiB / 715 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-05-03T21:16:11.471 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:16:11 smithi067 ceph-mon[29730]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 241 MiB used, 715 GiB / 715 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-05-03T21:16:11.495 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:16:11 smithi082 ceph-mon[32933]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 241 MiB used, 715 GiB / 715 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-05-03T21:16:13.400 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:16:13 smithi138 ceph-mon[33110]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 241 MiB used, 715 GiB / 715 GiB avail; 59 KiB/s, 0 objects/s recovering 2024-05-03T21:16:13.400 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:16:13 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-05-03T21:16:13.400 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:16:13 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T21:16:13.471 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:16:13 smithi067 ceph-mon[29730]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 241 MiB used, 715 GiB / 715 GiB avail; 59 KiB/s, 0 objects/s recovering 2024-05-03T21:16:13.471 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:16:13 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-05-03T21:16:13.471 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:16:13 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T21:16:13.495 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:16:13 smithi082 ceph-mon[32933]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 241 MiB used, 715 GiB / 715 GiB avail; 59 KiB/s, 0 objects/s recovering 2024-05-03T21:16:13.495 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:16:13 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-05-03T21:16:13.495 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:16:13 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T21:16:14.471 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:16:14 smithi067 ceph-mon[29730]: Deploying daemon osd.8 on smithi138 2024-05-03T21:16:14.481 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:16:14 smithi138 ceph-mon[33110]: Deploying daemon osd.8 on smithi138 2024-05-03T21:16:14.495 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:16:14 smithi082 ceph-mon[32933]: Deploying daemon osd.8 on smithi138 2024-05-03T21:16:15.471 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:16:15 smithi067 ceph-mon[29730]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 241 MiB used, 715 GiB / 715 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-05-03T21:16:15.488 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:16:15 smithi138 ceph-mon[33110]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 241 MiB used, 715 GiB / 715 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-05-03T21:16:15.495 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:16:15 smithi082 ceph-mon[32933]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 241 MiB used, 715 GiB / 715 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-05-03T21:16:17.303 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:16:17 smithi138 ceph-mon[33110]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 241 MiB used, 715 GiB / 715 GiB avail 2024-05-03T21:16:17.495 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:16:17 smithi082 ceph-mon[32933]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 241 MiB used, 715 GiB / 715 GiB avail 2024-05-03T21:16:17.721 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:16:17 smithi067 ceph-mon[29730]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 241 MiB used, 715 GiB / 715 GiB avail 2024-05-03T21:16:19.221 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:16:18 smithi067 ceph-mon[29730]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 241 MiB used, 715 GiB / 715 GiB avail 2024-05-03T21:16:19.221 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:16:18 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:16:19.221 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:16:18 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T21:16:19.221 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:16:18 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:16:19.245 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:16:18 smithi082 ceph-mon[32933]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 241 MiB used, 715 GiB / 715 GiB avail 2024-05-03T21:16:19.245 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:16:18 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:16:19.246 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:16:18 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T21:16:19.246 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:16:18 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:16:19.324 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:16:18 smithi138 ceph-mon[33110]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 241 MiB used, 715 GiB / 715 GiB avail 2024-05-03T21:16:19.325 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:16:18 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:16:19.325 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:16:18 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T21:16:19.325 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:16:18 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:16:21.002 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:16:20 smithi138 ceph-mon[33110]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 241 MiB used, 715 GiB / 715 GiB avail 2024-05-03T21:16:21.024 INFO:teuthology.orchestra.run.smithi138.stdout:Created osd(s) 8 on host 'smithi138' 2024-05-03T21:16:21.024 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:16:21.022+0000 7f89937fe700 1 -- 172.21.15.138:0/2927934871 <== mgr.14150 v2:172.21.15.67:6800/604451592 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f89a40638e0 con 0x7f898c0b2880 2024-05-03T21:16:21.026 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:16:21.024+0000 7f89a8a5c700 1 -- 172.21.15.138:0/2927934871 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f898c0b2880 msgr2=0x7f898c0b4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:16:21.026 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:16:21.024+0000 7f89a8a5c700 1 --2- 172.21.15.138:0/2927934871 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f898c0b2880 0x7f898c0b4d40 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f8998005950 tx=0x7f89980058e0 comp rx=0 tx=0).stop 2024-05-03T21:16:21.026 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:16:21.024+0000 7f89a8a5c700 1 -- 172.21.15.138:0/2927934871 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f89a41023f0 msgr2=0x7f89a41986e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:16:21.026 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:16:21.024+0000 7f89a8a5c700 1 --2- 172.21.15.138:0/2927934871 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f89a41023f0 0x7f89a41986e0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f899400b530 tx=0x7f899400b840 comp rx=0 tx=0).stop 2024-05-03T21:16:21.026 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:16:21.024+0000 7f89a8a5c700 1 -- 172.21.15.138:0/2927934871 shutdown_connections 2024-05-03T21:16:21.026 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:16:21.024+0000 7f89a8a5c700 1 --2- 172.21.15.138:0/2927934871 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f898c0b2880 0x7f898c0b4d40 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:16:21.027 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:16:21.024+0000 7f89a8a5c700 1 --2- 172.21.15.138:0/2927934871 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f89a4103f90 0x7f89a419fca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:16:21.027 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:16:21.024+0000 7f89a8a5c700 1 --2- 172.21.15.138:0/2927934871 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f89a41023f0 0x7f89a41986e0 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:16:21.027 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:16:21.024+0000 7f89a8a5c700 1 --2- 172.21.15.138:0/2927934871 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f89a41035f0 0x7f89a4198c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:16:21.027 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:16:21.024+0000 7f89a8a5c700 1 -- 172.21.15.138:0/2927934871 >> 172.21.15.138:0/2927934871 conn(0x7f89a40fd9d0 msgr2=0x7f89a4106890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T21:16:21.027 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:16:21.024+0000 7f89a8a5c700 1 -- 172.21.15.138:0/2927934871 shutdown_connections 2024-05-03T21:16:21.027 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:16:21.024+0000 7f89a8a5c700 1 -- 172.21.15.138:0/2927934871 wait complete. 2024-05-03T21:16:21.221 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:16:20 smithi067 ceph-mon[29730]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 241 MiB used, 715 GiB / 715 GiB avail 2024-05-03T21:16:21.245 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:16:20 smithi082 ceph-mon[32933]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 241 MiB used, 715 GiB / 715 GiB avail 2024-05-03T21:16:21.978 DEBUG:teuthology.orchestra.run.smithi138:osd.8> sudo journalctl -f -n 0 -u ceph-713bc57a-0991-11ef-bc97-c7b262605968@osd.8.service 2024-05-03T21:16:21.980 INFO:tasks.cephadm:Deploying osd.9 on smithi067 with /dev/vg_nvme/lv_1... 2024-05-03T21:16:21.980 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9a047fe7599a549042d06a3fb5b6813a203eabea shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 713bc57a-0991-11ef-bc97-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-05-03T21:16:22.245 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:16:21 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:16:22.245 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:16:21 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:16:22.245 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:16:21 smithi082 ceph-mon[32933]: from='osd.8 [v2:172.21.15.138:6816/2223920070,v1:172.21.15.138:6817/2223920070]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-05-03T21:16:22.246 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:16:21 smithi082 ceph-mon[32933]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-05-03T21:16:22.250 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/713bc57a-0991-11ef-bc97-c7b262605968/mon.a/config 2024-05-03T21:16:22.324 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:16:21 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:16:22.324 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:16:21 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:16:22.324 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:16:21 smithi138 ceph-mon[33110]: from='osd.8 [v2:172.21.15.138:6816/2223920070,v1:172.21.15.138:6817/2223920070]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-05-03T21:16:22.324 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:16:21 smithi138 ceph-mon[33110]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-05-03T21:16:22.373 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:16:21 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:16:22.374 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:16:21 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:16:22.374 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:16:21 smithi067 ceph-mon[29730]: from='osd.8 [v2:172.21.15.138:6816/2223920070,v1:172.21.15.138:6817/2223920070]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-05-03T21:16:22.374 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:16:21 smithi067 ceph-mon[29730]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-05-03T21:16:23.324 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:16:23 smithi138 ceph-mon[33110]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 241 MiB used, 715 GiB / 715 GiB avail 2024-05-03T21:16:23.325 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:16:23 smithi138 ceph-mon[33110]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-05-03T21:16:23.325 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:16:23 smithi138 ceph-mon[33110]: from='osd.8 [v2:172.21.15.138:6816/2223920070,v1:172.21.15.138:6817/2223920070]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi138", "root=default"]}]: dispatch 2024-05-03T21:16:23.325 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:16:23 smithi138 ceph-mon[33110]: osdmap e54: 9 total, 8 up, 9 in 2024-05-03T21:16:23.325 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:16:23 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-05-03T21:16:23.325 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:16:23 smithi138 ceph-mon[33110]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi138", "root=default"]}]: dispatch 2024-05-03T21:16:23.325 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:16:23 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:16:23.325 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:16:23 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:16:23.407 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:16:23 smithi067 ceph-mon[29730]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 241 MiB used, 715 GiB / 715 GiB avail 2024-05-03T21:16:23.407 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:16:23 smithi067 ceph-mon[29730]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-05-03T21:16:23.407 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:16:23 smithi067 ceph-mon[29730]: from='osd.8 [v2:172.21.15.138:6816/2223920070,v1:172.21.15.138:6817/2223920070]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi138", "root=default"]}]: dispatch 2024-05-03T21:16:23.407 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:16:23 smithi067 ceph-mon[29730]: osdmap e54: 9 total, 8 up, 9 in 2024-05-03T21:16:23.407 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:16:23 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-05-03T21:16:23.407 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:16:23 smithi067 ceph-mon[29730]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi138", "root=default"]}]: dispatch 2024-05-03T21:16:23.407 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:16:23 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:16:23.408 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:16:23 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:16:23.495 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:16:23 smithi082 ceph-mon[32933]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 241 MiB used, 715 GiB / 715 GiB avail 2024-05-03T21:16:23.495 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:16:23 smithi082 ceph-mon[32933]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-05-03T21:16:23.496 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:16:23 smithi082 ceph-mon[32933]: from='osd.8 [v2:172.21.15.138:6816/2223920070,v1:172.21.15.138:6817/2223920070]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi138", "root=default"]}]: dispatch 2024-05-03T21:16:23.496 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:16:23 smithi082 ceph-mon[32933]: osdmap e54: 9 total, 8 up, 9 in 2024-05-03T21:16:23.496 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:16:23 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-05-03T21:16:23.496 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:16:23 smithi082 ceph-mon[32933]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi138", "root=default"]}]: dispatch 2024-05-03T21:16:23.496 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:16:23 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:16:23.496 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:16:23 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:16:24.221 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:16:24 smithi067 ceph-mon[29730]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi138", "root=default"]}]': finished 2024-05-03T21:16:24.221 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:16:24 smithi067 ceph-mon[29730]: osdmap e55: 9 total, 8 up, 9 in 2024-05-03T21:16:24.221 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:16:24 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-05-03T21:16:24.221 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:16:24 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-05-03T21:16:24.221 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:16:24 smithi067 ceph-mon[29730]: from='osd.8 ' entity='osd.8' 2024-05-03T21:16:24.324 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:16:24 smithi138 ceph-mon[33110]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi138", "root=default"]}]': finished 2024-05-03T21:16:24.325 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:16:24 smithi138 ceph-mon[33110]: osdmap e55: 9 total, 8 up, 9 in 2024-05-03T21:16:24.325 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:16:24 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-05-03T21:16:24.325 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:16:24 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-05-03T21:16:24.325 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:16:24 smithi138 ceph-mon[33110]: from='osd.8 ' entity='osd.8' 2024-05-03T21:16:24.325 INFO:journalctl@ceph.osd.8.smithi138.stdout:May 03 21:16:23 smithi138 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-8[51920]: 2024-05-03T21:16:23.908+0000 7f4b892f1700 -1 osd.8 0 waiting for initial osdmap 2024-05-03T21:16:24.325 INFO:journalctl@ceph.osd.8.smithi138.stdout:May 03 21:16:24 smithi138 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-8[51920]: 2024-05-03T21:16:24.104+0000 7f4b810dc700 -1 osd.8 56 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-05-03T21:16:24.427 INFO:teuthology.orchestra.run.smithi067.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-05-03T21:16:24.427 INFO:teuthology.orchestra.run.smithi067.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-05-03T21:16:24.427 INFO:teuthology.orchestra.run.smithi067.stderr: stderr: 10+0 records in 2024-05-03T21:16:24.427 INFO:teuthology.orchestra.run.smithi067.stderr:10+0 records out 2024-05-03T21:16:24.428 INFO:teuthology.orchestra.run.smithi067.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0113741 s, 922 MB/s 2024-05-03T21:16:24.428 INFO:teuthology.orchestra.run.smithi067.stderr: stderr: 2024-05-03T21:16:24.428 INFO:teuthology.orchestra.run.smithi067.stderr:--> Zapping successful for: 2024-05-03T21:16:24.495 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:16:24 smithi082 ceph-mon[32933]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi138", "root=default"]}]': finished 2024-05-03T21:16:24.495 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:16:24 smithi082 ceph-mon[32933]: osdmap e55: 9 total, 8 up, 9 in 2024-05-03T21:16:24.496 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:16:24 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-05-03T21:16:24.496 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:16:24 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-05-03T21:16:24.496 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:16:24 smithi082 ceph-mon[32933]: from='osd.8 ' entity='osd.8' 2024-05-03T21:16:25.168 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9a047fe7599a549042d06a3fb5b6813a203eabea shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 713bc57a-0991-11ef-bc97-c7b262605968 -- ceph orch daemon add osd smithi067:vg_nvme/lv_1 2024-05-03T21:16:25.270 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:16:25 smithi138 ceph-mon[33110]: purged_snaps scrub starts 2024-05-03T21:16:25.270 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:16:25 smithi138 ceph-mon[33110]: purged_snaps scrub ok 2024-05-03T21:16:25.270 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:16:25 smithi138 ceph-mon[33110]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 241 MiB used, 715 GiB / 715 GiB avail 2024-05-03T21:16:25.270 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:16:25 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-05-03T21:16:25.270 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:16:25 smithi138 ceph-mon[33110]: osdmap e56: 9 total, 8 up, 9 in 2024-05-03T21:16:25.270 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:16:25 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-05-03T21:16:25.270 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:16:25 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-05-03T21:16:25.425 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/713bc57a-0991-11ef-bc97-c7b262605968/mon.a/config 2024-05-03T21:16:25.471 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:16:25 smithi067 ceph-mon[29730]: purged_snaps scrub starts 2024-05-03T21:16:25.471 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:16:25 smithi067 ceph-mon[29730]: purged_snaps scrub ok 2024-05-03T21:16:25.471 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:16:25 smithi067 ceph-mon[29730]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 241 MiB used, 715 GiB / 715 GiB avail 2024-05-03T21:16:25.471 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:16:25 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-05-03T21:16:25.471 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:16:25 smithi067 ceph-mon[29730]: osdmap e56: 9 total, 8 up, 9 in 2024-05-03T21:16:25.471 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:16:25 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-05-03T21:16:25.471 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:16:25 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-05-03T21:16:25.496 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:16:25 smithi082 ceph-mon[32933]: purged_snaps scrub starts 2024-05-03T21:16:25.496 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:16:25 smithi082 ceph-mon[32933]: purged_snaps scrub ok 2024-05-03T21:16:25.496 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:16:25 smithi082 ceph-mon[32933]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 241 MiB used, 715 GiB / 715 GiB avail 2024-05-03T21:16:25.496 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:16:25 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-05-03T21:16:25.496 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:16:25 smithi082 ceph-mon[32933]: osdmap e56: 9 total, 8 up, 9 in 2024-05-03T21:16:25.496 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:16:25 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-05-03T21:16:25.496 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:16:25 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-05-03T21:16:26.221 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:16:26 smithi067 ceph-mon[29730]: osd.8 [v2:172.21.15.138:6816/2223920070,v1:172.21.15.138:6817/2223920070] boot 2024-05-03T21:16:26.221 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:16:26 smithi067 ceph-mon[29730]: osdmap e57: 9 total, 9 up, 9 in 2024-05-03T21:16:26.222 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:16:26 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-05-03T21:16:26.222 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:16:26 smithi067 ceph-mon[29730]: Detected new or changed devices on smithi138 2024-05-03T21:16:26.222 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:16:26 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:16:26.222 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:16:26 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:16:26.222 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:16:26 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-05-03T21:16:26.222 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:16:26 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-05-03T21:16:26.222 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:16:26 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-05-03T21:16:26.222 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:16:26 smithi067 ceph-mon[29730]: Adjusting osd_memory_target on smithi138 to 5696M 2024-05-03T21:16:26.222 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:16:26 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:16:26.222 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:16:26 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T21:16:26.222 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:16:26 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T21:16:26.222 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:16:26 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:16:26.223 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:16:26 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T21:16:26.223 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:16:26 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T21:16:26.223 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:16:26 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T21:16:26.223 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:16:26 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:16:26.324 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:16:26 smithi138 ceph-mon[33110]: osd.8 [v2:172.21.15.138:6816/2223920070,v1:172.21.15.138:6817/2223920070] boot 2024-05-03T21:16:26.325 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:16:26 smithi138 ceph-mon[33110]: osdmap e57: 9 total, 9 up, 9 in 2024-05-03T21:16:26.325 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:16:26 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-05-03T21:16:26.325 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:16:26 smithi138 ceph-mon[33110]: Detected new or changed devices on smithi138 2024-05-03T21:16:26.325 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:16:26 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:16:26.325 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:16:26 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:16:26.325 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:16:26 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-05-03T21:16:26.325 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:16:26 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-05-03T21:16:26.325 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:16:26 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-05-03T21:16:26.325 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:16:26 smithi138 ceph-mon[33110]: Adjusting osd_memory_target on smithi138 to 5696M 2024-05-03T21:16:26.326 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:16:26 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:16:26.326 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:16:26 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T21:16:26.326 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:16:26 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T21:16:26.326 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:16:26 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:16:26.326 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:16:26 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T21:16:26.326 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:16:26 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T21:16:26.326 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:16:26 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T21:16:26.326 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:16:26 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:16:26.496 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:16:26 smithi082 ceph-mon[32933]: osd.8 [v2:172.21.15.138:6816/2223920070,v1:172.21.15.138:6817/2223920070] boot 2024-05-03T21:16:26.496 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:16:26 smithi082 ceph-mon[32933]: osdmap e57: 9 total, 9 up, 9 in 2024-05-03T21:16:26.496 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:16:26 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-05-03T21:16:26.496 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:16:26 smithi082 ceph-mon[32933]: Detected new or changed devices on smithi138 2024-05-03T21:16:26.496 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:16:26 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:16:26.496 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:16:26 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:16:26.496 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:16:26 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-05-03T21:16:26.496 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:16:26 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-05-03T21:16:26.497 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:16:26 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-05-03T21:16:26.497 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:16:26 smithi082 ceph-mon[32933]: Adjusting osd_memory_target on smithi138 to 5696M 2024-05-03T21:16:26.497 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:16:26 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:16:26.497 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:16:26 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T21:16:26.497 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:16:26 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T21:16:26.497 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:16:26 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:16:26.497 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:16:26 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T21:16:26.497 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:16:26 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T21:16:26.497 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:16:26 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T21:16:26.497 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:16:26 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:16:27.074 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:16:27 smithi138 ceph-mon[33110]: pgmap v193: 1 pgs: 1 remapped+peering; 577 KiB data, 241 MiB used, 715 GiB / 715 GiB avail 2024-05-03T21:16:27.075 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:16:27 smithi138 ceph-mon[33110]: osdmap e58: 9 total, 9 up, 9 in 2024-05-03T21:16:27.169 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:16:27.164+0000 7f0cc0ae8700 1 -- 172.21.15.67:0/2467800548 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0cbc103fa0 msgr2=0x7f0cbc10a850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:16:27.169 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:16:27.164+0000 7f0cc0ae8700 1 --2- 172.21.15.67:0/2467800548 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0cbc103fa0 0x7f0cbc10a850 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f0cb400b0d0 tx=0x7f0cb400b3e0 comp rx=0 tx=0).stop 2024-05-03T21:16:27.169 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:16:27.164+0000 7f0cc0ae8700 1 -- 172.21.15.67:0/2467800548 shutdown_connections 2024-05-03T21:16:27.169 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:16:27.164+0000 7f0cc0ae8700 1 --2- 172.21.15.67:0/2467800548 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0cbc103fa0 0x7f0cbc10a850 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:16:27.169 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:16:27.164+0000 7f0cc0ae8700 1 --2- 172.21.15.67:0/2467800548 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f0cbc102400 0x7f0cbc102820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:16:27.169 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:16:27.164+0000 7f0cc0ae8700 1 --2- 172.21.15.67:0/2467800548 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f0cbc103600 0x7f0cbc103a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:16:27.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:16:27.164+0000 7f0cc0ae8700 1 -- 172.21.15.67:0/2467800548 >> 172.21.15.67:0/2467800548 conn(0x7f0cbc0fd9a0 msgr2=0x7f0cbc0ffde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T21:16:27.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:16:27.164+0000 7f0cc0ae8700 1 -- 172.21.15.67:0/2467800548 shutdown_connections 2024-05-03T21:16:27.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:16:27.164+0000 7f0cc0ae8700 1 -- 172.21.15.67:0/2467800548 wait complete. 2024-05-03T21:16:27.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:16:27.165+0000 7f0cc0ae8700 1 Processor -- start 2024-05-03T21:16:27.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:16:27.165+0000 7f0cc0ae8700 1 -- start start 2024-05-03T21:16:27.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:16:27.165+0000 7f0cc0ae8700 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f0cbc102400 0x7f0cbc106390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:16:27.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:16:27.165+0000 7f0cc0ae8700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0cbc103600 0x7f0cbc1049e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:16:27.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:16:27.165+0000 7f0cc0ae8700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f0cbc103fa0 0x7f0cbc104f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:16:27.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:16:27.165+0000 7f0cc0ae8700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f0cbc10e120 con 0x7f0cbc103600 2024-05-03T21:16:27.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:16:27.165+0000 7f0cc0ae8700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f0cbc10dfa0 con 0x7f0cbc103fa0 2024-05-03T21:16:27.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:16:27.165+0000 7f0cc0ae8700 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f0cbc10e2a0 con 0x7f0cbc102400 2024-05-03T21:16:27.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:16:27.165+0000 7f0cb9d9b700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0cbc103600 0x7f0cbc1049e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:16:27.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:16:27.165+0000 7f0cb9d9b700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0cbc103600 0x7f0cbc1049e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:47498/0 (socket says 172.21.15.67:47498) 2024-05-03T21:16:27.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:16:27.165+0000 7f0cb9d9b700 1 -- 172.21.15.67:0/1874669515 learned_addr learned my addr 172.21.15.67:0/1874669515 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-03T21:16:27.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:16:27.166+0000 7f0cbad9d700 1 --2- 172.21.15.67:0/1874669515 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f0cbc103fa0 0x7f0cbc104f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:16:27.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:16:27.166+0000 7f0cba59c700 1 --2- 172.21.15.67:0/1874669515 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f0cbc102400 0x7f0cbc106390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:16:27.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:16:27.166+0000 7f0cb9d9b700 1 -- 172.21.15.67:0/1874669515 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f0cbc102400 msgr2=0x7f0cbc106390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:16:27.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:16:27.166+0000 7f0cb9d9b700 1 --2- 172.21.15.67:0/1874669515 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f0cbc102400 0x7f0cbc106390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:16:27.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:16:27.166+0000 7f0cb9d9b700 1 -- 172.21.15.67:0/1874669515 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f0cbc103fa0 msgr2=0x7f0cbc104f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:16:27.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:16:27.166+0000 7f0cb9d9b700 1 --2- 172.21.15.67:0/1874669515 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f0cbc103fa0 0x7f0cbc104f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:16:27.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:16:27.166+0000 7f0cb9d9b700 1 -- 172.21.15.67:0/1874669515 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0cbc1057d0 con 0x7f0cbc103600 2024-05-03T21:16:27.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:16:27.166+0000 7f0cb9d9b700 1 --2- 172.21.15.67:0/1874669515 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0cbc103600 0x7f0cbc1049e0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f0cb000d980 tx=0x7f0cb000dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T21:16:27.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:16:27.166+0000 7f0cab7fe700 1 -- 172.21.15.67:0/1874669515 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0cb0017070 con 0x7f0cbc103600 2024-05-03T21:16:27.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:16:27.166+0000 7f0cab7fe700 1 -- 172.21.15.67:0/1874669515 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(28 keys) ==== 1170+0+0 (secure 0 0 0) 0x7f0cb0005020 con 0x7f0cbc103600 2024-05-03T21:16:27.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:16:27.166+0000 7f0cc0ae8700 1 -- 172.21.15.67:0/1874669515 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0cbc1a3a30 con 0x7f0cbc103600 2024-05-03T21:16:27.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:16:27.166+0000 7f0cab7fe700 1 -- 172.21.15.67:0/1874669515 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0cb000fc60 con 0x7f0cbc103600 2024-05-03T21:16:27.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:16:27.166+0000 7f0cc0ae8700 1 -- 172.21.15.67:0/1874669515 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0cbc1a3ef0 con 0x7f0cbc103600 2024-05-03T21:16:27.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:16:27.167+0000 7f0cc0ae8700 1 -- 172.21.15.67:0/1874669515 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0cbc06a190 con 0x7f0cbc103600 2024-05-03T21:16:27.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:16:27.169+0000 7f0cab7fe700 1 -- 172.21.15.67:0/1874669515 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 16) ==== 149652+0+0 (secure 0 0 0) 0x7f0cb00051d0 con 0x7f0cbc103600 2024-05-03T21:16:27.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:16:27.169+0000 7f0cab7fe700 1 --2- 172.21.15.67:0/1874669515 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f0ca40b2880 0x7f0ca40b4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:16:27.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:16:27.169+0000 7f0cba59c700 1 --2- 172.21.15.67:0/1874669515 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f0ca40b2880 0x7f0ca40b4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:16:27.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:16:27.169+0000 7f0cab7fe700 1 -- 172.21.15.67:0/1874669515 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(59..59 src has 1..59) ==== 5829+0+0 (secure 0 0 0) 0x7f0cb00e16b0 con 0x7f0cbc103600 2024-05-03T21:16:27.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:16:27.170+0000 7f0cba59c700 1 --2- 172.21.15.67:0/1874669515 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f0ca40b2880 0x7f0ca40b4d40 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f0cbc103460 tx=0x7f0cac00b410 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T21:16:27.176 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:16:27.174+0000 7f0cab7fe700 1 -- 172.21.15.67:0/1874669515 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191839 (secure 0 0 0) 0x7f0cb0020020 con 0x7f0cbc103600 2024-05-03T21:16:27.221 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:16:27 smithi067 ceph-mon[29730]: pgmap v193: 1 pgs: 1 remapped+peering; 577 KiB data, 241 MiB used, 715 GiB / 715 GiB avail 2024-05-03T21:16:27.221 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:16:27 smithi067 ceph-mon[29730]: osdmap e58: 9 total, 9 up, 9 in 2024-05-03T21:16:27.399 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:16:27.397+0000 7f0cc0ae8700 1 -- 172.21.15.67:0/1874669515 --> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi067:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) -- 0x7f0cbc0ffac0 con 0x7f0ca40b2880 2024-05-03T21:16:27.495 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:16:27 smithi082 ceph-mon[32933]: pgmap v193: 1 pgs: 1 remapped+peering; 577 KiB data, 241 MiB used, 715 GiB / 715 GiB avail 2024-05-03T21:16:27.496 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:16:27 smithi082 ceph-mon[32933]: osdmap e58: 9 total, 9 up, 9 in 2024-05-03T21:16:28.324 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:16:28 smithi138 ceph-mon[33110]: osdmap e59: 9 total, 9 up, 9 in 2024-05-03T21:16:28.325 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:16:28 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-03T21:16:28.325 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:16:28 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-03T21:16:28.325 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:16:28 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T21:16:28.371 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:16:28 smithi067 ceph-mon[29730]: osdmap e59: 9 total, 9 up, 9 in 2024-05-03T21:16:28.371 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:16:28 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-03T21:16:28.371 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:16:28 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-03T21:16:28.371 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:16:28 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T21:16:28.495 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:16:28 smithi082 ceph-mon[32933]: osdmap e59: 9 total, 9 up, 9 in 2024-05-03T21:16:28.495 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:16:28 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-03T21:16:28.495 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:16:28 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-03T21:16:28.495 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:16:28 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T21:16:29.324 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:16:29 smithi138 ceph-mon[33110]: from='client.14427 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi067:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-05-03T21:16:29.324 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:16:29 smithi138 ceph-mon[33110]: pgmap v196: 1 pgs: 1 remapped+peering; 577 KiB data, 272 MiB used, 804 GiB / 805 GiB avail 2024-05-03T21:16:29.404 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:16:29 smithi067 ceph-mon[29730]: from='client.14427 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi067:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-05-03T21:16:29.404 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:16:29 smithi067 ceph-mon[29730]: pgmap v196: 1 pgs: 1 remapped+peering; 577 KiB data, 272 MiB used, 804 GiB / 805 GiB avail 2024-05-03T21:16:29.495 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:16:29 smithi082 ceph-mon[32933]: from='client.14427 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi067:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-05-03T21:16:29.495 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:16:29 smithi082 ceph-mon[32933]: pgmap v196: 1 pgs: 1 remapped+peering; 577 KiB data, 272 MiB used, 804 GiB / 805 GiB avail 2024-05-03T21:16:31.324 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:16:31 smithi138 ceph-mon[33110]: pgmap v197: 1 pgs: 1 remapped+peering; 577 KiB data, 272 MiB used, 804 GiB / 805 GiB avail 2024-05-03T21:16:31.324 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:16:31 smithi138 ceph-mon[33110]: from='client.? 172.21.15.67:0/960150946' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3d387dbf-0163-41b7-98af-4fc45ed73ad4"}]: dispatch 2024-05-03T21:16:31.324 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:16:31 smithi138 ceph-mon[33110]: from='client.? 172.21.15.67:0/960150946' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "3d387dbf-0163-41b7-98af-4fc45ed73ad4"}]': finished 2024-05-03T21:16:31.324 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:16:31 smithi138 ceph-mon[33110]: osdmap e60: 10 total, 9 up, 10 in 2024-05-03T21:16:31.325 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:16:31 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-05-03T21:16:31.325 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:16:31 smithi138 ceph-mon[33110]: from='client.? 172.21.15.67:0/2328501502' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-03T21:16:31.471 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:16:31 smithi067 ceph-mon[29730]: pgmap v197: 1 pgs: 1 remapped+peering; 577 KiB data, 272 MiB used, 804 GiB / 805 GiB avail 2024-05-03T21:16:31.471 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:16:31 smithi067 ceph-mon[29730]: from='client.? 172.21.15.67:0/960150946' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3d387dbf-0163-41b7-98af-4fc45ed73ad4"}]: dispatch 2024-05-03T21:16:31.471 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:16:31 smithi067 ceph-mon[29730]: from='client.? 172.21.15.67:0/960150946' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "3d387dbf-0163-41b7-98af-4fc45ed73ad4"}]': finished 2024-05-03T21:16:31.472 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:16:31 smithi067 ceph-mon[29730]: osdmap e60: 10 total, 9 up, 10 in 2024-05-03T21:16:31.472 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:16:31 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-05-03T21:16:31.472 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:16:31 smithi067 ceph-mon[29730]: from='client.? 172.21.15.67:0/2328501502' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-03T21:16:31.495 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:16:31 smithi082 ceph-mon[32933]: pgmap v197: 1 pgs: 1 remapped+peering; 577 KiB data, 272 MiB used, 804 GiB / 805 GiB avail 2024-05-03T21:16:31.495 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:16:31 smithi082 ceph-mon[32933]: from='client.? 172.21.15.67:0/960150946' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3d387dbf-0163-41b7-98af-4fc45ed73ad4"}]: dispatch 2024-05-03T21:16:31.495 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:16:31 smithi082 ceph-mon[32933]: from='client.? 172.21.15.67:0/960150946' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "3d387dbf-0163-41b7-98af-4fc45ed73ad4"}]': finished 2024-05-03T21:16:31.495 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:16:31 smithi082 ceph-mon[32933]: osdmap e60: 10 total, 9 up, 10 in 2024-05-03T21:16:31.495 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:16:31 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-05-03T21:16:31.496 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:16:31 smithi082 ceph-mon[32933]: from='client.? 172.21.15.67:0/2328501502' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-03T21:16:33.324 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:16:33 smithi138 ceph-mon[33110]: pgmap v199: 1 pgs: 1 active+clean; 577 KiB data, 273 MiB used, 804 GiB / 805 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-05-03T21:16:33.471 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:16:33 smithi067 ceph-mon[29730]: pgmap v199: 1 pgs: 1 active+clean; 577 KiB data, 273 MiB used, 804 GiB / 805 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-05-03T21:16:33.495 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:16:33 smithi082 ceph-mon[32933]: pgmap v199: 1 pgs: 1 active+clean; 577 KiB data, 273 MiB used, 804 GiB / 805 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-05-03T21:16:35.221 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:16:35 smithi067 ceph-mon[29730]: pgmap v200: 1 pgs: 1 active+clean; 577 KiB data, 273 MiB used, 804 GiB / 805 GiB avail; 74 KiB/s, 0 objects/s recovering 2024-05-03T21:16:35.324 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:16:35 smithi138 ceph-mon[33110]: pgmap v200: 1 pgs: 1 active+clean; 577 KiB data, 273 MiB used, 804 GiB / 805 GiB avail; 74 KiB/s, 0 objects/s recovering 2024-05-03T21:16:35.495 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:16:35 smithi082 ceph-mon[32933]: pgmap v200: 1 pgs: 1 active+clean; 577 KiB data, 273 MiB used, 804 GiB / 805 GiB avail; 74 KiB/s, 0 objects/s recovering 2024-05-03T21:16:37.324 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:16:37 smithi138 ceph-mon[33110]: pgmap v201: 1 pgs: 1 active+clean; 577 KiB data, 273 MiB used, 804 GiB / 805 GiB avail; 66 KiB/s, 0 objects/s recovering 2024-05-03T21:16:37.430 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:16:37 smithi067 ceph-mon[29730]: pgmap v201: 1 pgs: 1 active+clean; 577 KiB data, 273 MiB used, 804 GiB / 805 GiB avail; 66 KiB/s, 0 objects/s recovering 2024-05-03T21:16:37.495 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:16:37 smithi082 ceph-mon[32933]: pgmap v201: 1 pgs: 1 active+clean; 577 KiB data, 273 MiB used, 804 GiB / 805 GiB avail; 66 KiB/s, 0 objects/s recovering 2024-05-03T21:16:39.324 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:16:39 smithi138 ceph-mon[33110]: pgmap v202: 1 pgs: 1 active+clean; 577 KiB data, 273 MiB used, 804 GiB / 805 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-05-03T21:16:39.325 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:16:39 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-05-03T21:16:39.325 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:16:39 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T21:16:39.325 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:16:39 smithi138 ceph-mon[33110]: Deploying daemon osd.9 on smithi067 2024-05-03T21:16:39.471 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:16:39 smithi067 ceph-mon[29730]: pgmap v202: 1 pgs: 1 active+clean; 577 KiB data, 273 MiB used, 804 GiB / 805 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-05-03T21:16:39.471 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:16:39 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-05-03T21:16:39.471 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:16:39 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T21:16:39.471 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:16:39 smithi067 ceph-mon[29730]: Deploying daemon osd.9 on smithi067 2024-05-03T21:16:39.495 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:16:39 smithi082 ceph-mon[32933]: pgmap v202: 1 pgs: 1 active+clean; 577 KiB data, 273 MiB used, 804 GiB / 805 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-05-03T21:16:39.496 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:16:39 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-05-03T21:16:39.496 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:16:39 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T21:16:39.496 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:16:39 smithi082 ceph-mon[32933]: Deploying daemon osd.9 on smithi067 2024-05-03T21:16:41.221 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:16:41 smithi067 ceph-mon[29730]: pgmap v203: 1 pgs: 1 active+clean; 577 KiB data, 273 MiB used, 804 GiB / 805 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-05-03T21:16:41.324 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:16:41 smithi138 ceph-mon[33110]: pgmap v203: 1 pgs: 1 active+clean; 577 KiB data, 273 MiB used, 804 GiB / 805 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-05-03T21:16:41.495 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:16:41 smithi082 ceph-mon[32933]: pgmap v203: 1 pgs: 1 active+clean; 577 KiB data, 273 MiB used, 804 GiB / 805 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-05-03T21:16:43.324 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:16:43 smithi138 ceph-mon[33110]: pgmap v204: 1 pgs: 1 active+clean; 577 KiB data, 273 MiB used, 804 GiB / 805 GiB avail 2024-05-03T21:16:43.471 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:16:43 smithi067 ceph-mon[29730]: pgmap v204: 1 pgs: 1 active+clean; 577 KiB data, 273 MiB used, 804 GiB / 805 GiB avail 2024-05-03T21:16:43.495 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:16:43 smithi082 ceph-mon[32933]: pgmap v204: 1 pgs: 1 active+clean; 577 KiB data, 273 MiB used, 804 GiB / 805 GiB avail 2024-05-03T21:16:45.182 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:16:45 smithi067 ceph-mon[29730]: pgmap v205: 1 pgs: 1 active+clean; 577 KiB data, 273 MiB used, 804 GiB / 805 GiB avail 2024-05-03T21:16:45.182 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:16:45 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T21:16:45.182 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:16:45 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:16:45.182 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:16:45 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:16:45.324 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:16:45 smithi138 ceph-mon[33110]: pgmap v205: 1 pgs: 1 active+clean; 577 KiB data, 273 MiB used, 804 GiB / 805 GiB avail 2024-05-03T21:16:45.324 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:16:45 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T21:16:45.325 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:16:45 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:16:45.325 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:16:45 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:16:45.495 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:16:45 smithi082 ceph-mon[32933]: pgmap v205: 1 pgs: 1 active+clean; 577 KiB data, 273 MiB used, 804 GiB / 805 GiB avail 2024-05-03T21:16:45.495 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:16:45 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T21:16:45.495 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:16:45 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:16:45.495 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:16:45 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:16:47.324 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:16:47 smithi138 ceph-mon[33110]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 273 MiB used, 804 GiB / 805 GiB avail 2024-05-03T21:16:47.389 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:16:47 smithi067 ceph-mon[29730]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 273 MiB used, 804 GiB / 805 GiB avail 2024-05-03T21:16:47.463 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:16:47.461+0000 7f0cab7fe700 1 -- 172.21.15.67:0/1874669515 <== mgr.14150 v2:172.21.15.67:6800/604451592 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f0cbc0ffac0 con 0x7f0ca40b2880 2024-05-03T21:16:47.463 INFO:teuthology.orchestra.run.smithi067.stdout:Created osd(s) 9 on host 'smithi067' 2024-05-03T21:16:47.465 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:16:47.463+0000 7f0cc0ae8700 1 -- 172.21.15.67:0/1874669515 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f0ca40b2880 msgr2=0x7f0ca40b4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:16:47.465 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:16:47.463+0000 7f0cc0ae8700 1 --2- 172.21.15.67:0/1874669515 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f0ca40b2880 0x7f0ca40b4d40 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f0cbc103460 tx=0x7f0cac00b410 comp rx=0 tx=0).stop 2024-05-03T21:16:47.466 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:16:47.463+0000 7f0cc0ae8700 1 -- 172.21.15.67:0/1874669515 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0cbc103600 msgr2=0x7f0cbc1049e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:16:47.466 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:16:47.463+0000 7f0cc0ae8700 1 --2- 172.21.15.67:0/1874669515 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0cbc103600 0x7f0cbc1049e0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f0cb000d980 tx=0x7f0cb000dc90 comp rx=0 tx=0).stop 2024-05-03T21:16:47.466 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:16:47.463+0000 7f0cc0ae8700 1 -- 172.21.15.67:0/1874669515 shutdown_connections 2024-05-03T21:16:47.466 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:16:47.463+0000 7f0cc0ae8700 1 --2- 172.21.15.67:0/1874669515 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f0ca40b2880 0x7f0ca40b4d40 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:16:47.466 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:16:47.463+0000 7f0cc0ae8700 1 --2- 172.21.15.67:0/1874669515 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f0cbc103fa0 0x7f0cbc104f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:16:47.466 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:16:47.463+0000 7f0cc0ae8700 1 --2- 172.21.15.67:0/1874669515 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f0cbc102400 0x7f0cbc106390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:16:47.466 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:16:47.463+0000 7f0cc0ae8700 1 --2- 172.21.15.67:0/1874669515 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0cbc103600 0x7f0cbc1049e0 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:16:47.466 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:16:47.463+0000 7f0cc0ae8700 1 -- 172.21.15.67:0/1874669515 >> 172.21.15.67:0/1874669515 conn(0x7f0cbc0fd9a0 msgr2=0x7f0cbc0ff3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T21:16:47.466 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:16:47.463+0000 7f0cc0ae8700 1 -- 172.21.15.67:0/1874669515 shutdown_connections 2024-05-03T21:16:47.466 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:16:47.463+0000 7f0cc0ae8700 1 -- 172.21.15.67:0/1874669515 wait complete. 2024-05-03T21:16:47.495 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:16:47 smithi082 ceph-mon[32933]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 273 MiB used, 804 GiB / 805 GiB avail 2024-05-03T21:16:48.485 DEBUG:teuthology.orchestra.run.smithi067:osd.9> sudo journalctl -f -n 0 -u ceph-713bc57a-0991-11ef-bc97-c7b262605968@osd.9.service 2024-05-03T21:16:48.487 INFO:tasks.cephadm:Deploying osd.10 on smithi082 with /dev/vg_nvme/lv_1... 2024-05-03T21:16:48.487 DEBUG:teuthology.orchestra.run.smithi082:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9a047fe7599a549042d06a3fb5b6813a203eabea shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 713bc57a-0991-11ef-bc97-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-05-03T21:16:48.721 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:16:48 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:16:48.721 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:16:48 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:16:48.721 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:16:48 smithi067 ceph-mon[29730]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 273 MiB used, 804 GiB / 805 GiB avail 2024-05-03T21:16:48.721 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:16:48 smithi067 ceph-mon[29730]: from='osd.9 [v2:172.21.15.67:6826/3939417129,v1:172.21.15.67:6827/3939417129]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-05-03T21:16:48.733 INFO:teuthology.orchestra.run.smithi082.stderr:Inferring config /var/lib/ceph/713bc57a-0991-11ef-bc97-c7b262605968/mon.b/config 2024-05-03T21:16:48.745 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:16:48 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:16:48.745 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:16:48 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:16:48.745 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:16:48 smithi082 ceph-mon[32933]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 273 MiB used, 804 GiB / 805 GiB avail 2024-05-03T21:16:48.745 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:16:48 smithi082 ceph-mon[32933]: from='osd.9 [v2:172.21.15.67:6826/3939417129,v1:172.21.15.67:6827/3939417129]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-05-03T21:16:48.824 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:16:48 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:16:48.824 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:16:48 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:16:48.825 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:16:48 smithi138 ceph-mon[33110]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 273 MiB used, 804 GiB / 805 GiB avail 2024-05-03T21:16:48.825 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:16:48 smithi138 ceph-mon[33110]: from='osd.9 [v2:172.21.15.67:6826/3939417129,v1:172.21.15.67:6827/3939417129]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-05-03T21:16:49.690 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:16:49 smithi067 ceph-mon[29730]: from='osd.9 [v2:172.21.15.67:6826/3939417129,v1:172.21.15.67:6827/3939417129]' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-05-03T21:16:49.691 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:16:49 smithi067 ceph-mon[29730]: osdmap e61: 10 total, 9 up, 10 in 2024-05-03T21:16:49.691 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:16:49 smithi067 ceph-mon[29730]: from='osd.9 [v2:172.21.15.67:6826/3939417129,v1:172.21.15.67:6827/3939417129]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi067", "root=default"]}]: dispatch 2024-05-03T21:16:49.691 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:16:49 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-05-03T21:16:49.691 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:16:49 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:16:49.691 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:16:49 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:16:49.692 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:16:49 smithi082 ceph-mon[32933]: from='osd.9 [v2:172.21.15.67:6826/3939417129,v1:172.21.15.67:6827/3939417129]' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-05-03T21:16:49.692 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:16:49 smithi082 ceph-mon[32933]: osdmap e61: 10 total, 9 up, 10 in 2024-05-03T21:16:49.692 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:16:49 smithi082 ceph-mon[32933]: from='osd.9 [v2:172.21.15.67:6826/3939417129,v1:172.21.15.67:6827/3939417129]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi067", "root=default"]}]: dispatch 2024-05-03T21:16:49.692 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:16:49 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-05-03T21:16:49.692 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:16:49 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:16:49.692 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:16:49 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:16:49.824 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:16:49 smithi138 ceph-mon[33110]: from='osd.9 [v2:172.21.15.67:6826/3939417129,v1:172.21.15.67:6827/3939417129]' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-05-03T21:16:49.825 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:16:49 smithi138 ceph-mon[33110]: osdmap e61: 10 total, 9 up, 10 in 2024-05-03T21:16:49.825 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:16:49 smithi138 ceph-mon[33110]: from='osd.9 [v2:172.21.15.67:6826/3939417129,v1:172.21.15.67:6827/3939417129]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi067", "root=default"]}]: dispatch 2024-05-03T21:16:49.825 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:16:49 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-05-03T21:16:49.825 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:16:49 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:16:49.825 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:16:49 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:16:50.481 INFO:teuthology.orchestra.run.smithi082.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-05-03T21:16:50.481 INFO:teuthology.orchestra.run.smithi082.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-05-03T21:16:50.481 INFO:teuthology.orchestra.run.smithi082.stderr: stderr: 10+0 records in 2024-05-03T21:16:50.481 INFO:teuthology.orchestra.run.smithi082.stderr:10+0 records out 2024-05-03T21:16:50.482 INFO:teuthology.orchestra.run.smithi082.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.011366 s, 923 MB/s 2024-05-03T21:16:50.482 INFO:teuthology.orchestra.run.smithi082.stderr:--> Zapping successful for: 2024-05-03T21:16:50.505 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:16:50 smithi082 ceph-mon[32933]: from='osd.9 [v2:172.21.15.67:6826/3939417129,v1:172.21.15.67:6827/3939417129]' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi067", "root=default"]}]': finished 2024-05-03T21:16:50.505 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:16:50 smithi082 ceph-mon[32933]: osdmap e62: 10 total, 9 up, 10 in 2024-05-03T21:16:50.505 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:16:50 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-05-03T21:16:50.505 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:16:50 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-05-03T21:16:50.505 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:16:50 smithi082 ceph-mon[32933]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 273 MiB used, 804 GiB / 805 GiB avail 2024-05-03T21:16:50.505 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:16:50 smithi082 ceph-mon[32933]: from='osd.9 [v2:172.21.15.67:6826/3939417129,v1:172.21.15.67:6827/3939417129]' entity='osd.9' 2024-05-03T21:16:50.598 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:16:50 smithi067 ceph-mon[29730]: from='osd.9 [v2:172.21.15.67:6826/3939417129,v1:172.21.15.67:6827/3939417129]' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi067", "root=default"]}]': finished 2024-05-03T21:16:50.599 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:16:50 smithi067 ceph-mon[29730]: osdmap e62: 10 total, 9 up, 10 in 2024-05-03T21:16:50.599 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:16:50 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-05-03T21:16:50.599 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:16:50 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-05-03T21:16:50.599 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:16:50 smithi067 ceph-mon[29730]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 273 MiB used, 804 GiB / 805 GiB avail 2024-05-03T21:16:50.599 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:16:50 smithi067 ceph-mon[29730]: from='osd.9 [v2:172.21.15.67:6826/3939417129,v1:172.21.15.67:6827/3939417129]' entity='osd.9' 2024-05-03T21:16:50.599 INFO:journalctl@ceph.osd.9.smithi067.stdout:May 03 21:16:50 smithi067 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-9[62235]: 2024-05-03T21:16:50.335+0000 7fcfffcb4700 -1 osd.9 0 waiting for initial osdmap 2024-05-03T21:16:50.599 INFO:journalctl@ceph.osd.9.smithi067.stdout:May 03 21:16:50 smithi067 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-9[62235]: 2024-05-03T21:16:50.425+0000 7fcff7a9f700 -1 osd.9 62 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-05-03T21:16:50.824 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:16:50 smithi138 ceph-mon[33110]: from='osd.9 [v2:172.21.15.67:6826/3939417129,v1:172.21.15.67:6827/3939417129]' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi067", "root=default"]}]': finished 2024-05-03T21:16:50.824 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:16:50 smithi138 ceph-mon[33110]: osdmap e62: 10 total, 9 up, 10 in 2024-05-03T21:16:50.824 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:16:50 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-05-03T21:16:50.824 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:16:50 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-05-03T21:16:50.825 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:16:50 smithi138 ceph-mon[33110]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 273 MiB used, 804 GiB / 805 GiB avail 2024-05-03T21:16:50.825 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:16:50 smithi138 ceph-mon[33110]: from='osd.9 [v2:172.21.15.67:6826/3939417129,v1:172.21.15.67:6827/3939417129]' entity='osd.9' 2024-05-03T21:16:51.051 DEBUG:teuthology.orchestra.run.smithi082:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9a047fe7599a549042d06a3fb5b6813a203eabea shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 713bc57a-0991-11ef-bc97-c7b262605968 -- ceph orch daemon add osd smithi082:vg_nvme/lv_1 2024-05-03T21:16:51.286 INFO:teuthology.orchestra.run.smithi082.stderr:Inferring config /var/lib/ceph/713bc57a-0991-11ef-bc97-c7b262605968/mon.b/config 2024-05-03T21:16:51.493 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:16:51 smithi067 ceph-mon[29730]: purged_snaps scrub starts 2024-05-03T21:16:51.493 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:16:51 smithi067 ceph-mon[29730]: purged_snaps scrub ok 2024-05-03T21:16:51.493 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:16:51 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-05-03T21:16:51.493 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:16:51 smithi067 ceph-mon[29730]: osd.9 [v2:172.21.15.67:6826/3939417129,v1:172.21.15.67:6827/3939417129] boot 2024-05-03T21:16:51.494 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:16:51 smithi067 ceph-mon[29730]: osdmap e63: 10 total, 10 up, 10 in 2024-05-03T21:16:51.494 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:16:51 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-05-03T21:16:51.743 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:16:51 smithi082 ceph-mon[32933]: purged_snaps scrub starts 2024-05-03T21:16:51.743 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:16:51 smithi082 ceph-mon[32933]: purged_snaps scrub ok 2024-05-03T21:16:51.743 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:16:51 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-05-03T21:16:51.743 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:16:51 smithi082 ceph-mon[32933]: osd.9 [v2:172.21.15.67:6826/3939417129,v1:172.21.15.67:6827/3939417129] boot 2024-05-03T21:16:51.744 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:16:51 smithi082 ceph-mon[32933]: osdmap e63: 10 total, 10 up, 10 in 2024-05-03T21:16:51.744 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:16:51 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-05-03T21:16:51.824 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:16:51 smithi138 ceph-mon[33110]: purged_snaps scrub starts 2024-05-03T21:16:51.824 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:16:51 smithi138 ceph-mon[33110]: purged_snaps scrub ok 2024-05-03T21:16:51.825 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:16:51 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-05-03T21:16:51.825 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:16:51 smithi138 ceph-mon[33110]: osd.9 [v2:172.21.15.67:6826/3939417129,v1:172.21.15.67:6827/3939417129] boot 2024-05-03T21:16:51.825 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:16:51 smithi138 ceph-mon[33110]: osdmap e63: 10 total, 10 up, 10 in 2024-05-03T21:16:51.825 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:16:51 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-05-03T21:16:52.790 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T21:16:52.785+0000 7feac844e700 1 -- 172.21.15.82:0/709802459 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7feac01017b0 msgr2=0x7feac0101bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:16:52.791 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T21:16:52.785+0000 7feac844e700 1 --2- 172.21.15.82:0/709802459 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7feac01017b0 0x7feac0101bd0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7feab0009990 tx=0x7feab0009ca0 comp rx=0 tx=0).stop 2024-05-03T21:16:52.791 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T21:16:52.785+0000 7feac844e700 1 -- 172.21.15.82:0/709802459 shutdown_connections 2024-05-03T21:16:52.791 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T21:16:52.785+0000 7feac844e700 1 --2- 172.21.15.82:0/709802459 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7feac0105130 0x7feac010b9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:16:52.791 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T21:16:52.785+0000 7feac844e700 1 --2- 172.21.15.82:0/709802459 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7feac01017b0 0x7feac0101bd0 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:16:52.791 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T21:16:52.785+0000 7feac844e700 1 --2- 172.21.15.82:0/709802459 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7feac0104790 0x7feac0104bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:16:52.791 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T21:16:52.785+0000 7feac844e700 1 -- 172.21.15.82:0/709802459 >> 172.21.15.82:0/709802459 conn(0x7feac00fd130 msgr2=0x7feac00ff550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T21:16:52.791 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T21:16:52.785+0000 7feac844e700 1 -- 172.21.15.82:0/709802459 shutdown_connections 2024-05-03T21:16:52.791 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T21:16:52.785+0000 7feac844e700 1 -- 172.21.15.82:0/709802459 wait complete. 2024-05-03T21:16:52.791 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T21:16:52.786+0000 7feac844e700 1 Processor -- start 2024-05-03T21:16:52.791 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T21:16:52.786+0000 7feac844e700 1 -- start start 2024-05-03T21:16:52.791 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T21:16:52.786+0000 7feac844e700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7feac01017b0 0x7feac0199760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:16:52.791 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T21:16:52.786+0000 7feac844e700 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7feac0104790 0x7feac0199ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:16:52.791 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T21:16:52.786+0000 7feac844e700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7feac0105130 0x7feac01a0d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:16:52.792 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T21:16:52.786+0000 7feac844e700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7feac010f200 con 0x7feac0105130 2024-05-03T21:16:52.792 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T21:16:52.786+0000 7feac844e700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7feac010f080 con 0x7feac01017b0 2024-05-03T21:16:52.792 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T21:16:52.786+0000 7feac844e700 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7feac010f380 con 0x7feac0104790 2024-05-03T21:16:52.792 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T21:16:52.786+0000 7feac61ea700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7feac01017b0 0x7feac0199760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:16:52.792 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T21:16:52.786+0000 7feac61ea700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7feac01017b0 0x7feac0199760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.82:50550/0 (socket says 172.21.15.82:50550) 2024-05-03T21:16:52.792 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T21:16:52.786+0000 7feac61ea700 1 -- 172.21.15.82:0/416375462 learned_addr learned my addr 172.21.15.82:0/416375462 (peer_addr_for_me v2:172.21.15.82:0/0) 2024-05-03T21:16:52.792 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T21:16:52.786+0000 7feac69eb700 1 --2- 172.21.15.82:0/416375462 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7feac0105130 0x7feac01a0d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:16:52.792 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T21:16:52.786+0000 7feac59e9700 1 --2- 172.21.15.82:0/416375462 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7feac0104790 0x7feac0199ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:16:52.792 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T21:16:52.786+0000 7feac61ea700 1 -- 172.21.15.82:0/416375462 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7feac0104790 msgr2=0x7feac0199ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:16:52.792 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T21:16:52.786+0000 7feac61ea700 1 --2- 172.21.15.82:0/416375462 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7feac0104790 0x7feac0199ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:16:52.792 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T21:16:52.786+0000 7feac61ea700 1 -- 172.21.15.82:0/416375462 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7feac0105130 msgr2=0x7feac01a0d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:16:52.792 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T21:16:52.786+0000 7feac61ea700 1 --2- 172.21.15.82:0/416375462 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7feac0105130 0x7feac01a0d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:16:52.792 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T21:16:52.787+0000 7feac61ea700 1 -- 172.21.15.82:0/416375462 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feac01a13c0 con 0x7feac01017b0 2024-05-03T21:16:52.792 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T21:16:52.787+0000 7feac69eb700 1 --2- 172.21.15.82:0/416375462 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7feac0105130 0x7feac01a0d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-03T21:16:52.793 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T21:16:52.787+0000 7feac61ea700 1 --2- 172.21.15.82:0/416375462 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7feac01017b0 0x7feac0199760 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7feab0003990 tx=0x7feab000fad0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T21:16:52.793 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T21:16:52.787+0000 7feab77fe700 1 -- 172.21.15.82:0/416375462 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feab0027070 con 0x7feac01017b0 2024-05-03T21:16:52.793 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T21:16:52.787+0000 7feac844e700 1 -- 172.21.15.82:0/416375462 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7feac01a1650 con 0x7feac01017b0 2024-05-03T21:16:52.793 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T21:16:52.787+0000 7feab77fe700 1 -- 172.21.15.82:0/416375462 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(28 keys) ==== 1170+0+0 (secure 0 0 0) 0x7feab0022970 con 0x7feac01017b0 2024-05-03T21:16:52.793 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T21:16:52.787+0000 7feab77fe700 1 -- 172.21.15.82:0/416375462 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feab00176e0 con 0x7feac01017b0 2024-05-03T21:16:52.793 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T21:16:52.787+0000 7feac844e700 1 -- 172.21.15.82:0/416375462 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7feac01a1b00 con 0x7feac01017b0 2024-05-03T21:16:52.794 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T21:16:52.788+0000 7feac844e700 1 -- 172.21.15.82:0/416375462 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7feac006a190 con 0x7feac01017b0 2024-05-03T21:16:52.794 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T21:16:52.789+0000 7feab77fe700 1 -- 172.21.15.82:0/416375462 <== mon.2 v2:172.21.15.82:3300/0 4 ==== mgrmap(e 16) ==== 149652+0+0 (secure 0 0 0) 0x7feab002a070 con 0x7feac01017b0 2024-05-03T21:16:52.794 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T21:16:52.790+0000 7feab77fe700 1 --2- 172.21.15.82:0/416375462 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7feaac0b2900 0x7feaac0b4dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:16:52.794 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T21:16:52.790+0000 7feab77fe700 1 -- 172.21.15.82:0/416375462 <== mon.2 v2:172.21.15.82:3300/0 5 ==== osd_map(64..64 src has 1..64) ==== 6261+0+0 (secure 0 0 0) 0x7feab00e28b0 con 0x7feac01017b0 2024-05-03T21:16:52.794 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T21:16:52.790+0000 7feac59e9700 1 --2- 172.21.15.82:0/416375462 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7feaac0b2900 0x7feaac0b4dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:16:52.794 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T21:16:52.790+0000 7feac59e9700 1 --2- 172.21.15.82:0/416375462 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7feaac0b2900 0x7feaac0b4dc0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7feabc005fd0 tx=0x7feabc005f60 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T21:16:52.797 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T21:16:52.794+0000 7feab77fe700 1 -- 172.21.15.82:0/416375462 <== mon.2 v2:172.21.15.82:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191839 (secure 0 0 0) 0x7feab0013070 con 0x7feac01017b0 2024-05-03T21:16:53.011 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T21:16:53.009+0000 7feac844e700 1 -- 172.21.15.82:0/416375462 --> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi082:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) -- 0x7feac00638e0 con 0x7feaac0b2900 2024-05-03T21:16:53.471 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:16:53 smithi067 ceph-mon[29730]: pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 304 MiB used, 894 GiB / 894 GiB avail 2024-05-03T21:16:53.472 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:16:53 smithi067 ceph-mon[29730]: Detected new or changed devices on smithi067 2024-05-03T21:16:53.472 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:16:53 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:16:53.472 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:16:53 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:16:53.472 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:16:53 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-05-03T21:16:53.472 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:16:53 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-05-03T21:16:53.472 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:16:53 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-05-03T21:16:53.472 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:16:53 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-05-03T21:16:53.472 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:16:53 smithi067 ceph-mon[29730]: Adjusting osd_memory_target on smithi067 to 4272M 2024-05-03T21:16:53.472 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:16:53 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:16:53.472 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:16:53 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T21:16:53.472 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:16:53 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T21:16:53.473 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:16:53 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:16:53.473 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:16:53 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T21:16:53.473 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:16:53 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T21:16:53.473 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:16:53 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T21:16:53.473 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:16:53 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:16:53.473 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:16:53 smithi067 ceph-mon[29730]: osdmap e64: 10 total, 10 up, 10 in 2024-05-03T21:16:53.473 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:16:53 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-03T21:16:53.473 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:16:53 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-03T21:16:53.473 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:16:53 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T21:16:53.498 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:16:53 smithi082 ceph-mon[32933]: pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 304 MiB used, 894 GiB / 894 GiB avail 2024-05-03T21:16:53.499 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:16:53 smithi082 ceph-mon[32933]: Detected new or changed devices on smithi067 2024-05-03T21:16:53.499 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:16:53 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:16:53.499 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:16:53 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:16:53.499 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:16:53 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-05-03T21:16:53.499 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:16:53 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-05-03T21:16:53.499 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:16:53 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-05-03T21:16:53.499 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:16:53 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-05-03T21:16:53.499 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:16:53 smithi082 ceph-mon[32933]: Adjusting osd_memory_target on smithi067 to 4272M 2024-05-03T21:16:53.499 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:16:53 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:16:53.499 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:16:53 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T21:16:53.499 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:16:53 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T21:16:53.499 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:16:53 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:16:53.499 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:16:53 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T21:16:53.500 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:16:53 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T21:16:53.500 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:16:53 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T21:16:53.500 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:16:53 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:16:53.500 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:16:53 smithi082 ceph-mon[32933]: osdmap e64: 10 total, 10 up, 10 in 2024-05-03T21:16:53.500 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:16:53 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-03T21:16:53.500 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:16:53 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-03T21:16:53.500 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:16:53 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T21:16:53.575 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:16:53 smithi138 ceph-mon[33110]: pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 304 MiB used, 894 GiB / 894 GiB avail 2024-05-03T21:16:53.575 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:16:53 smithi138 ceph-mon[33110]: Detected new or changed devices on smithi067 2024-05-03T21:16:53.575 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:16:53 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:16:53.575 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:16:53 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:16:53.575 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:16:53 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-05-03T21:16:53.575 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:16:53 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-05-03T21:16:53.575 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:16:53 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-05-03T21:16:53.575 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:16:53 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-05-03T21:16:53.575 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:16:53 smithi138 ceph-mon[33110]: Adjusting osd_memory_target on smithi067 to 4272M 2024-05-03T21:16:53.575 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:16:53 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:16:53.575 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:16:53 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T21:16:53.575 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:16:53 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T21:16:53.575 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:16:53 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:16:53.575 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:16:53 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T21:16:53.576 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:16:53 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T21:16:53.576 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:16:53 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T21:16:53.576 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:16:53 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:16:53.576 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:16:53 smithi138 ceph-mon[33110]: osdmap e64: 10 total, 10 up, 10 in 2024-05-03T21:16:53.576 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:16:53 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-03T21:16:53.576 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:16:53 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-03T21:16:53.576 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:16:53 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T21:16:54.471 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:16:54 smithi067 ceph-mon[29730]: from='client.24332 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi082:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-05-03T21:16:54.495 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:16:54 smithi082 ceph-mon[32933]: from='client.24332 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi082:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-05-03T21:16:54.574 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:16:54 smithi138 ceph-mon[33110]: from='client.24332 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi082:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-05-03T21:16:55.471 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:16:55 smithi067 ceph-mon[29730]: pgmap v214: 1 pgs: 1 active+clean; 577 KiB data, 304 MiB used, 894 GiB / 894 GiB avail 2024-05-03T21:16:55.495 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:16:55 smithi082 ceph-mon[32933]: pgmap v214: 1 pgs: 1 active+clean; 577 KiB data, 304 MiB used, 894 GiB / 894 GiB avail 2024-05-03T21:16:55.574 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:16:55 smithi138 ceph-mon[33110]: pgmap v214: 1 pgs: 1 active+clean; 577 KiB data, 304 MiB used, 894 GiB / 894 GiB avail 2024-05-03T21:16:56.471 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:16:56 smithi067 ceph-mon[29730]: from='client.? 172.21.15.82:0/2603935845' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "db0305a0-a83b-474b-837a-fbc239ac1738"}]: dispatch 2024-05-03T21:16:56.471 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:16:56 smithi067 ceph-mon[29730]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "db0305a0-a83b-474b-837a-fbc239ac1738"}]: dispatch 2024-05-03T21:16:56.471 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:16:56 smithi067 ceph-mon[29730]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "db0305a0-a83b-474b-837a-fbc239ac1738"}]': finished 2024-05-03T21:16:56.471 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:16:56 smithi067 ceph-mon[29730]: osdmap e65: 11 total, 10 up, 11 in 2024-05-03T21:16:56.471 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:16:56 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-05-03T21:16:56.495 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:16:56 smithi082 ceph-mon[32933]: from='client.? 172.21.15.82:0/2603935845' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "db0305a0-a83b-474b-837a-fbc239ac1738"}]: dispatch 2024-05-03T21:16:56.495 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:16:56 smithi082 ceph-mon[32933]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "db0305a0-a83b-474b-837a-fbc239ac1738"}]: dispatch 2024-05-03T21:16:56.495 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:16:56 smithi082 ceph-mon[32933]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "db0305a0-a83b-474b-837a-fbc239ac1738"}]': finished 2024-05-03T21:16:56.495 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:16:56 smithi082 ceph-mon[32933]: osdmap e65: 11 total, 10 up, 11 in 2024-05-03T21:16:56.496 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:16:56 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-05-03T21:16:56.574 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:16:56 smithi138 ceph-mon[33110]: from='client.? 172.21.15.82:0/2603935845' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "db0305a0-a83b-474b-837a-fbc239ac1738"}]: dispatch 2024-05-03T21:16:56.574 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:16:56 smithi138 ceph-mon[33110]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "db0305a0-a83b-474b-837a-fbc239ac1738"}]: dispatch 2024-05-03T21:16:56.574 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:16:56 smithi138 ceph-mon[33110]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "db0305a0-a83b-474b-837a-fbc239ac1738"}]': finished 2024-05-03T21:16:56.575 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:16:56 smithi138 ceph-mon[33110]: osdmap e65: 11 total, 10 up, 11 in 2024-05-03T21:16:56.575 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:16:56 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-05-03T21:16:57.495 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:16:57 smithi082 ceph-mon[32933]: pgmap v216: 1 pgs: 1 active+clean; 577 KiB data, 304 MiB used, 894 GiB / 894 GiB avail 2024-05-03T21:16:57.495 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:16:57 smithi082 ceph-mon[32933]: from='client.? 172.21.15.82:0/1532715998' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-03T21:16:57.574 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:16:57 smithi138 ceph-mon[33110]: pgmap v216: 1 pgs: 1 active+clean; 577 KiB data, 304 MiB used, 894 GiB / 894 GiB avail 2024-05-03T21:16:57.574 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:16:57 smithi138 ceph-mon[33110]: from='client.? 172.21.15.82:0/1532715998' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-03T21:16:57.636 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:16:57 smithi067 ceph-mon[29730]: pgmap v216: 1 pgs: 1 active+clean; 577 KiB data, 304 MiB used, 894 GiB / 894 GiB avail 2024-05-03T21:16:57.636 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:16:57 smithi067 ceph-mon[29730]: from='client.? 172.21.15.82:0/1532715998' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-03T21:16:59.495 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:16:59 smithi082 ceph-mon[32933]: pgmap v217: 1 pgs: 1 active+clean; 577 KiB data, 304 MiB used, 894 GiB / 894 GiB avail 2024-05-03T21:16:59.574 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:16:59 smithi138 ceph-mon[33110]: pgmap v217: 1 pgs: 1 active+clean; 577 KiB data, 304 MiB used, 894 GiB / 894 GiB avail 2024-05-03T21:16:59.721 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:16:59 smithi067 ceph-mon[29730]: pgmap v217: 1 pgs: 1 active+clean; 577 KiB data, 304 MiB used, 894 GiB / 894 GiB avail 2024-05-03T21:17:01.495 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:17:01 smithi082 ceph-mon[32933]: pgmap v218: 1 pgs: 1 active+clean; 577 KiB data, 304 MiB used, 894 GiB / 894 GiB avail 2024-05-03T21:17:01.574 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:17:01 smithi138 ceph-mon[33110]: pgmap v218: 1 pgs: 1 active+clean; 577 KiB data, 304 MiB used, 894 GiB / 894 GiB avail 2024-05-03T21:17:01.721 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:17:01 smithi067 ceph-mon[29730]: pgmap v218: 1 pgs: 1 active+clean; 577 KiB data, 304 MiB used, 894 GiB / 894 GiB avail 2024-05-03T21:17:03.495 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:17:03 smithi082 ceph-mon[32933]: pgmap v219: 1 pgs: 1 active+clean; 577 KiB data, 304 MiB used, 894 GiB / 894 GiB avail 2024-05-03T21:17:03.495 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:17:03 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-05-03T21:17:03.495 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:17:03 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T21:17:03.574 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:17:03 smithi138 ceph-mon[33110]: pgmap v219: 1 pgs: 1 active+clean; 577 KiB data, 304 MiB used, 894 GiB / 894 GiB avail 2024-05-03T21:17:03.574 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:17:03 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-05-03T21:17:03.575 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:17:03 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T21:17:03.721 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:17:03 smithi067 ceph-mon[29730]: pgmap v219: 1 pgs: 1 active+clean; 577 KiB data, 304 MiB used, 894 GiB / 894 GiB avail 2024-05-03T21:17:03.721 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:17:03 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-05-03T21:17:03.721 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:17:03 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T21:17:04.388 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:17:04 smithi082 ceph-mon[32933]: Deploying daemon osd.10 on smithi082 2024-05-03T21:17:04.574 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:17:04 smithi138 ceph-mon[33110]: Deploying daemon osd.10 on smithi082 2024-05-03T21:17:04.721 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:17:04 smithi067 ceph-mon[29730]: Deploying daemon osd.10 on smithi082 2024-05-03T21:17:05.495 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:17:05 smithi082 ceph-mon[32933]: pgmap v220: 1 pgs: 1 active+clean; 577 KiB data, 304 MiB used, 894 GiB / 894 GiB avail 2024-05-03T21:17:05.574 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:17:05 smithi138 ceph-mon[33110]: pgmap v220: 1 pgs: 1 active+clean; 577 KiB data, 304 MiB used, 894 GiB / 894 GiB avail 2024-05-03T21:17:05.721 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:17:05 smithi067 ceph-mon[29730]: pgmap v220: 1 pgs: 1 active+clean; 577 KiB data, 304 MiB used, 894 GiB / 894 GiB avail 2024-05-03T21:17:07.471 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:17:07 smithi067 ceph-mon[29730]: pgmap v221: 1 pgs: 1 active+clean; 577 KiB data, 304 MiB used, 894 GiB / 894 GiB avail 2024-05-03T21:17:07.495 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:17:07 smithi082 ceph-mon[32933]: pgmap v221: 1 pgs: 1 active+clean; 577 KiB data, 304 MiB used, 894 GiB / 894 GiB avail 2024-05-03T21:17:07.574 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:17:07 smithi138 ceph-mon[33110]: pgmap v221: 1 pgs: 1 active+clean; 577 KiB data, 304 MiB used, 894 GiB / 894 GiB avail 2024-05-03T21:17:09.343 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:17:09 smithi082 ceph-mon[32933]: pgmap v222: 1 pgs: 1 active+clean; 577 KiB data, 304 MiB used, 894 GiB / 894 GiB avail 2024-05-03T21:17:09.343 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:17:09 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:17:09.343 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:17:09 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:17:09.343 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:17:09 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T21:17:09.574 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:17:09 smithi138 ceph-mon[33110]: pgmap v222: 1 pgs: 1 active+clean; 577 KiB data, 304 MiB used, 894 GiB / 894 GiB avail 2024-05-03T21:17:09.574 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:17:09 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:17:09.575 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:17:09 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:17:09.575 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:17:09 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T21:17:09.721 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:17:09 smithi067 ceph-mon[29730]: pgmap v222: 1 pgs: 1 active+clean; 577 KiB data, 304 MiB used, 894 GiB / 894 GiB avail 2024-05-03T21:17:09.721 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:17:09 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:17:09.721 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:17:09 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:17:09.721 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:17:09 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T21:17:11.574 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:17:11 smithi138 ceph-mon[33110]: pgmap v223: 1 pgs: 1 active+clean; 577 KiB data, 304 MiB used, 894 GiB / 894 GiB avail 2024-05-03T21:17:11.681 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:17:11 smithi082 ceph-mon[32933]: pgmap v223: 1 pgs: 1 active+clean; 577 KiB data, 304 MiB used, 894 GiB / 894 GiB avail 2024-05-03T21:17:11.721 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:17:11 smithi067 ceph-mon[29730]: pgmap v223: 1 pgs: 1 active+clean; 577 KiB data, 304 MiB used, 894 GiB / 894 GiB avail 2024-05-03T21:17:12.026 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T21:17:12.025+0000 7feab77fe700 1 -- 172.21.15.82:0/416375462 <== mgr.14150 v2:172.21.15.67:6800/604451592 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+38 (secure 0 0 0) 0x7feac00638e0 con 0x7feaac0b2900 2024-05-03T21:17:12.027 INFO:teuthology.orchestra.run.smithi082.stdout:Created osd(s) 10 on host 'smithi082' 2024-05-03T21:17:12.029 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T21:17:12.027+0000 7feac844e700 1 -- 172.21.15.82:0/416375462 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7feaac0b2900 msgr2=0x7feaac0b4dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:17:12.029 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T21:17:12.027+0000 7feac844e700 1 --2- 172.21.15.82:0/416375462 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7feaac0b2900 0x7feaac0b4dc0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7feabc005fd0 tx=0x7feabc005f60 comp rx=0 tx=0).stop 2024-05-03T21:17:12.029 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T21:17:12.027+0000 7feac844e700 1 -- 172.21.15.82:0/416375462 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7feac01017b0 msgr2=0x7feac0199760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:17:12.029 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T21:17:12.027+0000 7feac844e700 1 --2- 172.21.15.82:0/416375462 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7feac01017b0 0x7feac0199760 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7feab0003990 tx=0x7feab000fad0 comp rx=0 tx=0).stop 2024-05-03T21:17:12.029 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T21:17:12.027+0000 7feac844e700 1 -- 172.21.15.82:0/416375462 shutdown_connections 2024-05-03T21:17:12.029 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T21:17:12.027+0000 7feac844e700 1 --2- 172.21.15.82:0/416375462 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7feaac0b2900 0x7feaac0b4dc0 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:17:12.029 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T21:17:12.027+0000 7feac844e700 1 --2- 172.21.15.82:0/416375462 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7feac0105130 0x7feac01a0d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:17:12.029 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T21:17:12.027+0000 7feac844e700 1 --2- 172.21.15.82:0/416375462 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7feac01017b0 0x7feac0199760 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:17:12.029 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T21:17:12.027+0000 7feac844e700 1 --2- 172.21.15.82:0/416375462 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7feac0104790 0x7feac0199ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:17:12.029 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T21:17:12.027+0000 7feac844e700 1 -- 172.21.15.82:0/416375462 >> 172.21.15.82:0/416375462 conn(0x7feac00fd130 msgr2=0x7feac0107a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T21:17:12.029 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T21:17:12.027+0000 7feac844e700 1 -- 172.21.15.82:0/416375462 shutdown_connections 2024-05-03T21:17:12.030 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T21:17:12.027+0000 7feac844e700 1 -- 172.21.15.82:0/416375462 wait complete. 2024-05-03T21:17:12.574 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:17:12 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:17:12.575 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:17:12 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:17:12.575 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:17:12 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T21:17:12.575 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:17:12 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T21:17:12.575 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:17:12 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:17:12.575 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:17:12 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:17:12.575 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:17:12 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:17:12.575 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:17:12 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T21:17:12.618 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:17:12 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:17:12.619 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:17:12 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:17:12.619 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:17:12 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T21:17:12.619 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:17:12 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T21:17:12.619 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:17:12 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:17:12.619 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:17:12 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:17:12.619 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:17:12 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:17:12.619 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:17:12 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T21:17:12.649 DEBUG:teuthology.orchestra.run.smithi082:osd.10> sudo journalctl -f -n 0 -u ceph-713bc57a-0991-11ef-bc97-c7b262605968@osd.10.service 2024-05-03T21:17:12.652 INFO:tasks.cephadm:Deploying osd.11 on smithi138 with /dev/vg_nvme/lv_1... 2024-05-03T21:17:12.652 DEBUG:teuthology.orchestra.run.smithi138:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9a047fe7599a549042d06a3fb5b6813a203eabea shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 713bc57a-0991-11ef-bc97-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-05-03T21:17:12.721 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:17:12 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:17:12.721 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:17:12 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:17:12.721 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:17:12 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T21:17:12.721 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:17:12 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T21:17:12.721 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:17:12 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:17:12.721 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:17:12 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:17:12.721 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:17:12 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:17:12.722 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:17:12 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T21:17:12.895 INFO:teuthology.orchestra.run.smithi138.stderr:Inferring config /var/lib/ceph/713bc57a-0991-11ef-bc97-c7b262605968/mon.c/config 2024-05-03T21:17:13.574 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:17:13 smithi138 ceph-mon[33110]: pgmap v224: 1 pgs: 1 active+clean; 577 KiB data, 304 MiB used, 894 GiB / 894 GiB avail 2024-05-03T21:17:13.575 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:17:13 smithi138 ceph-mon[33110]: from='osd.10 [v2:172.21.15.82:6824/2505507029,v1:172.21.15.82:6825/2505507029]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-05-03T21:17:13.575 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:17:13 smithi138 ceph-mon[33110]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-05-03T21:17:13.721 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:17:13 smithi067 ceph-mon[29730]: pgmap v224: 1 pgs: 1 active+clean; 577 KiB data, 304 MiB used, 894 GiB / 894 GiB avail 2024-05-03T21:17:13.721 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:17:13 smithi067 ceph-mon[29730]: from='osd.10 [v2:172.21.15.82:6824/2505507029,v1:172.21.15.82:6825/2505507029]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-05-03T21:17:13.721 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:17:13 smithi067 ceph-mon[29730]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-05-03T21:17:13.733 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:17:13 smithi082 ceph-mon[32933]: pgmap v224: 1 pgs: 1 active+clean; 577 KiB data, 304 MiB used, 894 GiB / 894 GiB avail 2024-05-03T21:17:13.734 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:17:13 smithi082 ceph-mon[32933]: from='osd.10 [v2:172.21.15.82:6824/2505507029,v1:172.21.15.82:6825/2505507029]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-05-03T21:17:13.734 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:17:13 smithi082 ceph-mon[32933]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-05-03T21:17:14.575 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:17:14 smithi138 ceph-mon[33110]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-05-03T21:17:14.575 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:17:14 smithi138 ceph-mon[33110]: from='osd.10 [v2:172.21.15.82:6824/2505507029,v1:172.21.15.82:6825/2505507029]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi082", "root=default"]}]: dispatch 2024-05-03T21:17:14.575 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:17:14 smithi138 ceph-mon[33110]: osdmap e66: 11 total, 10 up, 11 in 2024-05-03T21:17:14.575 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:17:14 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-05-03T21:17:14.575 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:17:14 smithi138 ceph-mon[33110]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi082", "root=default"]}]: dispatch 2024-05-03T21:17:14.575 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:17:14 smithi138 ceph-mon[33110]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi082", "root=default"]}]': finished 2024-05-03T21:17:14.575 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:17:14 smithi138 ceph-mon[33110]: osdmap e67: 11 total, 10 up, 11 in 2024-05-03T21:17:14.575 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:17:14 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-05-03T21:17:14.677 INFO:teuthology.orchestra.run.smithi138.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-05-03T21:17:14.677 INFO:teuthology.orchestra.run.smithi138.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-05-03T21:17:14.677 INFO:teuthology.orchestra.run.smithi138.stderr: stderr: 10+0 records in 2024-05-03T21:17:14.677 INFO:teuthology.orchestra.run.smithi138.stderr:10+0 records out 2024-05-03T21:17:14.677 INFO:teuthology.orchestra.run.smithi138.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.012877 s, 814 MB/s 2024-05-03T21:17:14.677 INFO:teuthology.orchestra.run.smithi138.stderr:--> Zapping successful for: 2024-05-03T21:17:14.721 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:17:14 smithi067 ceph-mon[29730]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-05-03T21:17:14.721 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:17:14 smithi067 ceph-mon[29730]: from='osd.10 [v2:172.21.15.82:6824/2505507029,v1:172.21.15.82:6825/2505507029]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi082", "root=default"]}]: dispatch 2024-05-03T21:17:14.721 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:17:14 smithi067 ceph-mon[29730]: osdmap e66: 11 total, 10 up, 11 in 2024-05-03T21:17:14.721 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:17:14 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-05-03T21:17:14.721 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:17:14 smithi067 ceph-mon[29730]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi082", "root=default"]}]: dispatch 2024-05-03T21:17:14.721 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:17:14 smithi067 ceph-mon[29730]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi082", "root=default"]}]': finished 2024-05-03T21:17:14.721 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:17:14 smithi067 ceph-mon[29730]: osdmap e67: 11 total, 10 up, 11 in 2024-05-03T21:17:14.722 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:17:14 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-05-03T21:17:14.745 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:17:14 smithi082 ceph-mon[32933]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-05-03T21:17:14.746 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:17:14 smithi082 ceph-mon[32933]: from='osd.10 [v2:172.21.15.82:6824/2505507029,v1:172.21.15.82:6825/2505507029]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi082", "root=default"]}]: dispatch 2024-05-03T21:17:14.746 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:17:14 smithi082 ceph-mon[32933]: osdmap e66: 11 total, 10 up, 11 in 2024-05-03T21:17:14.746 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:17:14 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-05-03T21:17:14.746 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:17:14 smithi082 ceph-mon[32933]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi082", "root=default"]}]: dispatch 2024-05-03T21:17:14.746 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:17:14 smithi082 ceph-mon[32933]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi082", "root=default"]}]': finished 2024-05-03T21:17:14.746 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:17:14 smithi082 ceph-mon[32933]: osdmap e67: 11 total, 10 up, 11 in 2024-05-03T21:17:14.746 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:17:14 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-05-03T21:17:14.746 INFO:journalctl@ceph.osd.10.smithi082.stdout:May 03 21:17:14 smithi082 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-10[57676]: 2024-05-03T21:17:14.287+0000 7f790e276700 -1 osd.10 0 waiting for initial osdmap 2024-05-03T21:17:14.746 INFO:journalctl@ceph.osd.10.smithi082.stdout:May 03 21:17:14 smithi082 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-10[57676]: 2024-05-03T21:17:14.301+0000 7f790a069700 -1 osd.10 67 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-05-03T21:17:15.335 DEBUG:teuthology.orchestra.run.smithi138:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9a047fe7599a549042d06a3fb5b6813a203eabea shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 713bc57a-0991-11ef-bc97-c7b262605968 -- ceph orch daemon add osd smithi138:vg_nvme/lv_1 2024-05-03T21:17:15.490 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:17:15 smithi082 ceph-mon[32933]: purged_snaps scrub starts 2024-05-03T21:17:15.491 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:17:15 smithi082 ceph-mon[32933]: purged_snaps scrub ok 2024-05-03T21:17:15.491 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:17:15 smithi082 ceph-mon[32933]: pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 304 MiB used, 894 GiB / 894 GiB avail 2024-05-03T21:17:15.491 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:17:15 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-05-03T21:17:15.491 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:17:15 smithi082 ceph-mon[32933]: osd.10 [v2:172.21.15.82:6824/2505507029,v1:172.21.15.82:6825/2505507029] boot 2024-05-03T21:17:15.491 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:17:15 smithi082 ceph-mon[32933]: osdmap e68: 11 total, 11 up, 11 in 2024-05-03T21:17:15.491 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:17:15 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-05-03T21:17:15.571 INFO:teuthology.orchestra.run.smithi138.stderr:Inferring config /var/lib/ceph/713bc57a-0991-11ef-bc97-c7b262605968/mon.c/config 2024-05-03T21:17:15.574 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:17:15 smithi138 ceph-mon[33110]: purged_snaps scrub starts 2024-05-03T21:17:15.574 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:17:15 smithi138 ceph-mon[33110]: purged_snaps scrub ok 2024-05-03T21:17:15.574 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:17:15 smithi138 ceph-mon[33110]: pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 304 MiB used, 894 GiB / 894 GiB avail 2024-05-03T21:17:15.574 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:17:15 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-05-03T21:17:15.574 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:17:15 smithi138 ceph-mon[33110]: osd.10 [v2:172.21.15.82:6824/2505507029,v1:172.21.15.82:6825/2505507029] boot 2024-05-03T21:17:15.574 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:17:15 smithi138 ceph-mon[33110]: osdmap e68: 11 total, 11 up, 11 in 2024-05-03T21:17:15.574 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:17:15 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-05-03T21:17:15.721 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:17:15 smithi067 ceph-mon[29730]: purged_snaps scrub starts 2024-05-03T21:17:15.722 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:17:15 smithi067 ceph-mon[29730]: purged_snaps scrub ok 2024-05-03T21:17:15.722 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:17:15 smithi067 ceph-mon[29730]: pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 304 MiB used, 894 GiB / 894 GiB avail 2024-05-03T21:17:15.722 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:17:15 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-05-03T21:17:15.722 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:17:15 smithi067 ceph-mon[29730]: osd.10 [v2:172.21.15.82:6824/2505507029,v1:172.21.15.82:6825/2505507029] boot 2024-05-03T21:17:15.722 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:17:15 smithi067 ceph-mon[29730]: osdmap e68: 11 total, 11 up, 11 in 2024-05-03T21:17:15.722 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:17:15 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-05-03T21:17:16.622 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:17:16 smithi138 ceph-mon[33110]: Detected new or changed devices on smithi082 2024-05-03T21:17:16.622 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:17:16 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:17:16.622 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:17:16 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:17:16.622 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:17:16 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-05-03T21:17:16.622 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:17:16 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-05-03T21:17:16.622 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:17:16 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-05-03T21:17:16.622 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:17:16 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-05-03T21:17:16.623 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:17:16 smithi138 ceph-mon[33110]: Adjusting osd_memory_target on smithi082 to 4272M 2024-05-03T21:17:16.623 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:17:16 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:17:16.623 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:17:16 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T21:17:16.623 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:17:16 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T21:17:16.623 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:17:16 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:17:16.623 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:17:16 smithi138 ceph-mon[33110]: pgmap v229: 1 pgs: 1 peering; 577 KiB data, 332 MiB used, 983 GiB / 983 GiB avail 2024-05-03T21:17:16.623 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:17:16 smithi138 ceph-mon[33110]: osdmap e69: 11 total, 11 up, 11 in 2024-05-03T21:17:16.914 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:17:16 smithi082 ceph-mon[32933]: Detected new or changed devices on smithi082 2024-05-03T21:17:16.914 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:17:16 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:17:16.915 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:17:16 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:17:16.915 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:17:16 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-05-03T21:17:16.915 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:17:16 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-05-03T21:17:16.915 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:17:16 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-05-03T21:17:16.915 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:17:16 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-05-03T21:17:16.915 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:17:16 smithi082 ceph-mon[32933]: Adjusting osd_memory_target on smithi082 to 4272M 2024-05-03T21:17:16.915 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:17:16 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:17:16.915 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:17:16 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T21:17:16.915 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:17:16 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T21:17:16.915 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:17:16 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:17:16.915 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:17:16 smithi082 ceph-mon[32933]: pgmap v229: 1 pgs: 1 peering; 577 KiB data, 332 MiB used, 983 GiB / 983 GiB avail 2024-05-03T21:17:16.915 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:17:16 smithi082 ceph-mon[32933]: osdmap e69: 11 total, 11 up, 11 in 2024-05-03T21:17:16.971 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:17:16 smithi067 ceph-mon[29730]: Detected new or changed devices on smithi082 2024-05-03T21:17:16.971 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:17:16 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:17:16.971 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:17:16 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:17:16.971 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:17:16 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-05-03T21:17:16.971 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:17:16 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-05-03T21:17:16.971 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:17:16 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-05-03T21:17:16.971 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:17:16 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-05-03T21:17:16.971 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:17:16 smithi067 ceph-mon[29730]: Adjusting osd_memory_target on smithi082 to 4272M 2024-05-03T21:17:16.971 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:17:16 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:17:16.972 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:17:16 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T21:17:16.972 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:17:16 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T21:17:16.972 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:17:16 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:17:16.972 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:17:16 smithi067 ceph-mon[29730]: pgmap v229: 1 pgs: 1 peering; 577 KiB data, 332 MiB used, 983 GiB / 983 GiB avail 2024-05-03T21:17:16.972 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:17:16 smithi067 ceph-mon[29730]: osdmap e69: 11 total, 11 up, 11 in 2024-05-03T21:17:17.148 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:17:17.144+0000 7f7a7781e700 1 -- 172.21.15.138:0/83120914 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f7a70102dd0 msgr2=0x7f7a70103230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:17:17.148 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:17:17.144+0000 7f7a7781e700 1 --2- 172.21.15.138:0/83120914 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f7a70102dd0 0x7f7a70103230 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f7a6c009990 tx=0x7f7a6c009ca0 comp rx=0 tx=0).stop 2024-05-03T21:17:17.148 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:17:17.144+0000 7f7a7781e700 1 -- 172.21.15.138:0/83120914 shutdown_connections 2024-05-03T21:17:17.148 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:17:17.144+0000 7f7a7781e700 1 --2- 172.21.15.138:0/83120914 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7a70103770 0x7f7a7010a020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:17:17.149 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:17:17.144+0000 7f7a7781e700 1 --2- 172.21.15.138:0/83120914 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f7a70101bd0 0x7f7a70101ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:17:17.149 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:17:17.144+0000 7f7a7781e700 1 --2- 172.21.15.138:0/83120914 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f7a70102dd0 0x7f7a70103230 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:17:17.149 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:17:17.144+0000 7f7a7781e700 1 -- 172.21.15.138:0/83120914 >> 172.21.15.138:0/83120914 conn(0x7f7a700fd170 msgr2=0x7f7a700ff5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T21:17:17.149 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:17:17.145+0000 7f7a7781e700 1 -- 172.21.15.138:0/83120914 shutdown_connections 2024-05-03T21:17:17.149 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:17:17.145+0000 7f7a7781e700 1 -- 172.21.15.138:0/83120914 wait complete. 2024-05-03T21:17:17.149 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:17:17.145+0000 7f7a7781e700 1 Processor -- start 2024-05-03T21:17:17.149 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:17:17.145+0000 7f7a7781e700 1 -- start start 2024-05-03T21:17:17.149 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:17:17.145+0000 7f7a7781e700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f7a70101bd0 0x7f7a701997c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:17:17.149 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:17:17.145+0000 7f7a7781e700 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f7a70102dd0 0x7f7a70199d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:17:17.149 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:17:17.145+0000 7f7a7781e700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7a70103770 0x7f7a701a0d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:17:17.149 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:17:17.145+0000 7f7a7781e700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f7a7010f2f0 con 0x7f7a70103770 2024-05-03T21:17:17.149 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:17:17.145+0000 7f7a7781e700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f7a7010f170 con 0x7f7a70101bd0 2024-05-03T21:17:17.149 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:17:17.145+0000 7f7a7781e700 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f7a7010f470 con 0x7f7a70102dd0 2024-05-03T21:17:17.150 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:17:17.145+0000 7f7a74db9700 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f7a70102dd0 0x7f7a70199d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:17:17.150 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:17:17.146+0000 7f7a74db9700 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f7a70102dd0 0x7f7a70199d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3300/0 says I am v2:172.21.15.138:46904/0 (socket says 172.21.15.138:46904) 2024-05-03T21:17:17.150 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:17:17.146+0000 7f7a74db9700 1 -- 172.21.15.138:0/2536707470 learned_addr learned my addr 172.21.15.138:0/2536707470 (peer_addr_for_me v2:172.21.15.138:0/0) 2024-05-03T21:17:17.150 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:17:17.146+0000 7f7a755ba700 1 --2- 172.21.15.138:0/2536707470 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f7a70101bd0 0x7f7a701997c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:17:17.150 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:17:17.146+0000 7f7a75dbb700 1 --2- 172.21.15.138:0/2536707470 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7a70103770 0x7f7a701a0d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:17:17.150 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:17:17.146+0000 7f7a74db9700 1 -- 172.21.15.138:0/2536707470 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f7a70101bd0 msgr2=0x7f7a701997c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:17:17.150 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:17:17.146+0000 7f7a74db9700 1 --2- 172.21.15.138:0/2536707470 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f7a70101bd0 0x7f7a701997c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:17:17.150 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:17:17.146+0000 7f7a74db9700 1 -- 172.21.15.138:0/2536707470 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7a70103770 msgr2=0x7f7a701a0d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:17:17.150 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:17:17.146+0000 7f7a74db9700 1 --2- 172.21.15.138:0/2536707470 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7a70103770 0x7f7a701a0d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:17:17.150 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:17:17.146+0000 7f7a74db9700 1 -- 172.21.15.138:0/2536707470 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7a701a1420 con 0x7f7a70102dd0 2024-05-03T21:17:17.150 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:17:17.146+0000 7f7a74db9700 1 --2- 172.21.15.138:0/2536707470 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f7a70102dd0 0x7f7a70199d00 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f7a6c00f970 tx=0x7f7a6c00f9a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T21:17:17.150 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:17:17.146+0000 7f7a627fc700 1 -- 172.21.15.138:0/2536707470 <== mon.1 v2:172.21.15.138:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a6c027070 con 0x7f7a70102dd0 2024-05-03T21:17:17.150 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:17:17.146+0000 7f7a7781e700 1 -- 172.21.15.138:0/2536707470 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7a701a16b0 con 0x7f7a70102dd0 2024-05-03T21:17:17.151 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:17:17.146+0000 7f7a627fc700 1 -- 172.21.15.138:0/2536707470 <== mon.1 v2:172.21.15.138:3300/0 2 ==== config(28 keys) ==== 1170+0+0 (secure 0 0 0) 0x7f7a6c022970 con 0x7f7a70102dd0 2024-05-03T21:17:17.151 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:17:17.146+0000 7f7a627fc700 1 -- 172.21.15.138:0/2536707470 <== mon.1 v2:172.21.15.138:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a6c0176e0 con 0x7f7a70102dd0 2024-05-03T21:17:17.151 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:17:17.146+0000 7f7a7781e700 1 -- 172.21.15.138:0/2536707470 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7a701a1b60 con 0x7f7a70102dd0 2024-05-03T21:17:17.152 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:17:17.147+0000 7f7a7781e700 1 -- 172.21.15.138:0/2536707470 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7a7006a190 con 0x7f7a70102dd0 2024-05-03T21:17:17.152 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:17:17.148+0000 7f7a627fc700 1 -- 172.21.15.138:0/2536707470 <== mon.1 v2:172.21.15.138:3300/0 4 ==== mgrmap(e 16) ==== 149652+0+0 (secure 0 0 0) 0x7f7a6c02a070 con 0x7f7a70102dd0 2024-05-03T21:17:17.152 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:17:17.149+0000 7f7a627fc700 1 --2- 172.21.15.138:0/2536707470 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f7a5c0b2880 0x7f7a5c0b4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:17:17.152 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:17:17.149+0000 7f7a627fc700 1 -- 172.21.15.138:0/2536707470 <== mon.1 v2:172.21.15.138:3300/0 5 ==== osd_map(69..69 src has 1..69) ==== 6694+0+0 (secure 0 0 0) 0x7f7a6c0e29f0 con 0x7f7a70102dd0 2024-05-03T21:17:17.152 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:17:17.149+0000 7f7a755ba700 1 --2- 172.21.15.138:0/2536707470 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f7a5c0b2880 0x7f7a5c0b4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:17:17.152 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:17:17.150+0000 7f7a755ba700 1 --2- 172.21.15.138:0/2536707470 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f7a5c0b2880 0x7f7a5c0b4d40 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f7a70102c30 tx=0x7f7a6400b410 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T21:17:17.154 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:17:17.153+0000 7f7a627fc700 1 -- 172.21.15.138:0/2536707470 <== mon.1 v2:172.21.15.138:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191839 (secure 0 0 0) 0x7f7a6c013070 con 0x7f7a70102dd0 2024-05-03T21:17:17.371 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:17:17.369+0000 7f7a7781e700 1 -- 172.21.15.138:0/2536707470 --> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi138:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) -- 0x7f7a700638e0 con 0x7f7a5c0b2880 2024-05-03T21:17:18.573 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:17:18 smithi138 ceph-mon[33110]: osdmap e70: 11 total, 11 up, 11 in 2024-05-03T21:17:18.573 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:17:18 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-03T21:17:18.573 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:17:18 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-03T21:17:18.574 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:17:18 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T21:17:18.721 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:17:18 smithi067 ceph-mon[29730]: osdmap e70: 11 total, 11 up, 11 in 2024-05-03T21:17:18.721 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:17:18 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-03T21:17:18.721 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:17:18 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-03T21:17:18.721 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:17:18 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T21:17:18.746 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:17:18 smithi082 ceph-mon[32933]: osdmap e70: 11 total, 11 up, 11 in 2024-05-03T21:17:18.746 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:17:18 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-03T21:17:18.746 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:17:18 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-03T21:17:18.746 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:17:18 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T21:17:19.575 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:17:19 smithi138 ceph-mon[33110]: from='client.24322 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi138:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-05-03T21:17:19.575 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:17:19 smithi138 ceph-mon[33110]: pgmap v232: 1 pgs: 1 peering; 577 KiB data, 332 MiB used, 983 GiB / 983 GiB avail 2024-05-03T21:17:19.721 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:17:19 smithi067 ceph-mon[29730]: from='client.24322 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi138:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-05-03T21:17:19.721 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:17:19 smithi067 ceph-mon[29730]: pgmap v232: 1 pgs: 1 peering; 577 KiB data, 332 MiB used, 983 GiB / 983 GiB avail 2024-05-03T21:17:19.745 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:17:19 smithi082 ceph-mon[32933]: from='client.24322 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi138:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-05-03T21:17:19.745 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:17:19 smithi082 ceph-mon[32933]: pgmap v232: 1 pgs: 1 peering; 577 KiB data, 332 MiB used, 983 GiB / 983 GiB avail 2024-05-03T21:17:20.574 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:17:20 smithi138 ceph-mon[33110]: from='client.? 172.21.15.138:0/1848224307' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "479c2747-f28e-4704-87e4-d0c31782e2de"}]: dispatch 2024-05-03T21:17:20.574 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:17:20 smithi138 ceph-mon[33110]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "479c2747-f28e-4704-87e4-d0c31782e2de"}]: dispatch 2024-05-03T21:17:20.575 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:17:20 smithi138 ceph-mon[33110]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "479c2747-f28e-4704-87e4-d0c31782e2de"}]': finished 2024-05-03T21:17:20.575 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:17:20 smithi138 ceph-mon[33110]: osdmap e71: 12 total, 11 up, 12 in 2024-05-03T21:17:20.575 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:17:20 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-05-03T21:17:20.721 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:17:20 smithi067 ceph-mon[29730]: from='client.? 172.21.15.138:0/1848224307' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "479c2747-f28e-4704-87e4-d0c31782e2de"}]: dispatch 2024-05-03T21:17:20.721 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:17:20 smithi067 ceph-mon[29730]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "479c2747-f28e-4704-87e4-d0c31782e2de"}]: dispatch 2024-05-03T21:17:20.721 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:17:20 smithi067 ceph-mon[29730]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "479c2747-f28e-4704-87e4-d0c31782e2de"}]': finished 2024-05-03T21:17:20.721 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:17:20 smithi067 ceph-mon[29730]: osdmap e71: 12 total, 11 up, 12 in 2024-05-03T21:17:20.721 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:17:20 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-05-03T21:17:20.745 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:17:20 smithi082 ceph-mon[32933]: from='client.? 172.21.15.138:0/1848224307' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "479c2747-f28e-4704-87e4-d0c31782e2de"}]: dispatch 2024-05-03T21:17:20.745 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:17:20 smithi082 ceph-mon[32933]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "479c2747-f28e-4704-87e4-d0c31782e2de"}]: dispatch 2024-05-03T21:17:20.745 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:17:20 smithi082 ceph-mon[32933]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "479c2747-f28e-4704-87e4-d0c31782e2de"}]': finished 2024-05-03T21:17:20.745 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:17:20 smithi082 ceph-mon[32933]: osdmap e71: 12 total, 11 up, 12 in 2024-05-03T21:17:20.745 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:17:20 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-05-03T21:17:21.574 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:17:21 smithi138 ceph-mon[33110]: pgmap v233: 1 pgs: 1 peering; 577 KiB data, 332 MiB used, 983 GiB / 983 GiB avail 2024-05-03T21:17:21.575 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:17:21 smithi138 ceph-mon[33110]: from='client.? 172.21.15.138:0/954510659' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-03T21:17:21.721 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:17:21 smithi067 ceph-mon[29730]: pgmap v233: 1 pgs: 1 peering; 577 KiB data, 332 MiB used, 983 GiB / 983 GiB avail 2024-05-03T21:17:21.721 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:17:21 smithi067 ceph-mon[29730]: from='client.? 172.21.15.138:0/954510659' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-03T21:17:21.745 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:17:21 smithi082 ceph-mon[32933]: pgmap v233: 1 pgs: 1 peering; 577 KiB data, 332 MiB used, 983 GiB / 983 GiB avail 2024-05-03T21:17:21.745 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:17:21 smithi082 ceph-mon[32933]: from='client.? 172.21.15.138:0/954510659' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-03T21:17:23.577 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:17:23 smithi138 ceph-mon[33110]: pgmap v235: 1 pgs: 1 active+clean; 577 KiB data, 332 MiB used, 983 GiB / 983 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-05-03T21:17:23.721 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:17:23 smithi067 ceph-mon[29730]: pgmap v235: 1 pgs: 1 active+clean; 577 KiB data, 332 MiB used, 983 GiB / 983 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-05-03T21:17:23.745 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:17:23 smithi082 ceph-mon[32933]: pgmap v235: 1 pgs: 1 active+clean; 577 KiB data, 332 MiB used, 983 GiB / 983 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-05-03T21:17:25.574 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:17:25 smithi138 ceph-mon[33110]: pgmap v236: 1 pgs: 1 active+clean; 577 KiB data, 332 MiB used, 983 GiB / 983 GiB avail; 77 KiB/s, 0 objects/s recovering 2024-05-03T21:17:25.721 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:17:25 smithi067 ceph-mon[29730]: pgmap v236: 1 pgs: 1 active+clean; 577 KiB data, 332 MiB used, 983 GiB / 983 GiB avail; 77 KiB/s, 0 objects/s recovering 2024-05-03T21:17:25.745 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:17:25 smithi082 ceph-mon[32933]: pgmap v236: 1 pgs: 1 active+clean; 577 KiB data, 332 MiB used, 983 GiB / 983 GiB avail; 77 KiB/s, 0 objects/s recovering 2024-05-03T21:17:27.575 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:17:27 smithi138 ceph-mon[33110]: pgmap v237: 1 pgs: 1 active+clean; 577 KiB data, 332 MiB used, 983 GiB / 983 GiB avail; 68 KiB/s, 0 objects/s recovering 2024-05-03T21:17:27.721 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:17:27 smithi067 ceph-mon[29730]: pgmap v237: 1 pgs: 1 active+clean; 577 KiB data, 332 MiB used, 983 GiB / 983 GiB avail; 68 KiB/s, 0 objects/s recovering 2024-05-03T21:17:27.745 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:17:27 smithi082 ceph-mon[32933]: pgmap v237: 1 pgs: 1 active+clean; 577 KiB data, 332 MiB used, 983 GiB / 983 GiB avail; 68 KiB/s, 0 objects/s recovering 2024-05-03T21:17:28.574 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:17:28 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-05-03T21:17:28.574 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:17:28 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T21:17:28.721 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:17:28 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-05-03T21:17:28.721 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:17:28 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T21:17:28.745 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:17:28 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-05-03T21:17:28.745 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:17:28 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T21:17:29.520 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:17:29 smithi138 ceph-mon[33110]: Deploying daemon osd.11 on smithi138 2024-05-03T21:17:29.520 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:17:29 smithi138 ceph-mon[33110]: pgmap v238: 1 pgs: 1 active+clean; 577 KiB data, 332 MiB used, 983 GiB / 983 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-05-03T21:17:29.721 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:17:29 smithi067 ceph-mon[29730]: Deploying daemon osd.11 on smithi138 2024-05-03T21:17:29.721 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:17:29 smithi067 ceph-mon[29730]: pgmap v238: 1 pgs: 1 active+clean; 577 KiB data, 332 MiB used, 983 GiB / 983 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-05-03T21:17:29.745 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:17:29 smithi082 ceph-mon[32933]: Deploying daemon osd.11 on smithi138 2024-05-03T21:17:29.745 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:17:29 smithi082 ceph-mon[32933]: pgmap v238: 1 pgs: 1 active+clean; 577 KiB data, 332 MiB used, 983 GiB / 983 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-05-03T21:17:31.575 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:17:31 smithi138 ceph-mon[33110]: pgmap v239: 1 pgs: 1 active+clean; 577 KiB data, 332 MiB used, 983 GiB / 983 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-05-03T21:17:31.721 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:17:31 smithi067 ceph-mon[29730]: pgmap v239: 1 pgs: 1 active+clean; 577 KiB data, 332 MiB used, 983 GiB / 983 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-05-03T21:17:31.745 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:17:31 smithi082 ceph-mon[32933]: pgmap v239: 1 pgs: 1 active+clean; 577 KiB data, 332 MiB used, 983 GiB / 983 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-05-03T21:17:33.574 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:17:33 smithi138 ceph-mon[33110]: pgmap v240: 1 pgs: 1 active+clean; 577 KiB data, 332 MiB used, 983 GiB / 983 GiB avail; 49 KiB/s, 0 objects/s recovering 2024-05-03T21:17:33.575 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:17:33 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:17:33.575 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:17:33 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:17:33.575 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:17:33 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T21:17:33.721 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:17:33 smithi067 ceph-mon[29730]: pgmap v240: 1 pgs: 1 active+clean; 577 KiB data, 332 MiB used, 983 GiB / 983 GiB avail; 49 KiB/s, 0 objects/s recovering 2024-05-03T21:17:33.721 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:17:33 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:17:33.721 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:17:33 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:17:33.721 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:17:33 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T21:17:33.745 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:17:33 smithi082 ceph-mon[32933]: pgmap v240: 1 pgs: 1 active+clean; 577 KiB data, 332 MiB used, 983 GiB / 983 GiB avail; 49 KiB/s, 0 objects/s recovering 2024-05-03T21:17:33.745 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:17:33 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:17:33.745 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:17:33 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:17:33.745 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:17:33 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T21:17:35.574 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:17:35 smithi138 ceph-mon[33110]: pgmap v241: 1 pgs: 1 active+clean; 577 KiB data, 332 MiB used, 983 GiB / 983 GiB avail 2024-05-03T21:17:35.721 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:17:35 smithi067 ceph-mon[29730]: pgmap v241: 1 pgs: 1 active+clean; 577 KiB data, 332 MiB used, 983 GiB / 983 GiB avail 2024-05-03T21:17:35.745 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:17:35 smithi082 ceph-mon[32933]: pgmap v241: 1 pgs: 1 active+clean; 577 KiB data, 332 MiB used, 983 GiB / 983 GiB avail 2024-05-03T21:17:36.037 INFO:teuthology.orchestra.run.smithi138.stdout:Created osd(s) 11 on host 'smithi138' 2024-05-03T21:17:36.037 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:17:36.035+0000 7f7a627fc700 1 -- 172.21.15.138:0/2536707470 <== mgr.14150 v2:172.21.15.67:6800/604451592 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+38 (secure 0 0 0) 0x7f7a700638e0 con 0x7f7a5c0b2880 2024-05-03T21:17:36.039 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:17:36.036+0000 7f7a7781e700 1 -- 172.21.15.138:0/2536707470 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f7a5c0b2880 msgr2=0x7f7a5c0b4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:17:36.039 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:17:36.036+0000 7f7a7781e700 1 --2- 172.21.15.138:0/2536707470 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f7a5c0b2880 0x7f7a5c0b4d40 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f7a70102c30 tx=0x7f7a6400b410 comp rx=0 tx=0).stop 2024-05-03T21:17:36.039 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:17:36.036+0000 7f7a7781e700 1 -- 172.21.15.138:0/2536707470 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f7a70102dd0 msgr2=0x7f7a70199d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:17:36.039 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:17:36.036+0000 7f7a7781e700 1 --2- 172.21.15.138:0/2536707470 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f7a70102dd0 0x7f7a70199d00 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f7a6c00f970 tx=0x7f7a6c00f9a0 comp rx=0 tx=0).stop 2024-05-03T21:17:36.040 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:17:36.036+0000 7f7a7781e700 1 -- 172.21.15.138:0/2536707470 shutdown_connections 2024-05-03T21:17:36.040 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:17:36.036+0000 7f7a7781e700 1 --2- 172.21.15.138:0/2536707470 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f7a5c0b2880 0x7f7a5c0b4d40 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:17:36.040 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:17:36.037+0000 7f7a7781e700 1 --2- 172.21.15.138:0/2536707470 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7a70103770 0x7f7a701a0d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:17:36.040 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:17:36.037+0000 7f7a7781e700 1 --2- 172.21.15.138:0/2536707470 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f7a70101bd0 0x7f7a701997c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:17:36.040 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:17:36.037+0000 7f7a7781e700 1 --2- 172.21.15.138:0/2536707470 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f7a70102dd0 0x7f7a70199d00 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:17:36.040 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:17:36.037+0000 7f7a7781e700 1 -- 172.21.15.138:0/2536707470 >> 172.21.15.138:0/2536707470 conn(0x7f7a700fd170 msgr2=0x7f7a70106070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T21:17:36.040 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:17:36.037+0000 7f7a7781e700 1 -- 172.21.15.138:0/2536707470 shutdown_connections 2024-05-03T21:17:36.040 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:17:36.037+0000 7f7a7781e700 1 -- 172.21.15.138:0/2536707470 wait complete. 2024-05-03T21:17:36.850 DEBUG:teuthology.orchestra.run.smithi138:osd.11> sudo journalctl -f -n 0 -u ceph-713bc57a-0991-11ef-bc97-c7b262605968@osd.11.service 2024-05-03T21:17:36.853 INFO:tasks.cephadm:Waiting for 12 OSDs to come up... 2024-05-03T21:17:36.853 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9a047fe7599a549042d06a3fb5b6813a203eabea shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 713bc57a-0991-11ef-bc97-c7b262605968 -- ceph osd stat -f json 2024-05-03T21:17:37.127 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/713bc57a-0991-11ef-bc97-c7b262605968/mon.a/config 2024-05-03T21:17:37.221 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:17:36 smithi067 ceph-mon[29730]: pgmap v242: 1 pgs: 1 active+clean; 577 KiB data, 332 MiB used, 983 GiB / 983 GiB avail 2024-05-03T21:17:37.221 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:17:36 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:17:37.221 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:17:36 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:17:37.245 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:17:36 smithi082 ceph-mon[32933]: pgmap v242: 1 pgs: 1 active+clean; 577 KiB data, 332 MiB used, 983 GiB / 983 GiB avail 2024-05-03T21:17:37.245 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:17:36 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:17:37.246 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:17:36 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:17:37.324 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:17:36 smithi138 ceph-mon[33110]: pgmap v242: 1 pgs: 1 active+clean; 577 KiB data, 332 MiB used, 983 GiB / 983 GiB avail 2024-05-03T21:17:37.324 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:17:36 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:17:37.324 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:17:36 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:17:37.325 INFO:journalctl@ceph.osd.11.smithi138.stdout:May 03 21:17:37 smithi138 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-11[58547]: 2024-05-03T21:17:37.028+0000 7fdfe9eff7c0 -1 osd.11 0 log_to_monitors true 2024-05-03T21:17:38.245 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:17:37 smithi082 ceph-mon[32933]: from='osd.11 [v2:172.21.15.138:6824/2528685018,v1:172.21.15.138:6825/2528685018]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-05-03T21:17:38.245 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:17:37 smithi082 ceph-mon[32933]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-05-03T21:17:38.246 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:17:37 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:17:38.246 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:17:37 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:17:38.261 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:17:37 smithi067 ceph-mon[29730]: from='osd.11 [v2:172.21.15.138:6824/2528685018,v1:172.21.15.138:6825/2528685018]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-05-03T21:17:38.261 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:17:37 smithi067 ceph-mon[29730]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-05-03T21:17:38.261 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:17:37 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:17:38.261 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:17:37 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:17:38.325 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:17:37 smithi138 ceph-mon[33110]: from='osd.11 [v2:172.21.15.138:6824/2528685018,v1:172.21.15.138:6825/2528685018]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-05-03T21:17:38.325 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:17:37 smithi138 ceph-mon[33110]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-05-03T21:17:38.325 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:17:37 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:17:38.325 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:17:37 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:17:38.806 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:17:38.801+0000 7fd74cfaf700 1 -- 172.21.15.67:0/1238433304 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd748103600 msgr2=0x7fd748103a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:17:38.806 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:17:38.801+0000 7fd74cfaf700 1 --2- 172.21.15.67:0/1238433304 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd748103600 0x7fd748103a60 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fd7380099c0 tx=0x7fd738009cd0 comp rx=0 tx=0).stop 2024-05-03T21:17:38.806 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:17:38.801+0000 7fd74cfaf700 1 -- 172.21.15.67:0/1238433304 shutdown_connections 2024-05-03T21:17:38.806 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:17:38.801+0000 7fd74cfaf700 1 --2- 172.21.15.67:0/1238433304 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fd748103fa0 0x7fd74810a850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:17:38.806 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:17:38.801+0000 7fd74cfaf700 1 --2- 172.21.15.67:0/1238433304 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fd748102400 0x7fd748102820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:17:38.806 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:17:38.801+0000 7fd74cfaf700 1 --2- 172.21.15.67:0/1238433304 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd748103600 0x7fd748103a60 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:17:38.806 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:17:38.801+0000 7fd74cfaf700 1 -- 172.21.15.67:0/1238433304 >> 172.21.15.67:0/1238433304 conn(0x7fd7480fd9a0 msgr2=0x7fd7480ffde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T21:17:38.806 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:17:38.801+0000 7fd74cfaf700 1 -- 172.21.15.67:0/1238433304 shutdown_connections 2024-05-03T21:17:38.806 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:17:38.801+0000 7fd74cfaf700 1 -- 172.21.15.67:0/1238433304 wait complete. 2024-05-03T21:17:38.806 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:17:38.802+0000 7fd74cfaf700 1 Processor -- start 2024-05-03T21:17:38.807 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:17:38.802+0000 7fd74cfaf700 1 -- start start 2024-05-03T21:17:38.807 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:17:38.802+0000 7fd74cfaf700 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fd748102400 0x7fd748199850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:17:38.807 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:17:38.802+0000 7fd74cfaf700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd748103600 0x7fd748199d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:17:38.807 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:17:38.802+0000 7fd74cfaf700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fd748103fa0 0x7fd7481a0dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:17:38.807 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:17:38.802+0000 7fd74cfaf700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fd74810e130 con 0x7fd748103600 2024-05-03T21:17:38.807 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:17:38.802+0000 7fd74cfaf700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7fd74810dfb0 con 0x7fd748103fa0 2024-05-03T21:17:38.807 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:17:38.802+0000 7fd74cfaf700 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7fd74810e2b0 con 0x7fd748102400 2024-05-03T21:17:38.807 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:17:38.802+0000 7fd74659c700 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fd748102400 0x7fd748199850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:17:38.807 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:17:38.802+0000 7fd745d9b700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd748103600 0x7fd748199d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:17:38.807 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:17:38.802+0000 7fd74659c700 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fd748102400 0x7fd748199850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3300/0 says I am v2:172.21.15.67:50268/0 (socket says 172.21.15.67:50268) 2024-05-03T21:17:38.807 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:17:38.802+0000 7fd74659c700 1 -- 172.21.15.67:0/4000999002 learned_addr learned my addr 172.21.15.67:0/4000999002 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-03T21:17:38.807 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:17:38.803+0000 7fd746d9d700 1 --2- 172.21.15.67:0/4000999002 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fd748103fa0 0x7fd7481a0dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:17:38.807 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:17:38.803+0000 7fd74659c700 1 -- 172.21.15.67:0/4000999002 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fd748103fa0 msgr2=0x7fd7481a0dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:17:38.807 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:17:38.803+0000 7fd74659c700 1 --2- 172.21.15.67:0/4000999002 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fd748103fa0 0x7fd7481a0dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:17:38.808 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:17:38.803+0000 7fd74659c700 1 -- 172.21.15.67:0/4000999002 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd748103600 msgr2=0x7fd748199d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:17:38.808 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:17:38.803+0000 7fd74659c700 1 --2- 172.21.15.67:0/4000999002 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd748103600 0x7fd748199d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:17:38.808 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:17:38.803+0000 7fd74659c700 1 -- 172.21.15.67:0/4000999002 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd7481a1460 con 0x7fd748102400 2024-05-03T21:17:38.808 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:17:38.803+0000 7fd745d9b700 1 --2- 172.21.15.67:0/4000999002 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd748103600 0x7fd748199d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-03T21:17:38.808 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:17:38.803+0000 7fd74659c700 1 --2- 172.21.15.67:0/4000999002 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fd748102400 0x7fd748199850 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fd73c00c920 tx=0x7fd73c00cc30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T21:17:38.808 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:17:38.803+0000 7fd7377fe700 1 -- 172.21.15.67:0/4000999002 <== mon.1 v2:172.21.15.138:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd73c004220 con 0x7fd748102400 2024-05-03T21:17:38.808 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:17:38.803+0000 7fd74cfaf700 1 -- 172.21.15.67:0/4000999002 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd7481a1750 con 0x7fd748102400 2024-05-03T21:17:38.808 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:17:38.803+0000 7fd7377fe700 1 -- 172.21.15.67:0/4000999002 <== mon.1 v2:172.21.15.138:3300/0 2 ==== config(28 keys) ==== 1170+0+0 (secure 0 0 0) 0x7fd73c0043c0 con 0x7fd748102400 2024-05-03T21:17:38.808 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:17:38.803+0000 7fd7377fe700 1 -- 172.21.15.67:0/4000999002 <== mon.1 v2:172.21.15.138:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd73c003d20 con 0x7fd748102400 2024-05-03T21:17:38.808 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:17:38.803+0000 7fd74cfaf700 1 -- 172.21.15.67:0/4000999002 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd7481a1c40 con 0x7fd748102400 2024-05-03T21:17:38.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:17:38.804+0000 7fd74cfaf700 1 -- 172.21.15.67:0/4000999002 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd74806a190 con 0x7fd748102400 2024-05-03T21:17:38.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:17:38.806+0000 7fd7377fe700 1 -- 172.21.15.67:0/4000999002 <== mon.1 v2:172.21.15.138:3300/0 4 ==== mgrmap(e 16) ==== 149652+0+0 (secure 0 0 0) 0x7fd73c0076d0 con 0x7fd748102400 2024-05-03T21:17:38.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:17:38.806+0000 7fd7377fe700 1 --2- 172.21.15.67:0/4000999002 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7fd7300b2880 0x7fd7300b4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:17:38.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:17:38.807+0000 7fd745d9b700 1 --2- 172.21.15.67:0/4000999002 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7fd7300b2880 0x7fd7300b4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:17:38.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:17:38.807+0000 7fd7377fe700 1 -- 172.21.15.67:0/4000999002 <== mon.1 v2:172.21.15.138:3300/0 5 ==== osd_map(72..72 src has 1..72) ==== 6831+0+0 (secure 0 0 0) 0x7fd73c0e1be0 con 0x7fd748102400 2024-05-03T21:17:38.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:17:38.807+0000 7fd745d9b700 1 --2- 172.21.15.67:0/4000999002 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7fd7300b2880 0x7fd7300b4d40 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fd738009fd0 tx=0x7fd738005fb0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T21:17:38.812 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:17:38.810+0000 7fd7377fe700 1 -- 172.21.15.67:0/4000999002 <== mon.1 v2:172.21.15.138:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191839 (secure 0 0 0) 0x7fd73c014030 con 0x7fd748102400 2024-05-03T21:17:39.027 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:17:39.025+0000 7fd74cfaf700 1 -- 172.21.15.67:0/4000999002 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_command({"prefix": "osd stat", "format": "json"} v 0) -- 0x7fd7481a1f00 con 0x7fd748102400 2024-05-03T21:17:39.027 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:17:39.025+0000 7fd7377fe700 1 -- 172.21.15.67:0/4000999002 <== mon.1 v2:172.21.15.138:3300/0 7 ==== mon_command_ack([{"prefix": "osd stat", "format": "json"}]=0 v73) ==== 74+0+133 (secure 0 0 0) 0x7fd73c0a9eb0 con 0x7fd748102400 2024-05-03T21:17:39.028 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-05-03T21:17:39.030 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:17:39.027+0000 7fd74cfaf700 1 -- 172.21.15.67:0/4000999002 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7fd7300b2880 msgr2=0x7fd7300b4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:17:39.030 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:17:39.027+0000 7fd74cfaf700 1 --2- 172.21.15.67:0/4000999002 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7fd7300b2880 0x7fd7300b4d40 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fd738009fd0 tx=0x7fd738005fb0 comp rx=0 tx=0).stop 2024-05-03T21:17:39.030 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:17:39.027+0000 7fd74cfaf700 1 -- 172.21.15.67:0/4000999002 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fd748102400 msgr2=0x7fd748199850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:17:39.030 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:17:39.027+0000 7fd74cfaf700 1 --2- 172.21.15.67:0/4000999002 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fd748102400 0x7fd748199850 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fd73c00c920 tx=0x7fd73c00cc30 comp rx=0 tx=0).stop 2024-05-03T21:17:39.030 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:17:39.027+0000 7fd74cfaf700 1 -- 172.21.15.67:0/4000999002 shutdown_connections 2024-05-03T21:17:39.030 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:17:39.027+0000 7fd74cfaf700 1 --2- 172.21.15.67:0/4000999002 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7fd7300b2880 0x7fd7300b4d40 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:17:39.030 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:17:39.027+0000 7fd74cfaf700 1 --2- 172.21.15.67:0/4000999002 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fd748103fa0 0x7fd7481a0dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:17:39.030 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:17:39.027+0000 7fd74cfaf700 1 --2- 172.21.15.67:0/4000999002 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fd748102400 0x7fd748199850 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:17:39.030 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:17:39.027+0000 7fd74cfaf700 1 --2- 172.21.15.67:0/4000999002 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd748103600 0x7fd748199d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:17:39.031 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:17:39.028+0000 7fd74cfaf700 1 -- 172.21.15.67:0/4000999002 >> 172.21.15.67:0/4000999002 conn(0x7fd7480fd9a0 msgr2=0x7fd7480ff3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T21:17:39.031 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:17:39.028+0000 7fd74cfaf700 1 -- 172.21.15.67:0/4000999002 shutdown_connections 2024-05-03T21:17:39.031 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:17:39.028+0000 7fd74cfaf700 1 -- 172.21.15.67:0/4000999002 wait complete. 2024-05-03T21:17:39.061 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:17:38 smithi067 ceph-mon[29730]: pgmap v243: 1 pgs: 1 active+clean; 577 KiB data, 332 MiB used, 983 GiB / 983 GiB avail 2024-05-03T21:17:39.061 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:17:38 smithi067 ceph-mon[29730]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-05-03T21:17:39.061 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:17:38 smithi067 ceph-mon[29730]: osdmap e72: 12 total, 11 up, 12 in 2024-05-03T21:17:39.061 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:17:38 smithi067 ceph-mon[29730]: from='osd.11 [v2:172.21.15.138:6824/2528685018,v1:172.21.15.138:6825/2528685018]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi138", "root=default"]}]: dispatch 2024-05-03T21:17:39.061 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:17:38 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-05-03T21:17:39.061 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:17:38 smithi067 ceph-mon[29730]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi138", "root=default"]}]: dispatch 2024-05-03T21:17:39.324 INFO:journalctl@ceph.osd.11.smithi138.stdout:May 03 21:17:38 smithi138 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-11[58547]: 2024-05-03T21:17:38.995+0000 7fdfdf563700 -1 osd.11 0 waiting for initial osdmap 2024-05-03T21:17:39.325 INFO:journalctl@ceph.osd.11.smithi138.stdout:May 03 21:17:39 smithi138 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-11[58547]: 2024-05-03T21:17:39.008+0000 7fdfdb356700 -1 osd.11 73 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-05-03T21:17:39.325 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:17:38 smithi138 ceph-mon[33110]: pgmap v243: 1 pgs: 1 active+clean; 577 KiB data, 332 MiB used, 983 GiB / 983 GiB avail 2024-05-03T21:17:39.325 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:17:38 smithi138 ceph-mon[33110]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-05-03T21:17:39.325 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:17:38 smithi138 ceph-mon[33110]: osdmap e72: 12 total, 11 up, 12 in 2024-05-03T21:17:39.325 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:17:38 smithi138 ceph-mon[33110]: from='osd.11 [v2:172.21.15.138:6824/2528685018,v1:172.21.15.138:6825/2528685018]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi138", "root=default"]}]: dispatch 2024-05-03T21:17:39.325 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:17:38 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-05-03T21:17:39.325 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:17:38 smithi138 ceph-mon[33110]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi138", "root=default"]}]: dispatch 2024-05-03T21:17:39.495 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:17:38 smithi082 ceph-mon[32933]: pgmap v243: 1 pgs: 1 active+clean; 577 KiB data, 332 MiB used, 983 GiB / 983 GiB avail 2024-05-03T21:17:39.495 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:17:38 smithi082 ceph-mon[32933]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-05-03T21:17:39.495 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:17:38 smithi082 ceph-mon[32933]: osdmap e72: 12 total, 11 up, 12 in 2024-05-03T21:17:39.496 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:17:38 smithi082 ceph-mon[32933]: from='osd.11 [v2:172.21.15.138:6824/2528685018,v1:172.21.15.138:6825/2528685018]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi138", "root=default"]}]: dispatch 2024-05-03T21:17:39.496 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:17:38 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-05-03T21:17:39.496 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:17:38 smithi082 ceph-mon[32933]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi138", "root=default"]}]: dispatch 2024-05-03T21:17:39.696 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":73,"num_osds":12,"num_up_osds":11,"osd_up_since":1714771035,"num_in_osds":12,"osd_in_since":1714771039,"num_remapped_pgs":0} 2024-05-03T21:17:40.305 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:17:40 smithi067 ceph-mon[29730]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi138", "root=default"]}]': finished 2024-05-03T21:17:40.305 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:17:40 smithi067 ceph-mon[29730]: osdmap e73: 12 total, 11 up, 12 in 2024-05-03T21:17:40.305 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:17:40 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-05-03T21:17:40.305 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:17:40 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-05-03T21:17:40.306 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:17:40 smithi067 ceph-mon[29730]: from='client.? 172.21.15.67:0/4000999002' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-05-03T21:17:40.324 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:17:40 smithi138 ceph-mon[33110]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi138", "root=default"]}]': finished 2024-05-03T21:17:40.325 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:17:40 smithi138 ceph-mon[33110]: osdmap e73: 12 total, 11 up, 12 in 2024-05-03T21:17:40.325 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:17:40 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-05-03T21:17:40.325 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:17:40 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-05-03T21:17:40.325 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:17:40 smithi138 ceph-mon[33110]: from='client.? 172.21.15.67:0/4000999002' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-05-03T21:17:40.495 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:17:40 smithi082 ceph-mon[32933]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi138", "root=default"]}]': finished 2024-05-03T21:17:40.495 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:17:40 smithi082 ceph-mon[32933]: osdmap e73: 12 total, 11 up, 12 in 2024-05-03T21:17:40.495 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:17:40 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-05-03T21:17:40.496 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:17:40 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-05-03T21:17:40.496 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:17:40 smithi082 ceph-mon[32933]: from='client.? 172.21.15.67:0/4000999002' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-05-03T21:17:40.698 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9a047fe7599a549042d06a3fb5b6813a203eabea shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 713bc57a-0991-11ef-bc97-c7b262605968 -- ceph osd stat -f json 2024-05-03T21:17:40.946 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/713bc57a-0991-11ef-bc97-c7b262605968/mon.a/config 2024-05-03T21:17:41.260 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:17:41 smithi067 ceph-mon[29730]: purged_snaps scrub starts 2024-05-03T21:17:41.260 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:17:41 smithi067 ceph-mon[29730]: purged_snaps scrub ok 2024-05-03T21:17:41.260 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:17:41 smithi067 ceph-mon[29730]: pgmap v246: 1 pgs: 1 active+clean; 577 KiB data, 332 MiB used, 983 GiB / 983 GiB avail 2024-05-03T21:17:41.260 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:17:41 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-05-03T21:17:41.260 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:17:41 smithi067 ceph-mon[29730]: osd.11 [v2:172.21.15.138:6824/2528685018,v1:172.21.15.138:6825/2528685018] boot 2024-05-03T21:17:41.260 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:17:41 smithi067 ceph-mon[29730]: osdmap e74: 12 total, 12 up, 12 in 2024-05-03T21:17:41.260 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:17:41 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-05-03T21:17:41.260 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:17:41 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:17:41.260 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:17:41 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:17:41.260 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:17:41 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-05-03T21:17:41.260 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:17:41 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-05-03T21:17:41.260 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:17:41 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-05-03T21:17:41.261 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:17:41 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-05-03T21:17:41.261 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:17:41 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:17:41.261 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:17:41 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T21:17:41.261 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:17:41 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T21:17:41.261 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:17:41 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:17:41.261 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:17:41 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T21:17:41.261 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:17:41 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T21:17:41.261 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:17:41 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T21:17:41.261 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:17:41 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:17:41.324 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:17:41 smithi138 ceph-mon[33110]: purged_snaps scrub starts 2024-05-03T21:17:41.325 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:17:41 smithi138 ceph-mon[33110]: purged_snaps scrub ok 2024-05-03T21:17:41.325 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:17:41 smithi138 ceph-mon[33110]: pgmap v246: 1 pgs: 1 active+clean; 577 KiB data, 332 MiB used, 983 GiB / 983 GiB avail 2024-05-03T21:17:41.325 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:17:41 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-05-03T21:17:41.325 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:17:41 smithi138 ceph-mon[33110]: osd.11 [v2:172.21.15.138:6824/2528685018,v1:172.21.15.138:6825/2528685018] boot 2024-05-03T21:17:41.325 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:17:41 smithi138 ceph-mon[33110]: osdmap e74: 12 total, 12 up, 12 in 2024-05-03T21:17:41.325 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:17:41 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-05-03T21:17:41.325 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:17:41 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:17:41.325 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:17:41 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:17:41.325 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:17:41 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-05-03T21:17:41.325 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:17:41 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-05-03T21:17:41.325 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:17:41 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-05-03T21:17:41.325 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:17:41 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-05-03T21:17:41.326 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:17:41 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:17:41.326 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:17:41 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T21:17:41.326 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:17:41 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T21:17:41.326 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:17:41 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:17:41.326 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:17:41 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T21:17:41.326 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:17:41 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T21:17:41.326 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:17:41 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T21:17:41.326 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:17:41 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:17:41.495 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:17:41 smithi082 ceph-mon[32933]: purged_snaps scrub starts 2024-05-03T21:17:41.496 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:17:41 smithi082 ceph-mon[32933]: purged_snaps scrub ok 2024-05-03T21:17:41.496 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:17:41 smithi082 ceph-mon[32933]: pgmap v246: 1 pgs: 1 active+clean; 577 KiB data, 332 MiB used, 983 GiB / 983 GiB avail 2024-05-03T21:17:41.496 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:17:41 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-05-03T21:17:41.496 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:17:41 smithi082 ceph-mon[32933]: osd.11 [v2:172.21.15.138:6824/2528685018,v1:172.21.15.138:6825/2528685018] boot 2024-05-03T21:17:41.496 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:17:41 smithi082 ceph-mon[32933]: osdmap e74: 12 total, 12 up, 12 in 2024-05-03T21:17:41.496 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:17:41 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-05-03T21:17:41.496 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:17:41 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:17:41.496 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:17:41 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:17:41.496 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:17:41 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-05-03T21:17:41.496 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:17:41 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-05-03T21:17:41.496 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:17:41 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-05-03T21:17:41.496 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:17:41 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-05-03T21:17:41.496 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:17:41 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:17:41.496 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:17:41 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T21:17:41.497 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:17:41 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T21:17:41.497 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:17:41 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:17:41.497 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:17:41 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T21:17:41.497 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:17:41 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T21:17:41.497 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:17:41 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T21:17:41.497 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:17:41 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:17:42.281 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:17:42 smithi067 ceph-mon[29730]: Detected new or changed devices on smithi138 2024-05-03T21:17:42.281 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:17:42 smithi067 ceph-mon[29730]: Adjusting osd_memory_target on smithi138 to 4272M 2024-05-03T21:17:42.281 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:17:42 smithi067 ceph-mon[29730]: osdmap e75: 12 total, 12 up, 12 in 2024-05-03T21:17:42.324 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:17:42 smithi138 ceph-mon[33110]: Detected new or changed devices on smithi138 2024-05-03T21:17:42.324 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:17:42 smithi138 ceph-mon[33110]: Adjusting osd_memory_target on smithi138 to 4272M 2024-05-03T21:17:42.324 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:17:42 smithi138 ceph-mon[33110]: osdmap e75: 12 total, 12 up, 12 in 2024-05-03T21:17:42.495 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:17:42 smithi082 ceph-mon[32933]: Detected new or changed devices on smithi138 2024-05-03T21:17:42.495 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:17:42 smithi082 ceph-mon[32933]: Adjusting osd_memory_target on smithi138 to 4272M 2024-05-03T21:17:42.495 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:17:42 smithi082 ceph-mon[32933]: osdmap e75: 12 total, 12 up, 12 in 2024-05-03T21:17:42.612 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:17:42.608+0000 7f666f377700 1 -- 172.21.15.67:0/3991119705 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6668103500 msgr2=0x7f6668103920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:17:42.612 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:17:42.608+0000 7f666f377700 1 --2- 172.21.15.67:0/3991119705 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6668103500 0x7f6668103920 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f66640099c0 tx=0x7f6664009cd0 comp rx=0 tx=0).stop 2024-05-03T21:17:42.612 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:17:42.608+0000 7f666f377700 1 -- 172.21.15.67:0/3991119705 shutdown_connections 2024-05-03T21:17:42.613 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:17:42.608+0000 7f666f377700 1 --2- 172.21.15.67:0/3991119705 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6668103500 0x7f6668103920 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:17:42.613 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:17:42.608+0000 7f666f377700 1 --2- 172.21.15.67:0/3991119705 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f66681050a0 0x7f666810b950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:17:42.613 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:17:42.608+0000 7f666f377700 1 --2- 172.21.15.67:0/3991119705 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f6668104700 0x7f6668104b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:17:42.613 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:17:42.608+0000 7f666f377700 1 -- 172.21.15.67:0/3991119705 >> 172.21.15.67:0/3991119705 conn(0x7f66680feaa0 msgr2=0x7f6668100ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T21:17:42.613 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:17:42.608+0000 7f666f377700 1 -- 172.21.15.67:0/3991119705 shutdown_connections 2024-05-03T21:17:42.613 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:17:42.608+0000 7f666f377700 1 -- 172.21.15.67:0/3991119705 wait complete. 2024-05-03T21:17:42.613 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:17:42.608+0000 7f666f377700 1 Processor -- start 2024-05-03T21:17:42.613 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:17:42.608+0000 7f666f377700 1 -- start start 2024-05-03T21:17:42.613 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:17:42.609+0000 7f666f377700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f6668103500 0x7f6668199800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:17:42.613 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:17:42.609+0000 7f666f377700 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f6668104700 0x7f6668199d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:17:42.613 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:17:42.609+0000 7f666f377700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f66681050a0 0x7f66681a0dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:17:42.613 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:17:42.609+0000 7f666f377700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f666810f220 con 0x7f66681050a0 2024-05-03T21:17:42.614 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:17:42.609+0000 7f666f377700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f666810f0a0 con 0x7f6668103500 2024-05-03T21:17:42.614 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:17:42.609+0000 7f666f377700 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f666810f3a0 con 0x7f6668104700 2024-05-03T21:17:42.614 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:17:42.609+0000 7f666d914700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f66681050a0 0x7f66681a0dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:17:42.614 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:17:42.609+0000 7f666c912700 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f6668104700 0x7f6668199d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:17:42.614 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:17:42.609+0000 7f666d914700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f66681050a0 0x7f66681a0dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:52566/0 (socket says 172.21.15.67:52566) 2024-05-03T21:17:42.614 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:17:42.609+0000 7f666d914700 1 -- 172.21.15.67:0/1225415142 learned_addr learned my addr 172.21.15.67:0/1225415142 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-03T21:17:42.614 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:17:42.609+0000 7f666d113700 1 --2- 172.21.15.67:0/1225415142 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f6668103500 0x7f6668199800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:17:42.614 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:17:42.609+0000 7f666d914700 1 -- 172.21.15.67:0/1225415142 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f6668104700 msgr2=0x7f6668199d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:17:42.614 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:17:42.609+0000 7f666d914700 1 --2- 172.21.15.67:0/1225415142 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f6668104700 0x7f6668199d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:17:42.614 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:17:42.609+0000 7f666d914700 1 -- 172.21.15.67:0/1225415142 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f6668103500 msgr2=0x7f6668199800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:17:42.614 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:17:42.609+0000 7f666d914700 1 --2- 172.21.15.67:0/1225415142 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f6668103500 0x7f6668199800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:17:42.614 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:17:42.609+0000 7f666d914700 1 -- 172.21.15.67:0/1225415142 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f66681a1460 con 0x7f66681050a0 2024-05-03T21:17:42.615 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:17:42.609+0000 7f666d914700 1 --2- 172.21.15.67:0/1225415142 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f66681050a0 0x7f66681a0dc0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f6660007f10 tx=0x7f666000d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T21:17:42.615 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:17:42.609+0000 7f665e7fc700 1 -- 172.21.15.67:0/1225415142 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6660004bb0 con 0x7f66681050a0 2024-05-03T21:17:42.615 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:17:42.610+0000 7f666f377700 1 -- 172.21.15.67:0/1225415142 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f66681a1750 con 0x7f66681050a0 2024-05-03T21:17:42.615 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:17:42.610+0000 7f665e7fc700 1 -- 172.21.15.67:0/1225415142 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(28 keys) ==== 1170+0+0 (secure 0 0 0) 0x7f6660004d50 con 0x7f66681050a0 2024-05-03T21:17:42.615 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:17:42.610+0000 7f665e7fc700 1 -- 172.21.15.67:0/1225415142 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66600133f0 con 0x7f66681050a0 2024-05-03T21:17:42.615 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:17:42.610+0000 7f666f377700 1 -- 172.21.15.67:0/1225415142 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f66681a1c90 con 0x7f66681050a0 2024-05-03T21:17:42.616 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:17:42.611+0000 7f666f377700 1 -- 172.21.15.67:0/1225415142 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f666806a190 con 0x7f66681050a0 2024-05-03T21:17:42.616 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:17:42.612+0000 7f665e7fc700 1 -- 172.21.15.67:0/1225415142 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 16) ==== 149652+0+0 (secure 0 0 0) 0x7f666001a020 con 0x7f66681050a0 2024-05-03T21:17:42.616 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:17:42.613+0000 7f665e7fc700 1 --2- 172.21.15.67:0/1225415142 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f66540b2880 0x7f66540b4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:17:42.616 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:17:42.613+0000 7f666d113700 1 --2- 172.21.15.67:0/1225415142 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f66540b2880 0x7f66540b4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:17:42.616 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:17:42.613+0000 7f665e7fc700 1 -- 172.21.15.67:0/1225415142 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(75..75 src has 1..75) ==== 7127+0+0 (secure 0 0 0) 0x7f66600e1ac0 con 0x7f66681050a0 2024-05-03T21:17:42.616 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:17:42.613+0000 7f666d113700 1 --2- 172.21.15.67:0/1225415142 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f66540b2880 0x7f66540b4d40 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f6664009990 tx=0x7f6664005fb0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T21:17:42.618 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:17:42.616+0000 7f665e7fc700 1 -- 172.21.15.67:0/1225415142 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191839 (secure 0 0 0) 0x7f6660010070 con 0x7f66681050a0 2024-05-03T21:17:42.831 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:17:42.829+0000 7f666f377700 1 -- 172.21.15.67:0/1225415142 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd stat", "format": "json"} v 0) -- 0x7f666802e5b0 con 0x7f66681050a0 2024-05-03T21:17:42.831 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:17:42.829+0000 7f665e7fc700 1 -- 172.21.15.67:0/1225415142 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd stat", "format": "json"}]=0 v75) ==== 74+0+133 (secure 0 0 0) 0x7f666802e5b0 con 0x7f66681050a0 2024-05-03T21:17:42.831 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-05-03T21:17:42.834 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:17:42.831+0000 7f666f377700 1 -- 172.21.15.67:0/1225415142 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f66540b2880 msgr2=0x7f66540b4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:17:42.834 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:17:42.831+0000 7f666f377700 1 --2- 172.21.15.67:0/1225415142 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f66540b2880 0x7f66540b4d40 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f6664009990 tx=0x7f6664005fb0 comp rx=0 tx=0).stop 2024-05-03T21:17:42.834 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:17:42.831+0000 7f666f377700 1 -- 172.21.15.67:0/1225415142 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f66681050a0 msgr2=0x7f66681a0dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:17:42.834 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:17:42.831+0000 7f666f377700 1 --2- 172.21.15.67:0/1225415142 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f66681050a0 0x7f66681a0dc0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f6660007f10 tx=0x7f666000d3b0 comp rx=0 tx=0).stop 2024-05-03T21:17:42.834 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:17:42.831+0000 7f666f377700 1 -- 172.21.15.67:0/1225415142 shutdown_connections 2024-05-03T21:17:42.835 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:17:42.831+0000 7f666f377700 1 --2- 172.21.15.67:0/1225415142 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f66540b2880 0x7f66540b4d40 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:17:42.835 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:17:42.831+0000 7f666f377700 1 --2- 172.21.15.67:0/1225415142 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f66681050a0 0x7f66681a0dc0 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:17:42.835 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:17:42.831+0000 7f666f377700 1 --2- 172.21.15.67:0/1225415142 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f6668103500 0x7f6668199800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:17:42.835 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:17:42.831+0000 7f666f377700 1 --2- 172.21.15.67:0/1225415142 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f6668104700 0x7f6668199d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:17:42.835 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:17:42.831+0000 7f666f377700 1 -- 172.21.15.67:0/1225415142 >> 172.21.15.67:0/1225415142 conn(0x7f66680feaa0 msgr2=0x7f6668107a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T21:17:42.835 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:17:42.832+0000 7f666f377700 1 -- 172.21.15.67:0/1225415142 shutdown_connections 2024-05-03T21:17:42.835 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:17:42.832+0000 7f666f377700 1 -- 172.21.15.67:0/1225415142 wait complete. 2024-05-03T21:17:43.324 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:17:43 smithi138 ceph-mon[33110]: pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-03T21:17:43.325 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:17:43 smithi138 ceph-mon[33110]: from='client.? 172.21.15.67:0/1225415142' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-05-03T21:17:43.471 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:17:43 smithi067 ceph-mon[29730]: pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-03T21:17:43.471 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:17:43 smithi067 ceph-mon[29730]: from='client.? 172.21.15.67:0/1225415142' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-05-03T21:17:43.495 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:17:43 smithi082 ceph-mon[32933]: pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-03T21:17:43.495 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:17:43 smithi082 ceph-mon[32933]: from='client.? 172.21.15.67:0/1225415142' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-05-03T21:17:43.592 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":75,"num_osds":12,"num_up_osds":12,"osd_up_since":1714771059,"num_in_osds":12,"osd_in_since":1714771039,"num_remapped_pgs":0} 2024-05-03T21:17:43.592 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9a047fe7599a549042d06a3fb5b6813a203eabea shell --fsid 713bc57a-0991-11ef-bc97-c7b262605968 -- ceph osd dump --format=json 2024-05-03T21:17:43.840 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/713bc57a-0991-11ef-bc97-c7b262605968/mon.a/config 2024-05-03T21:17:45.203 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:17:45 smithi067 ceph-mon[29730]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-03T21:17:45.298 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:17:45 smithi082 ceph-mon[32933]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-03T21:17:45.324 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:17:45 smithi138 ceph-mon[33110]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-03T21:17:45.451 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:17:45.448+0000 7f138a4a1700 1 -- 172.21.15.67:0/1006237351 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1384108870 msgr2=0x7f13841103f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:17:45.451 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:17:45.448+0000 7f138a4a1700 1 --2- 172.21.15.67:0/1006237351 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1384108870 0x7f13841103f0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f137c00b0d0 tx=0x7f137c00b3e0 comp rx=0 tx=0).stop 2024-05-03T21:17:45.451 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:17:45.448+0000 7f138a4a1700 1 -- 172.21.15.67:0/1006237351 shutdown_connections 2024-05-03T21:17:45.451 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:17:45.448+0000 7f138a4a1700 1 --2- 172.21.15.67:0/1006237351 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1384108870 0x7f13841103f0 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:17:45.451 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:17:45.448+0000 7f138a4a1700 1 --2- 172.21.15.67:0/1006237351 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f1384103780 0x7f1384103b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:17:45.451 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:17:45.448+0000 7f138a4a1700 1 --2- 172.21.15.67:0/1006237351 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f1384104130 0x7f1384108140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:17:45.451 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:17:45.448+0000 7f138a4a1700 1 -- 172.21.15.67:0/1006237351 >> 172.21.15.67:0/1006237351 conn(0x7f138407a610 msgr2=0x7f138407aa20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T21:17:45.451 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:17:45.448+0000 7f138a4a1700 1 -- 172.21.15.67:0/1006237351 shutdown_connections 2024-05-03T21:17:45.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:17:45.448+0000 7f138a4a1700 1 -- 172.21.15.67:0/1006237351 wait complete. 2024-05-03T21:17:45.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:17:45.449+0000 7f138a4a1700 1 Processor -- start 2024-05-03T21:17:45.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:17:45.449+0000 7f138a4a1700 1 -- start start 2024-05-03T21:17:45.453 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:17:45.449+0000 7f138a4a1700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1384103780 0x7f138419f860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:17:45.454 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:17:45.449+0000 7f138a4a1700 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f1384104130 0x7f138419fda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:17:45.454 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:17:45.449+0000 7f138a4a1700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f1384108870 0x7f1384199a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:17:45.454 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:17:45.449+0000 7f138a4a1700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f13841138f0 con 0x7f1384103780 2024-05-03T21:17:45.454 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:17:45.449+0000 7f138a4a1700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f1384113770 con 0x7f1384108870 2024-05-03T21:17:45.454 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:17:45.449+0000 7f138a4a1700 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f1384113a70 con 0x7f1384104130 2024-05-03T21:17:45.454 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:17:45.449+0000 7f1383fff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1384103780 0x7f138419f860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:17:45.454 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:17:45.449+0000 7f1383fff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1384103780 0x7f138419f860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:52586/0 (socket says 172.21.15.67:52586) 2024-05-03T21:17:45.454 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:17:45.449+0000 7f1383fff700 1 -- 172.21.15.67:0/1063805839 learned_addr learned my addr 172.21.15.67:0/1063805839 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-03T21:17:45.454 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:17:45.449+0000 7f13837fe700 1 --2- 172.21.15.67:0/1063805839 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f1384104130 0x7f138419fda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:17:45.454 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:17:45.449+0000 7f1388a3e700 1 --2- 172.21.15.67:0/1063805839 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f1384108870 0x7f1384199a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:17:45.454 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:17:45.450+0000 7f1383fff700 1 -- 172.21.15.67:0/1063805839 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f1384104130 msgr2=0x7f138419fda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:17:45.454 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:17:45.450+0000 7f1383fff700 1 --2- 172.21.15.67:0/1063805839 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f1384104130 0x7f138419fda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:17:45.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:17:45.450+0000 7f1383fff700 1 -- 172.21.15.67:0/1063805839 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f1384108870 msgr2=0x7f1384199a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:17:45.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:17:45.450+0000 7f1383fff700 1 --2- 172.21.15.67:0/1063805839 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f1384108870 0x7f1384199a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:17:45.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:17:45.450+0000 7f1383fff700 1 -- 172.21.15.67:0/1063805839 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f138419a1d0 con 0x7f1384103780 2024-05-03T21:17:45.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:17:45.450+0000 7f13837fe700 1 --2- 172.21.15.67:0/1063805839 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f1384104130 0x7f138419fda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-03T21:17:45.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:17:45.450+0000 7f1388a3e700 1 --2- 172.21.15.67:0/1063805839 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f1384108870 0x7f1384199a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-03T21:17:45.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:17:45.450+0000 7f1383fff700 1 --2- 172.21.15.67:0/1063805839 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1384103780 0x7f138419f860 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f137400e9a0 tx=0x7f137400ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T21:17:45.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:17:45.450+0000 7f13817fa700 1 -- 172.21.15.67:0/1063805839 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f137400ca10 con 0x7f1384103780 2024-05-03T21:17:45.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:17:45.450+0000 7f138a4a1700 1 -- 172.21.15.67:0/1063805839 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f138419a4c0 con 0x7f1384103780 2024-05-03T21:17:45.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:17:45.450+0000 7f13817fa700 1 -- 172.21.15.67:0/1063805839 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(28 keys) ==== 1170+0+0 (secure 0 0 0) 0x7f137400cbb0 con 0x7f1384103780 2024-05-03T21:17:45.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:17:45.450+0000 7f13817fa700 1 -- 172.21.15.67:0/1063805839 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13740187a0 con 0x7f1384103780 2024-05-03T21:17:45.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:17:45.450+0000 7f138a4a1700 1 -- 172.21.15.67:0/1063805839 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f13841a6730 con 0x7f1384103780 2024-05-03T21:17:45.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:17:45.451+0000 7f138a4a1700 1 -- 172.21.15.67:0/1063805839 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f138406a190 con 0x7f1384103780 2024-05-03T21:17:45.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:17:45.453+0000 7f13817fa700 1 -- 172.21.15.67:0/1063805839 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 16) ==== 149652+0+0 (secure 0 0 0) 0x7f1374010430 con 0x7f1384103780 2024-05-03T21:17:45.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:17:45.453+0000 7f13817fa700 1 --2- 172.21.15.67:0/1063805839 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f136c0b2880 0x7f136c0b4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:17:45.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:17:45.453+0000 7f13837fe700 1 --2- 172.21.15.67:0/1063805839 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f136c0b2880 0x7f136c0b4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:17:45.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:17:45.453+0000 7f13817fa700 1 -- 172.21.15.67:0/1063805839 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(75..75 src has 1..75) ==== 7127+0+0 (secure 0 0 0) 0x7f13740e21b0 con 0x7f1384103780 2024-05-03T21:17:45.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:17:45.454+0000 7f13837fe700 1 --2- 172.21.15.67:0/1063805839 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f136c0b2880 0x7f136c0b4d40 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f138407af60 tx=0x7f137800b3c0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T21:17:45.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:17:45.457+0000 7f13817fa700 1 -- 172.21.15.67:0/1063805839 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191839 (secure 0 0 0) 0x7f13740aa370 con 0x7f1384103780 2024-05-03T21:17:45.678 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:17:45.675+0000 7f138a4a1700 1 -- 172.21.15.67:0/1063805839 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f138402e510 con 0x7f1384103780 2024-05-03T21:17:45.678 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:17:45.676+0000 7f13817fa700 1 -- 172.21.15.67:0/1063805839 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v75) ==== 74+0+18442 (secure 0 0 0) 0x7f13740b23f0 con 0x7f1384103780 2024-05-03T21:17:45.679 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-05-03T21:17:45.679 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":75,"fsid":"713bc57a-0991-11ef-bc97-c7b262605968","created":"2024-05-03T21:10:34.070178+0000","modified":"2024-05-03T21:17:40.999962+0000","last_up_change":"2024-05-03T21:17:39.993919+0000","last_in_change":"2024-05-03T21:17:19.978727+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-03T21:13:59.810446+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7939f31c-6fb7-42bd-af96-88a448d11bd2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":3692483807},{"type":"v1","addr":"172.21.15.67:6803","nonce":3692483807}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":3692483807},{"type":"v1","addr":"172.21.15.67:6805","nonce":3692483807}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":3692483807},{"type":"v1","addr":"172.21.15.67:6809","nonce":3692483807}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":3692483807},{"type":"v1","addr":"172.21.15.67:6807","nonce":3692483807}]},"public_addr":"172.21.15.67:6803/3692483807","cluster_addr":"172.21.15.67:6805/3692483807","heartbeat_back_addr":"172.21.15.67:6809/3692483807","heartbeat_front_addr":"172.21.15.67:6807/3692483807","state":["exists","up"]},{"osd":1,"uuid":"71858193-05a6-4c2c-9bb7-bc3d811d634e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6800","nonce":1139376397},{"type":"v1","addr":"172.21.15.82:6801","nonce":1139376397}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6802","nonce":1139376397},{"type":"v1","addr":"172.21.15.82:6803","nonce":1139376397}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6806","nonce":1139376397},{"type":"v1","addr":"172.21.15.82:6807","nonce":1139376397}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6804","nonce":1139376397},{"type":"v1","addr":"172.21.15.82:6805","nonce":1139376397}]},"public_addr":"172.21.15.82:6801/1139376397","cluster_addr":"172.21.15.82:6803/1139376397","heartbeat_back_addr":"172.21.15.82:6807/1139376397","heartbeat_front_addr":"172.21.15.82:6805/1139376397","state":["exists","up"]},{"osd":2,"uuid":"f9ece78d-6a19-4d46-9022-6ed184074745","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6800","nonce":2774330123},{"type":"v1","addr":"172.21.15.138:6801","nonce":2774330123}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6802","nonce":2774330123},{"type":"v1","addr":"172.21.15.138:6803","nonce":2774330123}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6806","nonce":2774330123},{"type":"v1","addr":"172.21.15.138:6807","nonce":2774330123}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6804","nonce":2774330123},{"type":"v1","addr":"172.21.15.138:6805","nonce":2774330123}]},"public_addr":"172.21.15.138:6801/2774330123","cluster_addr":"172.21.15.138:6803/2774330123","heartbeat_back_addr":"172.21.15.138:6807/2774330123","heartbeat_front_addr":"172.21.15.138:6805/2774330123","state":["exists","up"]},{"osd":3,"uuid":"3616df71-e7b3-4687-aaba-37b0b7766a3c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":27,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":3915934652},{"type":"v1","addr":"172.21.15.67:6811","nonce":3915934652}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":3915934652},{"type":"v1","addr":"172.21.15.67:6813","nonce":3915934652}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":3915934652},{"type":"v1","addr":"172.21.15.67:6817","nonce":3915934652}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":3915934652},{"type":"v1","addr":"172.21.15.67:6815","nonce":3915934652}]},"public_addr":"172.21.15.67:6811/3915934652","cluster_addr":"172.21.15.67:6813/3915934652","heartbeat_back_addr":"172.21.15.67:6817/3915934652","heartbeat_front_addr":"172.21.15.67:6815/3915934652","state":["exists","up"]},{"osd":4,"uuid":"c3fd0ffd-e23f-4279-b679-9d95a144cf32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6808","nonce":4031818951},{"type":"v1","addr":"172.21.15.82:6809","nonce":4031818951}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6810","nonce":4031818951},{"type":"v1","addr":"172.21.15.82:6811","nonce":4031818951}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6814","nonce":4031818951},{"type":"v1","addr":"172.21.15.82:6815","nonce":4031818951}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6812","nonce":4031818951},{"type":"v1","addr":"172.21.15.82:6813","nonce":4031818951}]},"public_addr":"172.21.15.82:6809/4031818951","cluster_addr":"172.21.15.82:6811/4031818951","heartbeat_back_addr":"172.21.15.82:6815/4031818951","heartbeat_front_addr":"172.21.15.82:6813/4031818951","state":["exists","up"]},{"osd":5,"uuid":"540ef0d9-a6dc-43c6-b09e-e87340ba24cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6808","nonce":2521889499},{"type":"v1","addr":"172.21.15.138:6809","nonce":2521889499}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6810","nonce":2521889499},{"type":"v1","addr":"172.21.15.138:6811","nonce":2521889499}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6814","nonce":2521889499},{"type":"v1","addr":"172.21.15.138:6815","nonce":2521889499}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6812","nonce":2521889499},{"type":"v1","addr":"172.21.15.138:6813","nonce":2521889499}]},"public_addr":"172.21.15.138:6809/2521889499","cluster_addr":"172.21.15.138:6811/2521889499","heartbeat_back_addr":"172.21.15.138:6815/2521889499","heartbeat_front_addr":"172.21.15.138:6813/2521889499","state":["exists","up"]},{"osd":6,"uuid":"49a89fd0-958a-4be9-a8db-6022f946153e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6818","nonce":3269682731},{"type":"v1","addr":"172.21.15.67:6819","nonce":3269682731}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":3269682731},{"type":"v1","addr":"172.21.15.67:6821","nonce":3269682731}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":3269682731},{"type":"v1","addr":"172.21.15.67:6825","nonce":3269682731}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":3269682731},{"type":"v1","addr":"172.21.15.67:6823","nonce":3269682731}]},"public_addr":"172.21.15.67:6819/3269682731","cluster_addr":"172.21.15.67:6821/3269682731","heartbeat_back_addr":"172.21.15.67:6825/3269682731","heartbeat_front_addr":"172.21.15.67:6823/3269682731","state":["exists","up"]},{"osd":7,"uuid":"8fa4cd64-ca6b-48ae-8db0-d369ce6102ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":50,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6816","nonce":3008589976},{"type":"v1","addr":"172.21.15.82:6817","nonce":3008589976}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6818","nonce":3008589976},{"type":"v1","addr":"172.21.15.82:6819","nonce":3008589976}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6822","nonce":3008589976},{"type":"v1","addr":"172.21.15.82:6823","nonce":3008589976}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6820","nonce":3008589976},{"type":"v1","addr":"172.21.15.82:6821","nonce":3008589976}]},"public_addr":"172.21.15.82:6817/3008589976","cluster_addr":"172.21.15.82:6819/3008589976","heartbeat_back_addr":"172.21.15.82:6823/3008589976","heartbeat_front_addr":"172.21.15.82:6821/3008589976","state":["exists","up"]},{"osd":8,"uuid":"fae39087-6c38-4771-a2b9-3f4fecb08edd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":57,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6816","nonce":2223920070},{"type":"v1","addr":"172.21.15.138:6817","nonce":2223920070}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6818","nonce":2223920070},{"type":"v1","addr":"172.21.15.138:6819","nonce":2223920070}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6822","nonce":2223920070},{"type":"v1","addr":"172.21.15.138:6823","nonce":2223920070}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6820","nonce":2223920070},{"type":"v1","addr":"172.21.15.138:6821","nonce":2223920070}]},"public_addr":"172.21.15.138:6817/2223920070","cluster_addr":"172.21.15.138:6819/2223920070","heartbeat_back_addr":"172.21.15.138:6823/2223920070","heartbeat_front_addr":"172.21.15.138:6821/2223920070","state":["exists","up"]},{"osd":9,"uuid":"3d387dbf-0163-41b7-98af-4fc45ed73ad4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":63,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":3939417129},{"type":"v1","addr":"172.21.15.67:6827","nonce":3939417129}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":3939417129},{"type":"v1","addr":"172.21.15.67:6829","nonce":3939417129}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":3939417129},{"type":"v1","addr":"172.21.15.67:6833","nonce":3939417129}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":3939417129},{"type":"v1","addr":"172.21.15.67:6831","nonce":3939417129}]},"public_addr":"172.21.15.67:6827/3939417129","cluster_addr":"172.21.15.67:6829/3939417129","heartbeat_back_addr":"172.21.15.67:6833/3939417129","heartbeat_front_addr":"172.21.15.67:6831/3939417129","state":["exists","up"]},{"osd":10,"uuid":"db0305a0-a83b-474b-837a-fbc239ac1738","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6824","nonce":2505507029},{"type":"v1","addr":"172.21.15.82:6825","nonce":2505507029}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6826","nonce":2505507029},{"type":"v1","addr":"172.21.15.82:6827","nonce":2505507029}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6830","nonce":2505507029},{"type":"v1","addr":"172.21.15.82:6831","nonce":2505507029}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6828","nonce":2505507029},{"type":"v1","addr":"172.21.15.82:6829","nonce":2505507029}]},"public_addr":"172.21.15.82:6825/2505507029","cluster_addr":"172.21.15.82:6827/2505507029","heartbeat_back_addr":"172.21.15.82:6831/2505507029","heartbeat_front_addr":"172.21.15.82:6829/2505507029","state":["exists","up"]},{"osd":11,"uuid":"479c2747-f28e-4704-87e4-d0c31782e2de","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":74,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6824","nonce":2528685018},{"type":"v1","addr":"172.21.15.138:6825","nonce":2528685018}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6826","nonce":2528685018},{"type":"v1","addr":"172.21.15.138:6827","nonce":2528685018}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6830","nonce":2528685018},{"type":"v1","addr":"172.21.15.138:6831","nonce":2528685018}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6828","nonce":2528685018},{"type":"v1","addr":"172.21.15.138:6829","nonce":2528685018}]},"public_addr":"172.21.15.138:6825/2528685018","cluster_addr":"172.21.15.138:6827/2528685018","heartbeat_back_addr":"172.21.15.138:6831/2528685018","heartbeat_front_addr":"172.21.15.138:6829/2528685018","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-03T21:13:07.768773+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-03T21:13:31.870232+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-03T21:13:55.394206+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-03T21:14:20.540281+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-03T21:14:44.318523+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-03T21:15:08.490829+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-03T21:15:34.144734+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-03T21:15:58.717240+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-03T21:16:22.743013+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-03T21:16:49.099588+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-03T21:17:13.321126+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-03T21:17:38.001978+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.67:0/1239643865":"2024-05-04T21:11:33.721416+0000","172.21.15.67:6801/2576113038":"2024-05-04T21:11:33.721416+0000","172.21.15.67:6800/2576113038":"2024-05-04T21:11:33.721416+0000","172.21.15.67:6800/1698171509":"2024-05-04T21:11:07.333651+0000","172.21.15.67:0/2217893923":"2024-05-04T21:11:33.721416+0000","172.21.15.67:0/20864811":"2024-05-04T21:11:07.333651+0000","172.21.15.67:6801/1698171509":"2024-05-04T21:11:07.333651+0000","172.21.15.67:0/773376553":"2024-05-04T21:11:33.721416+0000","172.21.15.67:0/847549114":"2024-05-04T21:11:07.333651+0000","172.21.15.67:0/1291778035":"2024-05-04T21:11:07.333651+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-03T21:17:45.681 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:17:45.679+0000 7f138a4a1700 1 -- 172.21.15.67:0/1063805839 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f136c0b2880 msgr2=0x7f136c0b4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:17:45.681 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:17:45.679+0000 7f138a4a1700 1 --2- 172.21.15.67:0/1063805839 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f136c0b2880 0x7f136c0b4d40 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f138407af60 tx=0x7f137800b3c0 comp rx=0 tx=0).stop 2024-05-03T21:17:45.681 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:17:45.679+0000 7f138a4a1700 1 -- 172.21.15.67:0/1063805839 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1384103780 msgr2=0x7f138419f860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:17:45.681 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:17:45.679+0000 7f138a4a1700 1 --2- 172.21.15.67:0/1063805839 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1384103780 0x7f138419f860 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f137400e9a0 tx=0x7f137400ecb0 comp rx=0 tx=0).stop 2024-05-03T21:17:45.681 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:17:45.679+0000 7f138a4a1700 1 -- 172.21.15.67:0/1063805839 shutdown_connections 2024-05-03T21:17:45.681 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:17:45.679+0000 7f138a4a1700 1 --2- 172.21.15.67:0/1063805839 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f136c0b2880 0x7f136c0b4d40 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:17:45.681 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:17:45.679+0000 7f138a4a1700 1 --2- 172.21.15.67:0/1063805839 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1384103780 0x7f138419f860 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:17:45.681 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:17:45.679+0000 7f138a4a1700 1 --2- 172.21.15.67:0/1063805839 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f1384108870 0x7f1384199a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:17:45.682 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:17:45.679+0000 7f138a4a1700 1 --2- 172.21.15.67:0/1063805839 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f1384104130 0x7f138419fda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:17:45.682 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:17:45.679+0000 7f138a4a1700 1 -- 172.21.15.67:0/1063805839 >> 172.21.15.67:0/1063805839 conn(0x7f138407a610 msgr2=0x7f1384102580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T21:17:45.682 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:17:45.679+0000 7f138a4a1700 1 -- 172.21.15.67:0/1063805839 shutdown_connections 2024-05-03T21:17:45.682 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:17:45.679+0000 7f138a4a1700 1 -- 172.21.15.67:0/1063805839 wait complete. 2024-05-03T21:17:46.324 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:17:46 smithi138 ceph-mon[33110]: from='client.? 172.21.15.67:0/1063805839' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-03T21:17:46.408 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:17:46 smithi067 ceph-mon[29730]: from='client.? 172.21.15.67:0/1063805839' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-03T21:17:46.441 INFO:tasks.cephadm.ceph_manager.ceph:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-05-03T21:13:59.810446+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-05-03T21:17:46.442 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9a047fe7599a549042d06a3fb5b6813a203eabea shell --fsid 713bc57a-0991-11ef-bc97-c7b262605968 -- ceph osd pool get .mgr pg_num 2024-05-03T21:17:46.487 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:17:46 smithi082 ceph-mon[32933]: from='client.? 172.21.15.67:0/1063805839' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-03T21:17:46.691 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/713bc57a-0991-11ef-bc97-c7b262605968/mon.a/config 2024-05-03T21:17:47.145 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:17:47 smithi067 ceph-mon[29730]: pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-03T21:17:47.324 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:17:47 smithi138 ceph-mon[33110]: pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-03T21:17:47.495 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:17:47 smithi082 ceph-mon[32933]: pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-03T21:17:48.413 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:17:48.412+0000 7f68eca40700 1 -- 172.21.15.67:0/750650731 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f68e81020a0 msgr2=0x7f68e8102480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:17:48.414 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:17:48.412+0000 7f68eca40700 1 --2- 172.21.15.67:0/750650731 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f68e81020a0 0x7f68e8102480 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f68d8009990 tx=0x7f68d8009ca0 comp rx=0 tx=0).stop 2024-05-03T21:17:48.414 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:17:48.412+0000 7f68eca40700 1 -- 172.21.15.67:0/750650731 shutdown_connections 2024-05-03T21:17:48.414 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:17:48.412+0000 7f68eca40700 1 --2- 172.21.15.67:0/750650731 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f68e81020a0 0x7f68e8102480 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:17:48.414 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:17:48.412+0000 7f68eca40700 1 --2- 172.21.15.67:0/750650731 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f68e8106f40 0x7f68e810ea80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:17:48.414 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:17:48.412+0000 7f68eca40700 1 --2- 172.21.15.67:0/750650731 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f68e8102a50 0x7f68e8106810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:17:48.414 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:17:48.412+0000 7f68eca40700 1 -- 172.21.15.67:0/750650731 >> 172.21.15.67:0/750650731 conn(0x7f68e80776c0 msgr2=0x7f68e8077ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T21:17:48.414 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:17:48.412+0000 7f68eca40700 1 -- 172.21.15.67:0/750650731 shutdown_connections 2024-05-03T21:17:48.414 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:17:48.412+0000 7f68eca40700 1 -- 172.21.15.67:0/750650731 wait complete. 2024-05-03T21:17:48.416 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:17:48.413+0000 7f68eca40700 1 Processor -- start 2024-05-03T21:17:48.416 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:17:48.413+0000 7f68eca40700 1 -- start start 2024-05-03T21:17:48.416 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:17:48.413+0000 7f68eca40700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f68e81020a0 0x7f68e8199d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:17:48.417 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:17:48.413+0000 7f68eca40700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f68e8102a50 0x7f68e819a2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:17:48.417 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:17:48.413+0000 7f68eca40700 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f68e8106f40 0x7f68e819e660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:17:48.417 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:17:48.413+0000 7f68eca40700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f68e8113980 con 0x7f68e8102a50 2024-05-03T21:17:48.417 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:17:48.413+0000 7f68eca40700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f68e8113800 con 0x7f68e81020a0 2024-05-03T21:17:48.417 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:17:48.413+0000 7f68eca40700 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f68e8113b00 con 0x7f68e8106f40 2024-05-03T21:17:48.417 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:17:48.413+0000 7f68e5d9b700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f68e8102a50 0x7f68e819a2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:17:48.417 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:17:48.413+0000 7f68e5d9b700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f68e8102a50 0x7f68e819a2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:50254/0 (socket says 172.21.15.67:50254) 2024-05-03T21:17:48.417 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:17:48.413+0000 7f68e5d9b700 1 -- 172.21.15.67:0/1045662869 learned_addr learned my addr 172.21.15.67:0/1045662869 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-03T21:17:48.417 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:17:48.413+0000 7f68e659c700 1 --2- 172.21.15.67:0/1045662869 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f68e81020a0 0x7f68e8199d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:17:48.417 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:17:48.414+0000 7f68e6d9d700 1 --2- 172.21.15.67:0/1045662869 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f68e8106f40 0x7f68e819e660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:17:48.417 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:17:48.414+0000 7f68e5d9b700 1 -- 172.21.15.67:0/1045662869 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f68e8106f40 msgr2=0x7f68e819e660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:17:48.417 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:17:48.414+0000 7f68e5d9b700 1 --2- 172.21.15.67:0/1045662869 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f68e8106f40 0x7f68e819e660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:17:48.417 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:17:48.414+0000 7f68e5d9b700 1 -- 172.21.15.67:0/1045662869 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f68e81020a0 msgr2=0x7f68e8199d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:17:48.418 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:17:48.414+0000 7f68e5d9b700 1 --2- 172.21.15.67:0/1045662869 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f68e81020a0 0x7f68e8199d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:17:48.418 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:17:48.414+0000 7f68e5d9b700 1 -- 172.21.15.67:0/1045662869 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f68e819eda0 con 0x7f68e8102a50 2024-05-03T21:17:48.418 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:17:48.414+0000 7f68e6d9d700 1 --2- 172.21.15.67:0/1045662869 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f68e8106f40 0x7f68e819e660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-03T21:17:48.418 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:17:48.414+0000 7f68e659c700 1 --2- 172.21.15.67:0/1045662869 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f68e81020a0 0x7f68e8199d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-03T21:17:48.418 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:17:48.414+0000 7f68e5d9b700 1 --2- 172.21.15.67:0/1045662869 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f68e8102a50 0x7f68e819a2d0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f68dc00d950 tx=0x7f68dc00dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T21:17:48.418 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:17:48.414+0000 7f68d77fe700 1 -- 172.21.15.67:0/1045662869 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68dc017070 con 0x7f68e8102a50 2024-05-03T21:17:48.418 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:17:48.414+0000 7f68d77fe700 1 -- 172.21.15.67:0/1045662869 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(28 keys) ==== 1170+0+0 (secure 0 0 0) 0x7f68dc009e30 con 0x7f68e8102a50 2024-05-03T21:17:48.418 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:17:48.414+0000 7f68eca40700 1 -- 172.21.15.67:0/1045662869 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f68e819f090 con 0x7f68e8102a50 2024-05-03T21:17:48.418 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:17:48.414+0000 7f68d77fe700 1 -- 172.21.15.67:0/1045662869 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68dc00f620 con 0x7f68e8102a50 2024-05-03T21:17:48.418 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:17:48.414+0000 7f68eca40700 1 -- 172.21.15.67:0/1045662869 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f68e81a6910 con 0x7f68e8102a50 2024-05-03T21:17:48.418 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:17:48.415+0000 7f68eca40700 1 -- 172.21.15.67:0/1045662869 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f68e806a190 con 0x7f68e8102a50 2024-05-03T21:17:48.419 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:17:48.417+0000 7f68d77fe700 1 -- 172.21.15.67:0/1045662869 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 16) ==== 149652+0+0 (secure 0 0 0) 0x7f68dc01a070 con 0x7f68e8102a50 2024-05-03T21:17:48.419 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:17:48.417+0000 7f68d77fe700 1 --2- 172.21.15.67:0/1045662869 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f68d00b2880 0x7f68d00b4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:17:48.419 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:17:48.417+0000 7f68e659c700 1 --2- 172.21.15.67:0/1045662869 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f68d00b2880 0x7f68d00b4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:17:48.419 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:17:48.418+0000 7f68d77fe700 1 -- 172.21.15.67:0/1045662869 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(75..75 src has 1..75) ==== 7127+0+0 (secure 0 0 0) 0x7f68dc0e15a0 con 0x7f68e8102a50 2024-05-03T21:17:48.419 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:17:48.418+0000 7f68e659c700 1 --2- 172.21.15.67:0/1045662869 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f68d00b2880 0x7f68d00b4d40 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f68d80098e0 tx=0x7f68d8005960 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T21:17:48.422 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:17:48.421+0000 7f68d77fe700 1 -- 172.21.15.67:0/1045662869 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191839 (secure 0 0 0) 0x7f68dc0a9760 con 0x7f68e8102a50 2024-05-03T21:17:48.632 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:17:48.631+0000 7f68eca40700 1 -- 172.21.15.67:0/1045662869 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f68e802e510 con 0x7f68e8102a50 2024-05-03T21:17:48.633 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:17:48.632+0000 7f68d77fe700 1 -- 172.21.15.67:0/1045662869 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v75) ==== 93+0+10 (secure 0 0 0) 0x7f68dc0a9060 con 0x7f68e8102a50 2024-05-03T21:17:48.633 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 1 2024-05-03T21:17:48.636 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:17:48.634+0000 7f68eca40700 1 -- 172.21.15.67:0/1045662869 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f68d00b2880 msgr2=0x7f68d00b4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:17:48.636 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:17:48.634+0000 7f68eca40700 1 --2- 172.21.15.67:0/1045662869 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f68d00b2880 0x7f68d00b4d40 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f68d80098e0 tx=0x7f68d8005960 comp rx=0 tx=0).stop 2024-05-03T21:17:48.636 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:17:48.634+0000 7f68eca40700 1 -- 172.21.15.67:0/1045662869 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f68e8102a50 msgr2=0x7f68e819a2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:17:48.636 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:17:48.634+0000 7f68eca40700 1 --2- 172.21.15.67:0/1045662869 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f68e8102a50 0x7f68e819a2d0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f68dc00d950 tx=0x7f68dc00dc60 comp rx=0 tx=0).stop 2024-05-03T21:17:48.636 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:17:48.634+0000 7f68eca40700 1 -- 172.21.15.67:0/1045662869 shutdown_connections 2024-05-03T21:17:48.636 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:17:48.634+0000 7f68eca40700 1 --2- 172.21.15.67:0/1045662869 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f68d00b2880 0x7f68d00b4d40 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:17:48.636 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:17:48.634+0000 7f68eca40700 1 --2- 172.21.15.67:0/1045662869 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f68e8106f40 0x7f68e819e660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:17:48.636 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:17:48.634+0000 7f68eca40700 1 --2- 172.21.15.67:0/1045662869 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f68e81020a0 0x7f68e8199d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:17:48.636 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:17:48.634+0000 7f68eca40700 1 --2- 172.21.15.67:0/1045662869 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f68e8102a50 0x7f68e819a2d0 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:17:48.636 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:17:48.634+0000 7f68eca40700 1 -- 172.21.15.67:0/1045662869 >> 172.21.15.67:0/1045662869 conn(0x7f68e80776c0 msgr2=0x7f68e80ffe50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T21:17:48.636 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:17:48.634+0000 7f68eca40700 1 -- 172.21.15.67:0/1045662869 shutdown_connections 2024-05-03T21:17:48.636 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:17:48.634+0000 7f68eca40700 1 -- 172.21.15.67:0/1045662869 wait complete. 2024-05-03T21:17:49.289 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:17:49 smithi067 ceph-mon[29730]: pgmap v252: 1 pgs: 1 active+clean; 577 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-03T21:17:49.289 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:17:49 smithi067 ceph-mon[29730]: from='client.? 172.21.15.67:0/1045662869' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-05-03T21:17:49.316 INFO:tasks.cephadm:Adding mds.a on smithi067 2024-05-03T21:17:49.317 INFO:tasks.cephadm:Adding mds.d on smithi067 2024-05-03T21:17:49.317 INFO:tasks.cephadm:Adding mds.g on smithi067 2024-05-03T21:17:49.317 INFO:tasks.cephadm:Adding mds.j on smithi067 2024-05-03T21:17:49.317 INFO:tasks.cephadm:Adding mds.b on smithi082 2024-05-03T21:17:49.317 INFO:tasks.cephadm:Adding mds.e on smithi082 2024-05-03T21:17:49.317 INFO:tasks.cephadm:Adding mds.h on smithi082 2024-05-03T21:17:49.317 INFO:tasks.cephadm:Adding mds.k on smithi082 2024-05-03T21:17:49.317 INFO:tasks.cephadm:Adding mds.c on smithi138 2024-05-03T21:17:49.317 INFO:tasks.cephadm:Adding mds.f on smithi138 2024-05-03T21:17:49.317 INFO:tasks.cephadm:Adding mds.i on smithi138 2024-05-03T21:17:49.317 INFO:tasks.cephadm:Adding mds.l on smithi138 2024-05-03T21:17:49.318 DEBUG:teuthology.orchestra.run.smithi138:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9a047fe7599a549042d06a3fb5b6813a203eabea shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 713bc57a-0991-11ef-bc97-c7b262605968 -- ceph orch apply mds all '12;smithi067=a;smithi067=d;smithi067=g;smithi067=j;smithi082=b;smithi082=e;smithi082=h;smithi082=k;smithi138=c;smithi138=f;smithi138=i;smithi138=l' 2024-05-03T21:17:49.324 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:17:49 smithi138 ceph-mon[33110]: pgmap v252: 1 pgs: 1 active+clean; 577 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-03T21:17:49.324 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:17:49 smithi138 ceph-mon[33110]: from='client.? 172.21.15.67:0/1045662869' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-05-03T21:17:49.495 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:17:49 smithi082 ceph-mon[32933]: pgmap v252: 1 pgs: 1 active+clean; 577 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-03T21:17:49.495 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:17:49 smithi082 ceph-mon[32933]: from='client.? 172.21.15.67:0/1045662869' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-05-03T21:17:49.640 INFO:teuthology.orchestra.run.smithi138.stderr:Inferring config /var/lib/ceph/713bc57a-0991-11ef-bc97-c7b262605968/mon.c/config 2024-05-03T21:17:51.178 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:17:51.176+0000 7fd5509d0700 1 -- 172.21.15.138:0/2185566704 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fd54c103fc0 msgr2=0x7fd54c1063b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:17:51.178 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:17:51.176+0000 7fd5509d0700 1 --2- 172.21.15.138:0/2185566704 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fd54c103fc0 0x7fd54c1063b0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fd53c009990 tx=0x7fd53c009ca0 comp rx=0 tx=0).stop 2024-05-03T21:17:51.178 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:17:51.176+0000 7fd5509d0700 1 -- 172.21.15.138:0/2185566704 shutdown_connections 2024-05-03T21:17:51.178 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:17:51.176+0000 7fd5509d0700 1 --2- 172.21.15.138:0/2185566704 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fd54c109220 0x7fd54c10b690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:17:51.178 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:17:51.176+0000 7fd5509d0700 1 --2- 172.21.15.138:0/2185566704 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fd54c103fc0 0x7fd54c1063b0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:17:51.178 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:17:51.176+0000 7fd5509d0700 1 --2- 172.21.15.138:0/2185566704 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd54c1068f0 0x7fd54c108ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:17:51.179 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:17:51.176+0000 7fd5509d0700 1 -- 172.21.15.138:0/2185566704 >> 172.21.15.138:0/2185566704 conn(0x7fd54c0fd9f0 msgr2=0x7fd54c0ffe10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T21:17:51.179 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:17:51.176+0000 7fd5509d0700 1 -- 172.21.15.138:0/2185566704 shutdown_connections 2024-05-03T21:17:51.179 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:17:51.176+0000 7fd5509d0700 1 -- 172.21.15.138:0/2185566704 wait complete. 2024-05-03T21:17:51.181 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:17:51.177+0000 7fd5509d0700 1 Processor -- start 2024-05-03T21:17:51.181 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:17:51.177+0000 7fd5509d0700 1 -- start start 2024-05-03T21:17:51.181 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:17:51.177+0000 7fd5509d0700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd54c103fc0 0x7fd54c199760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:17:51.181 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:17:51.177+0000 7fd5509d0700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fd54c1068f0 0x7fd54c199ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:17:51.181 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:17:51.177+0000 7fd5509d0700 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fd54c109220 0x7fd54c1a0d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:17:51.181 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:17:51.177+0000 7fd5509d0700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fd54c10f2c0 con 0x7fd54c103fc0 2024-05-03T21:17:51.181 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:17:51.177+0000 7fd5509d0700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7fd54c10f140 con 0x7fd54c1068f0 2024-05-03T21:17:51.181 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:17:51.177+0000 7fd5509d0700 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7fd54c10f440 con 0x7fd54c109220 2024-05-03T21:17:51.182 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:17:51.177+0000 7fd54ad9d700 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fd54c109220 0x7fd54c1a0d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:17:51.182 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:17:51.177+0000 7fd54ad9d700 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fd54c109220 0x7fd54c1a0d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3300/0 says I am v2:172.21.15.138:54760/0 (socket says 172.21.15.138:54760) 2024-05-03T21:17:51.182 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:17:51.177+0000 7fd54ad9d700 1 -- 172.21.15.138:0/2621169113 learned_addr learned my addr 172.21.15.138:0/2621169113 (peer_addr_for_me v2:172.21.15.138:0/0) 2024-05-03T21:17:51.182 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:17:51.177+0000 7fd54a59c700 1 --2- 172.21.15.138:0/2621169113 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd54c103fc0 0x7fd54c199760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:17:51.182 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:17:51.177+0000 7fd541d9b700 1 --2- 172.21.15.138:0/2621169113 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fd54c1068f0 0x7fd54c199ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:17:51.182 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:17:51.177+0000 7fd54ad9d700 1 -- 172.21.15.138:0/2621169113 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fd54c1068f0 msgr2=0x7fd54c199ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:17:51.182 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:17:51.177+0000 7fd54ad9d700 1 --2- 172.21.15.138:0/2621169113 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fd54c1068f0 0x7fd54c199ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:17:51.182 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:17:51.177+0000 7fd54ad9d700 1 -- 172.21.15.138:0/2621169113 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd54c103fc0 msgr2=0x7fd54c199760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:17:51.182 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:17:51.177+0000 7fd54ad9d700 1 --2- 172.21.15.138:0/2621169113 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd54c103fc0 0x7fd54c199760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:17:51.182 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:17:51.177+0000 7fd54ad9d700 1 -- 172.21.15.138:0/2621169113 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd54c1a13c0 con 0x7fd54c109220 2024-05-03T21:17:51.182 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:17:51.178+0000 7fd54ad9d700 1 --2- 172.21.15.138:0/2621169113 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fd54c109220 0x7fd54c1a0d20 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fd54400ba10 tx=0x7fd54400bd20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T21:17:51.182 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:17:51.178+0000 7fd543fff700 1 -- 172.21.15.138:0/2621169113 <== mon.1 v2:172.21.15.138:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd544019070 con 0x7fd54c109220 2024-05-03T21:17:51.182 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:17:51.178+0000 7fd5509d0700 1 -- 172.21.15.138:0/2621169113 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd54c1a16b0 con 0x7fd54c109220 2024-05-03T21:17:51.182 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:17:51.178+0000 7fd543fff700 1 -- 172.21.15.138:0/2621169113 <== mon.1 v2:172.21.15.138:3300/0 2 ==== config(28 keys) ==== 1170+0+0 (secure 0 0 0) 0x7fd54400ce30 con 0x7fd54c109220 2024-05-03T21:17:51.183 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:17:51.178+0000 7fd543fff700 1 -- 172.21.15.138:0/2621169113 <== mon.1 v2:172.21.15.138:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5440125a0 con 0x7fd54c109220 2024-05-03T21:17:51.183 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:17:51.178+0000 7fd5509d0700 1 -- 172.21.15.138:0/2621169113 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd54c1a1c40 con 0x7fd54c109220 2024-05-03T21:17:51.183 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:17:51.179+0000 7fd5509d0700 1 -- 172.21.15.138:0/2621169113 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd54c06a190 con 0x7fd54c109220 2024-05-03T21:17:51.183 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:17:51.180+0000 7fd543fff700 1 -- 172.21.15.138:0/2621169113 <== mon.1 v2:172.21.15.138:3300/0 4 ==== mgrmap(e 16) ==== 149652+0+0 (secure 0 0 0) 0x7fd54401c070 con 0x7fd54c109220 2024-05-03T21:17:51.184 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:17:51.181+0000 7fd543fff700 1 --2- 172.21.15.138:0/2621169113 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7fd5380b2880 0x7fd5380b4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:17:51.184 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:17:51.181+0000 7fd543fff700 1 -- 172.21.15.138:0/2621169113 <== mon.1 v2:172.21.15.138:3300/0 5 ==== osd_map(75..75 src has 1..75) ==== 7127+0+0 (secure 0 0 0) 0x7fd5440e1ae0 con 0x7fd54c109220 2024-05-03T21:17:51.184 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:17:51.181+0000 7fd54a59c700 1 --2- 172.21.15.138:0/2621169113 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7fd5380b2880 0x7fd5380b4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:17:51.184 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:17:51.181+0000 7fd54a59c700 1 --2- 172.21.15.138:0/2621169113 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7fd5380b2880 0x7fd5380b4d40 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fd53c00b5c0 tx=0x7fd53c005fd0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T21:17:51.186 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:17:51.184+0000 7fd543fff700 1 -- 172.21.15.138:0/2621169113 <== mon.1 v2:172.21.15.138:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191839 (secure 0 0 0) 0x7fd544023070 con 0x7fd54c109220 2024-05-03T21:17:51.324 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:17:51 smithi138 ceph-mon[33110]: pgmap v253: 1 pgs: 1 active+clean; 577 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-03T21:17:51.404 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:17:51.402+0000 7fd5509d0700 1 -- 172.21.15.138:0/2621169113 --> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] -- mgr_command(tid 0: {"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi067=a;smithi067=d;smithi067=g;smithi067=j;smithi082=b;smithi082=e;smithi082=h;smithi082=k;smithi138=c;smithi138=f;smithi138=i;smithi138=l", "target": ["mon-mgr", ""]}) -- 0x7fd54c0639e0 con 0x7fd5380b2880 2024-05-03T21:17:51.412 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:17:51.411+0000 7fd543fff700 1 -- 172.21.15.138:0/2621169113 <== mgr.14150 v2:172.21.15.67:6800/604451592 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+28 (secure 0 0 0) 0x7fd54c0639e0 con 0x7fd5380b2880 2024-05-03T21:17:51.412 INFO:teuthology.orchestra.run.smithi138.stdout:Scheduled mds.all update... 2024-05-03T21:17:51.415 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:17:51.413+0000 7fd5509d0700 1 -- 172.21.15.138:0/2621169113 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7fd5380b2880 msgr2=0x7fd5380b4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:17:51.415 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:17:51.413+0000 7fd5509d0700 1 --2- 172.21.15.138:0/2621169113 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7fd5380b2880 0x7fd5380b4d40 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fd53c00b5c0 tx=0x7fd53c005fd0 comp rx=0 tx=0).stop 2024-05-03T21:17:51.415 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:17:51.413+0000 7fd5509d0700 1 -- 172.21.15.138:0/2621169113 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fd54c109220 msgr2=0x7fd54c1a0d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:17:51.415 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:17:51.413+0000 7fd5509d0700 1 --2- 172.21.15.138:0/2621169113 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fd54c109220 0x7fd54c1a0d20 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fd54400ba10 tx=0x7fd54400bd20 comp rx=0 tx=0).stop 2024-05-03T21:17:51.415 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:17:51.413+0000 7fd5509d0700 1 -- 172.21.15.138:0/2621169113 shutdown_connections 2024-05-03T21:17:51.415 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:17:51.413+0000 7fd5509d0700 1 --2- 172.21.15.138:0/2621169113 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7fd5380b2880 0x7fd5380b4d40 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:17:51.415 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:17:51.413+0000 7fd5509d0700 1 --2- 172.21.15.138:0/2621169113 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd54c103fc0 0x7fd54c199760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:17:51.415 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:17:51.413+0000 7fd5509d0700 1 --2- 172.21.15.138:0/2621169113 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fd54c109220 0x7fd54c1a0d20 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:17:51.416 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:17:51.413+0000 7fd5509d0700 1 --2- 172.21.15.138:0/2621169113 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fd54c1068f0 0x7fd54c199ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:17:51.416 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:17:51.413+0000 7fd5509d0700 1 -- 172.21.15.138:0/2621169113 >> 172.21.15.138:0/2621169113 conn(0x7fd54c0fd9f0 msgr2=0x7fd54c104900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T21:17:51.416 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:17:51.413+0000 7fd5509d0700 1 -- 172.21.15.138:0/2621169113 shutdown_connections 2024-05-03T21:17:51.416 INFO:teuthology.orchestra.run.smithi138.stderr:2024-05-03T21:17:51.413+0000 7fd5509d0700 1 -- 172.21.15.138:0/2621169113 wait complete. 2024-05-03T21:17:51.471 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:17:51 smithi067 ceph-mon[29730]: pgmap v253: 1 pgs: 1 active+clean; 577 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-03T21:17:51.495 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:17:51 smithi082 ceph-mon[32933]: pgmap v253: 1 pgs: 1 active+clean; 577 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-03T21:17:51.996 DEBUG:teuthology.orchestra.run.smithi067:mds.a> sudo journalctl -f -n 0 -u ceph-713bc57a-0991-11ef-bc97-c7b262605968@mds.a.service 2024-05-03T21:17:51.998 DEBUG:teuthology.orchestra.run.smithi067:mds.d> sudo journalctl -f -n 0 -u ceph-713bc57a-0991-11ef-bc97-c7b262605968@mds.d.service 2024-05-03T21:17:52.041 DEBUG:teuthology.orchestra.run.smithi067:mds.g> sudo journalctl -f -n 0 -u ceph-713bc57a-0991-11ef-bc97-c7b262605968@mds.g.service 2024-05-03T21:17:52.084 DEBUG:teuthology.orchestra.run.smithi067:mds.j> sudo journalctl -f -n 0 -u ceph-713bc57a-0991-11ef-bc97-c7b262605968@mds.j.service 2024-05-03T21:17:52.127 DEBUG:teuthology.orchestra.run.smithi082:mds.b> sudo journalctl -f -n 0 -u ceph-713bc57a-0991-11ef-bc97-c7b262605968@mds.b.service 2024-05-03T21:17:52.129 DEBUG:teuthology.orchestra.run.smithi082:mds.e> sudo journalctl -f -n 0 -u ceph-713bc57a-0991-11ef-bc97-c7b262605968@mds.e.service 2024-05-03T21:17:52.173 DEBUG:teuthology.orchestra.run.smithi082:mds.h> sudo journalctl -f -n 0 -u ceph-713bc57a-0991-11ef-bc97-c7b262605968@mds.h.service 2024-05-03T21:17:52.218 DEBUG:teuthology.orchestra.run.smithi082:mds.k> sudo journalctl -f -n 0 -u ceph-713bc57a-0991-11ef-bc97-c7b262605968@mds.k.service 2024-05-03T21:17:52.261 DEBUG:teuthology.orchestra.run.smithi138:mds.c> sudo journalctl -f -n 0 -u ceph-713bc57a-0991-11ef-bc97-c7b262605968@mds.c.service 2024-05-03T21:17:52.263 DEBUG:teuthology.orchestra.run.smithi138:mds.f> sudo journalctl -f -n 0 -u ceph-713bc57a-0991-11ef-bc97-c7b262605968@mds.f.service 2024-05-03T21:17:52.306 DEBUG:teuthology.orchestra.run.smithi138:mds.i> sudo journalctl -f -n 0 -u ceph-713bc57a-0991-11ef-bc97-c7b262605968@mds.i.service 2024-05-03T21:17:52.349 DEBUG:teuthology.orchestra.run.smithi138:mds.l> sudo journalctl -f -n 0 -u ceph-713bc57a-0991-11ef-bc97-c7b262605968@mds.l.service 2024-05-03T21:17:52.392 INFO:tasks.cephadm:Setting up CephFS filesystem(s)... 2024-05-03T21:17:52.393 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9a047fe7599a549042d06a3fb5b6813a203eabea shell --fsid 713bc57a-0991-11ef-bc97-c7b262605968 -- ceph fs dump --format=json 2024-05-03T21:17:52.575 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:17:52 smithi138 ceph-mon[33110]: from='client.24367 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi067=a;smithi067=d;smithi067=g;smithi067=j;smithi082=b;smithi082=e;smithi082=h;smithi082=k;smithi138=c;smithi138=f;smithi138=i;smithi138=l", "target": ["mon-mgr", ""]}]: dispatch 2024-05-03T21:17:52.575 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:17:52 smithi138 ceph-mon[33110]: Saving service mds.all spec with placement smithi067=a;smithi067=d;smithi067=g;smithi067=j;smithi082=b;smithi082=e;smithi082=h;smithi082=k;smithi138=c;smithi138=f;smithi138=i;smithi138=l;count:12 2024-05-03T21:17:52.575 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:17:52 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:17:52.575 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:17:52 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T21:17:52.575 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:17:52 smithi138 ceph-mon[33110]: pgmap v254: 1 pgs: 1 active+clean; 577 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-03T21:17:52.575 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:17:52 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T21:17:52.575 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:17:52 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T21:17:52.575 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:17:52 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:17:52.575 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:17:52 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-03T21:17:52.575 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:17:52 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-03T21:17:52.575 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:17:52 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T21:17:52.575 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:17:52 smithi138 ceph-mon[33110]: Deploying daemon mds.l on smithi138 2024-05-03T21:17:52.637 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/713bc57a-0991-11ef-bc97-c7b262605968/mon.a/config 2024-05-03T21:17:52.721 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:17:52 smithi067 ceph-mon[29730]: from='client.24367 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi067=a;smithi067=d;smithi067=g;smithi067=j;smithi082=b;smithi082=e;smithi082=h;smithi082=k;smithi138=c;smithi138=f;smithi138=i;smithi138=l", "target": ["mon-mgr", ""]}]: dispatch 2024-05-03T21:17:52.721 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:17:52 smithi067 ceph-mon[29730]: Saving service mds.all spec with placement smithi067=a;smithi067=d;smithi067=g;smithi067=j;smithi082=b;smithi082=e;smithi082=h;smithi082=k;smithi138=c;smithi138=f;smithi138=i;smithi138=l;count:12 2024-05-03T21:17:52.721 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:17:52 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:17:52.721 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:17:52 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T21:17:52.722 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:17:52 smithi067 ceph-mon[29730]: pgmap v254: 1 pgs: 1 active+clean; 577 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-03T21:17:52.722 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:17:52 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T21:17:52.722 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:17:52 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T21:17:52.722 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:17:52 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:17:52.722 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:17:52 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-03T21:17:52.722 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:17:52 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-03T21:17:52.722 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:17:52 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T21:17:52.722 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:17:52 smithi067 ceph-mon[29730]: Deploying daemon mds.l on smithi138 2024-05-03T21:17:52.745 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:17:52 smithi082 ceph-mon[32933]: from='client.24367 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi067=a;smithi067=d;smithi067=g;smithi067=j;smithi082=b;smithi082=e;smithi082=h;smithi082=k;smithi138=c;smithi138=f;smithi138=i;smithi138=l", "target": ["mon-mgr", ""]}]: dispatch 2024-05-03T21:17:52.746 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:17:52 smithi082 ceph-mon[32933]: Saving service mds.all spec with placement smithi067=a;smithi067=d;smithi067=g;smithi067=j;smithi082=b;smithi082=e;smithi082=h;smithi082=k;smithi138=c;smithi138=f;smithi138=i;smithi138=l;count:12 2024-05-03T21:17:52.746 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:17:52 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:17:52.746 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:17:52 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T21:17:52.746 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:17:52 smithi082 ceph-mon[32933]: pgmap v254: 1 pgs: 1 active+clean; 577 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-03T21:17:52.746 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:17:52 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T21:17:52.746 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:17:52 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T21:17:52.746 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:17:52 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:17:52.746 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:17:52 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-03T21:17:52.746 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:17:52 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-03T21:17:52.746 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:17:52 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T21:17:52.746 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:17:52 smithi082 ceph-mon[32933]: Deploying daemon mds.l on smithi138 2024-05-03T21:17:54.074 INFO:journalctl@ceph.mds.l.smithi138.stdout:May 03 21:17:54 smithi138 systemd[1]: Starting Ceph mds.l for 713bc57a-0991-11ef-bc97-c7b262605968... 2024-05-03T21:17:54.366 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:17:54.362+0000 7f3fbb813700 1 -- 172.21.15.67:0/943849676 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3fb4108880 msgr2=0x7f3fb4110400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:17:54.366 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:17:54.362+0000 7f3fbb813700 1 --2- 172.21.15.67:0/943849676 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3fb4108880 0x7f3fb4110400 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f3fac0099c0 tx=0x7f3fac009cd0 comp rx=0 tx=0).stop 2024-05-03T21:17:54.366 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:17:54.362+0000 7f3fbb813700 1 -- 172.21.15.67:0/943849676 shutdown_connections 2024-05-03T21:17:54.367 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:17:54.362+0000 7f3fbb813700 1 --2- 172.21.15.67:0/943849676 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3fb4108880 0x7f3fb4110400 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:17:54.367 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:17:54.362+0000 7f3fbb813700 1 --2- 172.21.15.67:0/943849676 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f3fb41037b0 0x7f3fb4103b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:17:54.367 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:17:54.362+0000 7f3fbb813700 1 --2- 172.21.15.67:0/943849676 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f3fb4104160 0x7f3fb4108150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:17:54.367 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:17:54.362+0000 7f3fbb813700 1 -- 172.21.15.67:0/943849676 >> 172.21.15.67:0/943849676 conn(0x7f3fb407a530 msgr2=0x7f3fb407a940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T21:17:54.367 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:17:54.362+0000 7f3fbb813700 1 -- 172.21.15.67:0/943849676 shutdown_connections 2024-05-03T21:17:54.367 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:17:54.362+0000 7f3fbb813700 1 -- 172.21.15.67:0/943849676 wait complete. 2024-05-03T21:17:54.367 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:17:54.363+0000 7f3fbb813700 1 Processor -- start 2024-05-03T21:17:54.367 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:17:54.363+0000 7f3fbb813700 1 -- start start 2024-05-03T21:17:54.367 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:17:54.363+0000 7f3fbb813700 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f3fb41037b0 0x7f3fb40757a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:17:54.367 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:17:54.363+0000 7f3fbb813700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3fb4104160 0x7f3fb4075ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:17:54.367 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:17:54.363+0000 7f3fbb813700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f3fb4108880 0x7f3fb4076220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:17:54.367 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:17:54.363+0000 7f3fbb813700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f3fb4113970 con 0x7f3fb4104160 2024-05-03T21:17:54.368 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:17:54.363+0000 7f3fbb813700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f3fb41137f0 con 0x7f3fb4108880 2024-05-03T21:17:54.368 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:17:54.363+0000 7f3fbb813700 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f3fb4113af0 con 0x7f3fb41037b0 2024-05-03T21:17:54.368 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:17:54.363+0000 7f3fb8dae700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3fb4104160 0x7f3fb4075ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:17:54.368 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:17:54.363+0000 7f3fb8dae700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3fb4104160 0x7f3fb4075ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:50274/0 (socket says 172.21.15.67:50274) 2024-05-03T21:17:54.368 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:17:54.363+0000 7f3fb8dae700 1 -- 172.21.15.67:0/2059830268 learned_addr learned my addr 172.21.15.67:0/2059830268 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-03T21:17:54.368 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:17:54.363+0000 7f3fb95af700 1 --2- 172.21.15.67:0/2059830268 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f3fb41037b0 0x7f3fb40757a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:17:54.368 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:17:54.363+0000 7f3fb9db0700 1 --2- 172.21.15.67:0/2059830268 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f3fb4108880 0x7f3fb4076220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:17:54.368 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:17:54.363+0000 7f3fb8dae700 1 -- 172.21.15.67:0/2059830268 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f3fb41037b0 msgr2=0x7f3fb40757a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:17:54.368 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:17:54.363+0000 7f3fb8dae700 1 --2- 172.21.15.67:0/2059830268 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f3fb41037b0 0x7f3fb40757a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:17:54.368 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:17:54.363+0000 7f3fb8dae700 1 -- 172.21.15.67:0/2059830268 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f3fb4108880 msgr2=0x7f3fb4076220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:17:54.368 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:17:54.363+0000 7f3fb8dae700 1 --2- 172.21.15.67:0/2059830268 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f3fb4108880 0x7f3fb4076220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:17:54.368 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:17:54.363+0000 7f3fb8dae700 1 -- 172.21.15.67:0/2059830268 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3fb41a8280 con 0x7f3fb4104160 2024-05-03T21:17:54.368 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:17:54.363+0000 7f3fb8dae700 1 --2- 172.21.15.67:0/2059830268 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3fb4104160 0x7f3fb4075ce0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f3fa800b550 tx=0x7f3fa800b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T21:17:54.368 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:17:54.364+0000 7f3fb27fc700 1 -- 172.21.15.67:0/2059830268 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3fa8015020 con 0x7f3fb4104160 2024-05-03T21:17:54.368 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:17:54.364+0000 7f3fb27fc700 1 -- 172.21.15.67:0/2059830268 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(28 keys) ==== 1170+0+0 (secure 0 0 0) 0x7f3fa8011d30 con 0x7f3fb4104160 2024-05-03T21:17:54.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:17:54.364+0000 7f3fbb813700 1 -- 172.21.15.67:0/2059830268 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3fb41a8570 con 0x7f3fb4104160 2024-05-03T21:17:54.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:17:54.364+0000 7f3fb27fc700 1 -- 172.21.15.67:0/2059830268 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3fa8010e00 con 0x7f3fb4104160 2024-05-03T21:17:54.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:17:54.364+0000 7f3fbb813700 1 -- 172.21.15.67:0/2059830268 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3fb41a8ab0 con 0x7f3fb4104160 2024-05-03T21:17:54.370 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:17:54.365+0000 7f3fbb813700 1 -- 172.21.15.67:0/2059830268 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3fb406a190 con 0x7f3fb4104160 2024-05-03T21:17:54.370 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:17:54.366+0000 7f3fb27fc700 1 -- 172.21.15.67:0/2059830268 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 16) ==== 149652+0+0 (secure 0 0 0) 0x7f3fa8018020 con 0x7f3fb4104160 2024-05-03T21:17:54.370 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:17:54.367+0000 7f3fb27fc700 1 --2- 172.21.15.67:0/2059830268 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f3f9c0b2880 0x7f3f9c0b4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:17:54.370 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:17:54.367+0000 7f3fb95af700 1 --2- 172.21.15.67:0/2059830268 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f3f9c0b2880 0x7f3f9c0b4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:17:54.370 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:17:54.367+0000 7f3fb27fc700 1 -- 172.21.15.67:0/2059830268 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(75..75 src has 1..75) ==== 7127+0+0 (secure 0 0 0) 0x7f3fa80e1a70 con 0x7f3fb4104160 2024-05-03T21:17:54.370 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:17:54.368+0000 7f3fb95af700 1 --2- 172.21.15.67:0/2059830268 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f3f9c0b2880 0x7f3f9c0b4d40 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f3fa4009550 tx=0x7f3fa40091b0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T21:17:54.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:17:54.371+0000 7f3fb27fc700 1 -- 172.21.15.67:0/2059830268 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191839 (secure 0 0 0) 0x7f3fa801f020 con 0x7f3fb4104160 2024-05-03T21:17:54.491 INFO:journalctl@ceph.mds.l.smithi138.stdout:May 03 21:17:54 smithi138 podman[62798]: 2024-05-03 21:17:54.121389224 +0000 UTC m=+0.017776804 image pull ee4d846202496ba6096593e8e8becc0700a5eb7ebd95536031187f5f337d4012 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:68e15ae152ef5c805c733e76da8fc1d3dd4ee669d5ba025f457e05725baa24dd 2024-05-03T21:17:54.491 INFO:journalctl@ceph.mds.l.smithi138.stdout:May 03 21:17:54 smithi138 podman[62798]: 2024-05-03 21:17:54.235364045 +0000 UTC m=+0.131751661 container create 5f5854e648980748e880e5dde6d9246965c6d3619f3d86924e319aa22490960d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:68e15ae152ef5c805c733e76da8fc1d3dd4ee669d5ba025f457e05725baa24dd, name=ceph-713bc57a-0991-11ef-bc97-c7b262605968-mds-l, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240430, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_BRANCH=HEAD, RELEASE=wip-rishabh-testing-20240501.193033-9a047fe, org.label-schema.vendor=CentOS, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b) 2024-05-03T21:17:54.491 INFO:journalctl@ceph.mds.l.smithi138.stdout:May 03 21:17:54 smithi138 podman[62798]: 2024-05-03 21:17:54.463274556 +0000 UTC m=+0.359662127 container init 5f5854e648980748e880e5dde6d9246965c6d3619f3d86924e319aa22490960d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:68e15ae152ef5c805c733e76da8fc1d3dd4ee669d5ba025f457e05725baa24dd, name=ceph-713bc57a-0991-11ef-bc97-c7b262605968-mds-l, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240430, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_CLEAN=True, RELEASE=wip-rishabh-testing-20240501.193033-9a047fe) 2024-05-03T21:17:54.491 INFO:journalctl@ceph.mds.l.smithi138.stdout:May 03 21:17:54 smithi138 podman[62798]: 2024-05-03 21:17:54.466091006 +0000 UTC m=+0.362478578 container start 5f5854e648980748e880e5dde6d9246965c6d3619f3d86924e319aa22490960d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:68e15ae152ef5c805c733e76da8fc1d3dd4ee669d5ba025f457e05725baa24dd, name=ceph-713bc57a-0991-11ef-bc97-c7b262605968-mds-l, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, RELEASE=wip-rishabh-testing-20240501.193033-9a047fe, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.license=GPLv2, ceph=True, org.label-schema.build-date=20240430, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True) 2024-05-03T21:17:54.636 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:17:54.634+0000 7f3fbb813700 1 -- 172.21.15.67:0/2059830268 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f3fb4076c80 con 0x7f3fb4104160 2024-05-03T21:17:54.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:17:54.634+0000 7f3fb27fc700 1 -- 172.21.15.67:0/2059830268 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 2 v2) ==== 93+0+1396 (secure 0 0 0) 0x7f3fa80a9c10 con 0x7f3fb4104160 2024-05-03T21:17:54.637 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-05-03T21:17:54.637 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":2,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24376,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.138:6833/3028022602","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6832","nonce":3028022602},{"type":"v1","addr":"172.21.15.138:6833","nonce":3028022602}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[]} 2024-05-03T21:17:54.639 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:17:54.637+0000 7f3fbb813700 1 -- 172.21.15.67:0/2059830268 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f3f9c0b2880 msgr2=0x7f3f9c0b4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:17:54.639 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:17:54.637+0000 7f3fbb813700 1 --2- 172.21.15.67:0/2059830268 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f3f9c0b2880 0x7f3f9c0b4d40 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f3fa4009550 tx=0x7f3fa40091b0 comp rx=0 tx=0).stop 2024-05-03T21:17:54.639 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:17:54.637+0000 7f3fbb813700 1 -- 172.21.15.67:0/2059830268 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3fb4104160 msgr2=0x7f3fb4075ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:17:54.639 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:17:54.637+0000 7f3fbb813700 1 --2- 172.21.15.67:0/2059830268 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3fb4104160 0x7f3fb4075ce0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f3fa800b550 tx=0x7f3fa800b860 comp rx=0 tx=0).stop 2024-05-03T21:17:54.639 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:17:54.637+0000 7f3fbb813700 1 -- 172.21.15.67:0/2059830268 shutdown_connections 2024-05-03T21:17:54.640 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:17:54.637+0000 7f3fbb813700 1 --2- 172.21.15.67:0/2059830268 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f3f9c0b2880 0x7f3f9c0b4d40 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:17:54.640 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:17:54.637+0000 7f3fbb813700 1 --2- 172.21.15.67:0/2059830268 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f3fb4108880 0x7f3fb4076220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:17:54.640 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:17:54.637+0000 7f3fbb813700 1 --2- 172.21.15.67:0/2059830268 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f3fb41037b0 0x7f3fb40757a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:17:54.640 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:17:54.637+0000 7f3fbb813700 1 --2- 172.21.15.67:0/2059830268 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3fb4104160 0x7f3fb4075ce0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:17:54.640 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:17:54.637+0000 7f3fbb813700 1 -- 172.21.15.67:0/2059830268 >> 172.21.15.67:0/2059830268 conn(0x7f3fb407a530 msgr2=0x7f3fb4102650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T21:17:54.640 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:17:54.637+0000 7f3fbb813700 1 -- 172.21.15.67:0/2059830268 shutdown_connections 2024-05-03T21:17:54.640 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:17:54.637+0000 7f3fbb813700 1 -- 172.21.15.67:0/2059830268 wait complete. 2024-05-03T21:17:54.640 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 2 2024-05-03T21:17:54.824 INFO:journalctl@ceph.mds.l.smithi138.stdout:May 03 21:17:54 smithi138 ceph-mds[62817]: -- 172.21.15.138:0/3974147945 <== mon.1 v2:172.21.15.138:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5647ebc33a40 con 0x5647ebc06800 2024-05-03T21:17:54.824 INFO:journalctl@ceph.mds.l.smithi138.stdout:May 03 21:17:54 smithi138 ceph-713bc57a-0991-11ef-bc97-c7b262605968-mds-l[62813]: starting mds.l at 2024-05-03T21:17:54.825 INFO:journalctl@ceph.mds.l.smithi138.stdout:May 03 21:17:54 smithi138 bash[62798]: 5f5854e648980748e880e5dde6d9246965c6d3619f3d86924e319aa22490960d 2024-05-03T21:17:54.825 INFO:journalctl@ceph.mds.l.smithi138.stdout:May 03 21:17:54 smithi138 systemd[1]: Started Ceph mds.l for 713bc57a-0991-11ef-bc97-c7b262605968. 2024-05-03T21:17:54.936 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:17:54 smithi067 ceph-mon[29730]: pgmap v255: 1 pgs: 1 active+clean; 577 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-03T21:17:54.936 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:17:54 smithi067 ceph-mon[29730]: mds.? [v2:172.21.15.138:6832/3028022602,v1:172.21.15.138:6833/3028022602] up:boot 2024-05-03T21:17:54.936 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:17:54 smithi067 ceph-mon[29730]: fsmap 1 up:standby 2024-05-03T21:17:54.936 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:17:54 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-05-03T21:17:54.936 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:17:54 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:17:54.937 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:17:54 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:17:54.937 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:17:54 smithi067 ceph-mon[29730]: from='client.? 172.21.15.67:0/2059830268' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-03T21:17:54.937 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:17:54 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:17:54.937 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:17:54 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-03T21:17:54.937 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:17:54 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-03T21:17:54.937 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:17:54 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T21:17:55.245 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:17:54 smithi082 ceph-mon[32933]: pgmap v255: 1 pgs: 1 active+clean; 577 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-03T21:17:55.246 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:17:54 smithi082 ceph-mon[32933]: mds.? [v2:172.21.15.138:6832/3028022602,v1:172.21.15.138:6833/3028022602] up:boot 2024-05-03T21:17:55.246 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:17:54 smithi082 ceph-mon[32933]: fsmap 1 up:standby 2024-05-03T21:17:55.246 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:17:54 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-05-03T21:17:55.246 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:17:54 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:17:55.246 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:17:54 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:17:55.246 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:17:54 smithi082 ceph-mon[32933]: from='client.? 172.21.15.67:0/2059830268' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-03T21:17:55.246 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:17:54 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:17:55.246 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:17:54 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-03T21:17:55.246 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:17:54 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-03T21:17:55.246 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:17:54 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T21:17:55.324 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:17:54 smithi138 ceph-mon[33110]: pgmap v255: 1 pgs: 1 active+clean; 577 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-03T21:17:55.325 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:17:54 smithi138 ceph-mon[33110]: mds.? [v2:172.21.15.138:6832/3028022602,v1:172.21.15.138:6833/3028022602] up:boot 2024-05-03T21:17:55.325 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:17:54 smithi138 ceph-mon[33110]: fsmap 1 up:standby 2024-05-03T21:17:55.325 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:17:54 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-05-03T21:17:55.325 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:17:54 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:17:55.325 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:17:54 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:17:55.325 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:17:54 smithi138 ceph-mon[33110]: from='client.? 172.21.15.67:0/2059830268' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-03T21:17:55.325 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:17:54 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:17:55.325 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:17:54 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-03T21:17:55.325 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:17:54 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-03T21:17:55.325 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:17:54 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T21:17:56.221 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:17:55 smithi067 ceph-mon[29730]: Deploying daemon mds.k on smithi082 2024-05-03T21:17:56.245 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:17:55 smithi082 ceph-mon[32933]: Deploying daemon mds.k on smithi082 2024-05-03T21:17:56.324 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:17:55 smithi138 ceph-mon[33110]: Deploying daemon mds.k on smithi082 2024-05-03T21:17:56.996 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:17:56 smithi082 ceph-mon[32933]: pgmap v256: 1 pgs: 1 active+clean; 577 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-03T21:17:56.996 INFO:journalctl@ceph.mds.k.smithi082.stdout:May 03 21:17:56 smithi082 systemd[1]: Starting Ceph mds.k for 713bc57a-0991-11ef-bc97-c7b262605968... 2024-05-03T21:17:57.011 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:17:56 smithi067 ceph-mon[29730]: pgmap v256: 1 pgs: 1 active+clean; 577 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-03T21:17:57.324 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:17:56 smithi138 ceph-mon[33110]: pgmap v256: 1 pgs: 1 active+clean; 577 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-03T21:17:57.335 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9a047fe7599a549042d06a3fb5b6813a203eabea shell --fsid 713bc57a-0991-11ef-bc97-c7b262605968 -- ceph fs dump --format=json 2024-05-03T21:17:57.381 INFO:journalctl@ceph.mds.k.smithi082.stdout:May 03 21:17:57 smithi082 podman[62388]: 2024-05-03 21:17:56.979822087 +0000 UTC m=+0.018031819 image pull ee4d846202496ba6096593e8e8becc0700a5eb7ebd95536031187f5f337d4012 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:68e15ae152ef5c805c733e76da8fc1d3dd4ee669d5ba025f457e05725baa24dd 2024-05-03T21:17:57.382 INFO:journalctl@ceph.mds.k.smithi082.stdout:May 03 21:17:57 smithi082 podman[62388]: 2024-05-03 21:17:57.094577243 +0000 UTC m=+0.132786964 container create 9acc39735ce2143db050ee2ca62f4458c112d4ac6be4c8133414b02c5e81bd87 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:68e15ae152ef5c805c733e76da8fc1d3dd4ee669d5ba025f457e05725baa24dd, name=ceph-713bc57a-0991-11ef-bc97-c7b262605968-mds-k, GIT_CLEAN=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, maintainer=Guillaume Abrioux , RELEASE=wip-rishabh-testing-20240501.193033-9a047fe, org.label-schema.license=GPLv2, org.label-schema.build-date=20240430, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0) 2024-05-03T21:17:57.382 INFO:journalctl@ceph.mds.k.smithi082.stdout:May 03 21:17:57 smithi082 podman[62388]: 2024-05-03 21:17:57.280351248 +0000 UTC m=+0.318560971 container init 9acc39735ce2143db050ee2ca62f4458c112d4ac6be4c8133414b02c5e81bd87 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:68e15ae152ef5c805c733e76da8fc1d3dd4ee669d5ba025f457e05725baa24dd, name=ceph-713bc57a-0991-11ef-bc97-c7b262605968-mds-k, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, RELEASE=wip-rishabh-testing-20240501.193033-9a047fe, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240430, org.label-schema.vendor=CentOS) 2024-05-03T21:17:57.382 INFO:journalctl@ceph.mds.k.smithi082.stdout:May 03 21:17:57 smithi082 podman[62388]: 2024-05-03 21:17:57.282837604 +0000 UTC m=+0.321047329 container start 9acc39735ce2143db050ee2ca62f4458c112d4ac6be4c8133414b02c5e81bd87 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:68e15ae152ef5c805c733e76da8fc1d3dd4ee669d5ba025f457e05725baa24dd, name=ceph-713bc57a-0991-11ef-bc97-c7b262605968-mds-k, CEPH_POINT_RELEASE=, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-rishabh-testing-20240501.193033-9a047fe, io.buildah.version=1.33.5, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.license=GPLv2, org.label-schema.build-date=20240430, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0) 2024-05-03T21:17:57.382 INFO:journalctl@ceph.mds.k.smithi082.stdout:May 03 21:17:57 smithi082 ceph-mds[62407]: -- 172.21.15.82:0/3572633189 <== mon.2 v2:172.21.15.82:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55ef471e1a40 con 0x55ef471b5000 2024-05-03T21:17:57.382 INFO:journalctl@ceph.mds.k.smithi082.stdout:May 03 21:17:57 smithi082 ceph-713bc57a-0991-11ef-bc97-c7b262605968-mds-k[62403]: starting mds.k at 2024-05-03T21:17:57.382 INFO:journalctl@ceph.mds.k.smithi082.stdout:May 03 21:17:57 smithi082 bash[62388]: 9acc39735ce2143db050ee2ca62f4458c112d4ac6be4c8133414b02c5e81bd87 2024-05-03T21:17:57.597 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/713bc57a-0991-11ef-bc97-c7b262605968/mon.a/config 2024-05-03T21:17:57.745 INFO:journalctl@ceph.mds.k.smithi082.stdout:May 03 21:17:57 smithi082 systemd[1]: Started Ceph mds.k for 713bc57a-0991-11ef-bc97-c7b262605968. 2024-05-03T21:17:58.706 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:17:58 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:17:58.706 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:17:58 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:17:58.706 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:17:58 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:17:58.706 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:17:58 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-03T21:17:58.706 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:17:58 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-03T21:17:58.706 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:17:58 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T21:17:58.706 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:17:58 smithi082 ceph-mon[32933]: Deploying daemon mds.j on smithi067 2024-05-03T21:17:58.707 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:17:58 smithi082 ceph-mon[32933]: pgmap v257: 1 pgs: 1 active+clean; 577 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-03T21:17:58.707 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:17:58 smithi082 ceph-mon[32933]: mds.? [v2:172.21.15.82:6832/1127287628,v1:172.21.15.82:6833/1127287628] up:boot 2024-05-03T21:17:58.707 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:17:58 smithi082 ceph-mon[32933]: fsmap 2 up:standby 2024-05-03T21:17:58.707 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:17:58 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-05-03T21:17:58.719 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:17:58 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:17:58.719 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:17:58 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:17:58.720 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:17:58 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:17:58.720 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:17:58 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-03T21:17:58.720 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:17:58 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-03T21:17:58.720 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:17:58 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T21:17:58.720 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:17:58 smithi067 ceph-mon[29730]: Deploying daemon mds.j on smithi067 2024-05-03T21:17:58.720 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:17:58 smithi067 ceph-mon[29730]: pgmap v257: 1 pgs: 1 active+clean; 577 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-03T21:17:58.720 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:17:58 smithi067 ceph-mon[29730]: mds.? [v2:172.21.15.82:6832/1127287628,v1:172.21.15.82:6833/1127287628] up:boot 2024-05-03T21:17:58.720 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:17:58 smithi067 ceph-mon[29730]: fsmap 2 up:standby 2024-05-03T21:17:58.720 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:17:58 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-05-03T21:17:58.824 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:17:58 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:17:58.825 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:17:58 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:17:58.825 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:17:58 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:17:58.825 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:17:58 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-03T21:17:58.825 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:17:58 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-03T21:17:58.825 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:17:58 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T21:17:58.825 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:17:58 smithi138 ceph-mon[33110]: Deploying daemon mds.j on smithi067 2024-05-03T21:17:58.825 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:17:58 smithi138 ceph-mon[33110]: pgmap v257: 1 pgs: 1 active+clean; 577 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-03T21:17:58.825 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:17:58 smithi138 ceph-mon[33110]: mds.? [v2:172.21.15.82:6832/1127287628,v1:172.21.15.82:6833/1127287628] up:boot 2024-05-03T21:17:58.825 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:17:58 smithi138 ceph-mon[33110]: fsmap 2 up:standby 2024-05-03T21:17:58.825 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:17:58 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-05-03T21:17:59.487 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:17:59.484+0000 7f5a32983700 1 -- 172.21.15.67:0/2906528114 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5a2c108560 msgr2=0x7f5a2c108940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:17:59.487 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:17:59.484+0000 7f5a32983700 1 --2- 172.21.15.67:0/2906528114 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5a2c108560 0x7f5a2c108940 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f5a1c009990 tx=0x7f5a1c009ca0 comp rx=0 tx=0).stop 2024-05-03T21:17:59.487 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:17:59.484+0000 7f5a32983700 1 -- 172.21.15.67:0/2906528114 shutdown_connections 2024-05-03T21:17:59.487 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:17:59.484+0000 7f5a32983700 1 --2- 172.21.15.67:0/2906528114 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5a2c108560 0x7f5a2c108940 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:17:59.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:17:59.484+0000 7f5a32983700 1 --2- 172.21.15.67:0/2906528114 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f5a2c102800 0x7f5a2c110270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:17:59.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:17:59.484+0000 7f5a32983700 1 --2- 172.21.15.67:0/2906528114 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f5a2c101e40 0x7f5a2c1022c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:17:59.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:17:59.484+0000 7f5a32983700 1 -- 172.21.15.67:0/2906528114 >> 172.21.15.67:0/2906528114 conn(0x7f5a2c0776c0 msgr2=0x7f5a2c077ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T21:17:59.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:17:59.485+0000 7f5a32983700 1 -- 172.21.15.67:0/2906528114 shutdown_connections 2024-05-03T21:17:59.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:17:59.485+0000 7f5a32983700 1 -- 172.21.15.67:0/2906528114 wait complete. 2024-05-03T21:17:59.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:17:59.485+0000 7f5a32983700 1 Processor -- start 2024-05-03T21:17:59.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:17:59.485+0000 7f5a32983700 1 -- start start 2024-05-03T21:17:59.489 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:17:59.486+0000 7f5a32983700 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f5a2c101e40 0x7f5a2c199df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:17:59.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:17:59.486+0000 7f5a32983700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5a2c102800 0x7f5a2c19a330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:17:59.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:17:59.486+0000 7f5a32983700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f5a2c108560 0x7f5a2c19e6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:17:59.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:17:59.486+0000 7f5a32983700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f5a2c113960 con 0x7f5a2c102800 2024-05-03T21:17:59.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:17:59.486+0000 7f5a32983700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f5a2c1137e0 con 0x7f5a2c108560 2024-05-03T21:17:59.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:17:59.486+0000 7f5a32983700 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f5a2c113ae0 con 0x7f5a2c101e40 2024-05-03T21:17:59.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:17:59.486+0000 7f5a2b7fe700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5a2c102800 0x7f5a2c19a330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:17:59.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:17:59.486+0000 7f5a2b7fe700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5a2c102800 0x7f5a2c19a330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:60492/0 (socket says 172.21.15.67:60492) 2024-05-03T21:17:59.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:17:59.486+0000 7f5a2b7fe700 1 -- 172.21.15.67:0/1629199239 learned_addr learned my addr 172.21.15.67:0/1629199239 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-03T21:17:59.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:17:59.486+0000 7f5a30f20700 1 --2- 172.21.15.67:0/1629199239 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f5a2c108560 0x7f5a2c19e6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:17:59.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:17:59.486+0000 7f5a2bfff700 1 --2- 172.21.15.67:0/1629199239 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f5a2c101e40 0x7f5a2c199df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:17:59.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:17:59.486+0000 7f5a2b7fe700 1 -- 172.21.15.67:0/1629199239 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f5a2c101e40 msgr2=0x7f5a2c199df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:17:59.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:17:59.486+0000 7f5a2b7fe700 1 --2- 172.21.15.67:0/1629199239 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f5a2c101e40 0x7f5a2c199df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:17:59.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:17:59.486+0000 7f5a2b7fe700 1 -- 172.21.15.67:0/1629199239 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f5a2c108560 msgr2=0x7f5a2c19e6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:17:59.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:17:59.486+0000 7f5a2b7fe700 1 --2- 172.21.15.67:0/1629199239 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f5a2c108560 0x7f5a2c19e6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:17:59.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:17:59.486+0000 7f5a2b7fe700 1 -- 172.21.15.67:0/1629199239 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5a2c19ee00 con 0x7f5a2c102800 2024-05-03T21:17:59.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:17:59.486+0000 7f5a2b7fe700 1 --2- 172.21.15.67:0/1629199239 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5a2c102800 0x7f5a2c19a330 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f5a2000b530 tx=0x7f5a2000b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T21:17:59.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:17:59.486+0000 7f5a297fa700 1 -- 172.21.15.67:0/1629199239 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a20015020 con 0x7f5a2c102800 2024-05-03T21:17:59.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:17:59.486+0000 7f5a32983700 1 -- 172.21.15.67:0/1629199239 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5a2c19f0f0 con 0x7f5a2c102800 2024-05-03T21:17:59.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:17:59.486+0000 7f5a297fa700 1 -- 172.21.15.67:0/1629199239 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(28 keys) ==== 1170+0+0 (secure 0 0 0) 0x7f5a20011d30 con 0x7f5a2c102800 2024-05-03T21:17:59.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:17:59.486+0000 7f5a297fa700 1 -- 172.21.15.67:0/1629199239 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a2000f2e0 con 0x7f5a2c102800 2024-05-03T21:17:59.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:17:59.486+0000 7f5a32983700 1 -- 172.21.15.67:0/1629199239 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5a2c1a6970 con 0x7f5a2c102800 2024-05-03T21:17:59.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:17:59.488+0000 7f5a32983700 1 -- 172.21.15.67:0/1629199239 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5a2c06a190 con 0x7f5a2c102800 2024-05-03T21:17:59.492 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:17:59.489+0000 7f5a297fa700 1 -- 172.21.15.67:0/1629199239 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 16) ==== 149652+0+0 (secure 0 0 0) 0x7f5a20018020 con 0x7f5a2c102800 2024-05-03T21:17:59.492 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:17:59.489+0000 7f5a297fa700 1 --2- 172.21.15.67:0/1629199239 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f5a140b2880 0x7f5a140b4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:17:59.492 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:17:59.490+0000 7f5a2bfff700 1 --2- 172.21.15.67:0/1629199239 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f5a140b2880 0x7f5a140b4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:17:59.492 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:17:59.490+0000 7f5a297fa700 1 -- 172.21.15.67:0/1629199239 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(75..75 src has 1..75) ==== 7127+0+0 (secure 0 0 0) 0x7f5a200e1b80 con 0x7f5a2c102800 2024-05-03T21:17:59.492 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:17:59.490+0000 7f5a2bfff700 1 --2- 172.21.15.67:0/1629199239 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f5a140b2880 0x7f5a140b4d40 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f5a1c006010 tx=0x7f5a1c01a040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T21:17:59.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:17:59.493+0000 7f5a297fa700 1 -- 172.21.15.67:0/1629199239 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191839 (secure 0 0 0) 0x7f5a2001f020 con 0x7f5a2c102800 2024-05-03T21:17:59.765 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:17:59.762+0000 7f5a32983700 1 -- 172.21.15.67:0/1629199239 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f5a2c02e510 con 0x7f5a2c102800 2024-05-03T21:17:59.765 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:17:59.763+0000 7f5a297fa700 1 -- 172.21.15.67:0/1629199239 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 3 v3) ==== 93+0+2221 (secure 0 0 0) 0x7f5a200a9d40 con 0x7f5a2c102800 2024-05-03T21:17:59.765 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-05-03T21:17:59.765 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":3,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24376,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.138:6833/3028022602","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6832","nonce":3028022602},{"type":"v1","addr":"172.21.15.138:6833","nonce":3028022602}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24407,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.82:6833/1127287628","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6832","nonce":1127287628},{"type":"v1","addr":"172.21.15.82:6833","nonce":1127287628}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[]} 2024-05-03T21:17:59.767 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:17:59.765+0000 7f5a32983700 1 -- 172.21.15.67:0/1629199239 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f5a140b2880 msgr2=0x7f5a140b4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:17:59.767 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:17:59.765+0000 7f5a32983700 1 --2- 172.21.15.67:0/1629199239 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f5a140b2880 0x7f5a140b4d40 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f5a1c006010 tx=0x7f5a1c01a040 comp rx=0 tx=0).stop 2024-05-03T21:17:59.767 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:17:59.765+0000 7f5a32983700 1 -- 172.21.15.67:0/1629199239 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5a2c102800 msgr2=0x7f5a2c19a330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:17:59.767 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:17:59.765+0000 7f5a32983700 1 --2- 172.21.15.67:0/1629199239 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5a2c102800 0x7f5a2c19a330 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f5a2000b530 tx=0x7f5a2000b840 comp rx=0 tx=0).stop 2024-05-03T21:17:59.767 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:17:59.765+0000 7f5a32983700 1 -- 172.21.15.67:0/1629199239 shutdown_connections 2024-05-03T21:17:59.767 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:17:59.765+0000 7f5a32983700 1 --2- 172.21.15.67:0/1629199239 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f5a140b2880 0x7f5a140b4d40 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:17:59.767 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:17:59.765+0000 7f5a32983700 1 --2- 172.21.15.67:0/1629199239 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f5a2c108560 0x7f5a2c19e6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:17:59.768 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:17:59.765+0000 7f5a32983700 1 --2- 172.21.15.67:0/1629199239 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f5a2c101e40 0x7f5a2c199df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:17:59.768 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:17:59.765+0000 7f5a32983700 1 --2- 172.21.15.67:0/1629199239 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5a2c102800 0x7f5a2c19a330 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:17:59.768 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:17:59.765+0000 7f5a32983700 1 -- 172.21.15.67:0/1629199239 >> 172.21.15.67:0/1629199239 conn(0x7f5a2c0776c0 msgr2=0x7f5a2c100860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T21:17:59.768 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:17:59.765+0000 7f5a32983700 1 -- 172.21.15.67:0/1629199239 shutdown_connections 2024-05-03T21:17:59.768 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:17:59.765+0000 7f5a32983700 1 -- 172.21.15.67:0/1629199239 wait complete. 2024-05-03T21:17:59.768 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 3 2024-05-03T21:17:59.907 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:17:59 smithi067 ceph-mon[29730]: from='client.? 172.21.15.67:0/1629199239' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-03T21:17:59.995 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:17:59 smithi082 ceph-mon[32933]: from='client.? 172.21.15.67:0/1629199239' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-03T21:18:00.074 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:17:59 smithi138 ceph-mon[33110]: from='client.? 172.21.15.67:0/1629199239' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-03T21:18:01.109 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:18:00 smithi067 ceph-mon[29730]: pgmap v258: 1 pgs: 1 active+clean; 577 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-03T21:18:01.245 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:18:00 smithi082 ceph-mon[32933]: pgmap v258: 1 pgs: 1 active+clean; 577 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-03T21:18:01.324 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:18:00 smithi138 ceph-mon[33110]: pgmap v258: 1 pgs: 1 active+clean; 577 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-03T21:18:02.471 INFO:journalctl@ceph.mds.j.smithi067.stdout:May 03 21:18:02 smithi067 systemd[1]: Started Ceph mds.j for 713bc57a-0991-11ef-bc97-c7b262605968. 2024-05-03T21:18:02.614 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9a047fe7599a549042d06a3fb5b6813a203eabea shell --fsid 713bc57a-0991-11ef-bc97-c7b262605968 -- ceph fs dump --format=json 2024-05-03T21:18:03.221 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:18:03 smithi067 ceph-mon[29730]: pgmap v259: 1 pgs: 1 active+clean; 577 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-03T21:18:03.221 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:18:03 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:18:03.222 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:18:03 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:18:03.222 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:18:03 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:18:03.222 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:18:03 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-03T21:18:03.222 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:18:03 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-03T21:18:03.222 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:18:03 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T21:18:03.222 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:18:03 smithi067 ceph-mon[29730]: Deploying daemon mds.d on smithi067 2024-05-03T21:18:03.222 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:18:03 smithi067 ceph-mon[29730]: mds.? [v2:172.21.15.67:6834/4224826546,v1:172.21.15.67:6835/4224826546] up:boot 2024-05-03T21:18:03.222 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:18:03 smithi067 ceph-mon[29730]: fsmap 3 up:standby 2024-05-03T21:18:03.222 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:18:03 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-05-03T21:18:03.495 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:18:03 smithi082 ceph-mon[32933]: pgmap v259: 1 pgs: 1 active+clean; 577 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-03T21:18:03.495 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:18:03 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:18:03.495 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:18:03 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:18:03.496 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:18:03 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:18:03.496 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:18:03 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-03T21:18:03.496 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:18:03 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-03T21:18:03.496 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:18:03 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T21:18:03.496 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:18:03 smithi082 ceph-mon[32933]: Deploying daemon mds.d on smithi067 2024-05-03T21:18:03.496 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:18:03 smithi082 ceph-mon[32933]: mds.? [v2:172.21.15.67:6834/4224826546,v1:172.21.15.67:6835/4224826546] up:boot 2024-05-03T21:18:03.496 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:18:03 smithi082 ceph-mon[32933]: fsmap 3 up:standby 2024-05-03T21:18:03.496 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:18:03 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-05-03T21:18:03.575 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:18:03 smithi138 ceph-mon[33110]: pgmap v259: 1 pgs: 1 active+clean; 577 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-03T21:18:03.575 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:18:03 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:18:03.575 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:18:03 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:18:03.575 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:18:03 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:18:03.575 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:18:03 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-03T21:18:03.575 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:18:03 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-03T21:18:03.575 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:18:03 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T21:18:03.575 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:18:03 smithi138 ceph-mon[33110]: Deploying daemon mds.d on smithi067 2024-05-03T21:18:03.575 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:18:03 smithi138 ceph-mon[33110]: mds.? [v2:172.21.15.67:6834/4224826546,v1:172.21.15.67:6835/4224826546] up:boot 2024-05-03T21:18:03.575 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:18:03 smithi138 ceph-mon[33110]: fsmap 3 up:standby 2024-05-03T21:18:03.576 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:18:03 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-05-03T21:18:03.979 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/713bc57a-0991-11ef-bc97-c7b262605968/mon.a/config 2024-05-03T21:18:05.221 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:18:05 smithi067 ceph-mon[29730]: pgmap v260: 1 pgs: 1 active+clean; 577 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-03T21:18:05.495 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:18:05 smithi082 ceph-mon[32933]: pgmap v260: 1 pgs: 1 active+clean; 577 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-03T21:18:05.574 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:18:05 smithi138 ceph-mon[33110]: pgmap v260: 1 pgs: 1 active+clean; 577 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-03T21:18:05.832 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:05.828+0000 7fdcc29a7700 1 -- 172.21.15.67:0/901824996 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdcbc102d60 msgr2=0x7fdcbc10f250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:18:05.832 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:05.828+0000 7fdcc29a7700 1 --2- 172.21.15.67:0/901824996 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdcbc102d60 0x7fdcbc10f250 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fdcb400b0a0 tx=0x7fdcb400b3b0 comp rx=0 tx=0).stop 2024-05-03T21:18:05.833 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:05.828+0000 7fdcc29a7700 1 -- 172.21.15.67:0/901824996 shutdown_connections 2024-05-03T21:18:05.833 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:05.828+0000 7fdcc29a7700 1 --2- 172.21.15.67:0/901824996 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdcbc102d60 0x7fdcbc10f250 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:18:05.833 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:05.828+0000 7fdcc29a7700 1 --2- 172.21.15.67:0/901824996 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fdcbc108410 0x7fdcbc1087f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:18:05.833 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:05.828+0000 7fdcc29a7700 1 --2- 172.21.15.67:0/901824996 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fdcbc1023a0 0x7fdcbc102820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:18:05.833 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:05.828+0000 7fdcc29a7700 1 -- 172.21.15.67:0/901824996 >> 172.21.15.67:0/901824996 conn(0x7fdcbc079350 msgr2=0x7fdcbc079760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T21:18:05.833 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:05.828+0000 7fdcc29a7700 1 -- 172.21.15.67:0/901824996 shutdown_connections 2024-05-03T21:18:05.833 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:05.828+0000 7fdcc29a7700 1 -- 172.21.15.67:0/901824996 wait complete. 2024-05-03T21:18:05.833 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:05.829+0000 7fdcc29a7700 1 Processor -- start 2024-05-03T21:18:05.833 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:05.829+0000 7fdcc29a7700 1 -- start start 2024-05-03T21:18:05.833 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:05.829+0000 7fdcc29a7700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdcbc1023a0 0x7fdcbc19f7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:18:05.833 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:05.829+0000 7fdcc29a7700 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fdcbc102d60 0x7fdcbc19fce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:18:05.833 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:05.829+0000 7fdcc29a7700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fdcbc108410 0x7fdcbc1999b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:18:05.833 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:05.829+0000 7fdcc29a7700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fdcbc1127c0 con 0x7fdcbc1023a0 2024-05-03T21:18:05.834 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:05.829+0000 7fdcc29a7700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7fdcbc112640 con 0x7fdcbc108410 2024-05-03T21:18:05.834 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:05.829+0000 7fdcc29a7700 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7fdcbc112940 con 0x7fdcbc102d60 2024-05-03T21:18:05.834 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:05.829+0000 7fdcbbfff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdcbc1023a0 0x7fdcbc19f7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:18:05.834 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:05.829+0000 7fdcbbfff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdcbc1023a0 0x7fdcbc19f7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:59410/0 (socket says 172.21.15.67:59410) 2024-05-03T21:18:05.834 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:05.829+0000 7fdcbbfff700 1 -- 172.21.15.67:0/4054637988 learned_addr learned my addr 172.21.15.67:0/4054637988 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-03T21:18:05.834 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:05.830+0000 7fdcc0f44700 1 --2- 172.21.15.67:0/4054637988 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fdcbc108410 0x7fdcbc1999b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:18:05.834 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:05.830+0000 7fdcbb7fe700 1 --2- 172.21.15.67:0/4054637988 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fdcbc102d60 0x7fdcbc19fce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:18:05.834 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:05.830+0000 7fdcbbfff700 1 -- 172.21.15.67:0/4054637988 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fdcbc102d60 msgr2=0x7fdcbc19fce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:18:05.834 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:05.830+0000 7fdcbbfff700 1 --2- 172.21.15.67:0/4054637988 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fdcbc102d60 0x7fdcbc19fce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:18:05.834 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:05.830+0000 7fdcbbfff700 1 -- 172.21.15.67:0/4054637988 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fdcbc108410 msgr2=0x7fdcbc1999b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:18:05.834 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:05.830+0000 7fdcbbfff700 1 --2- 172.21.15.67:0/4054637988 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fdcbc108410 0x7fdcbc1999b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:18:05.834 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:05.830+0000 7fdcbbfff700 1 -- 172.21.15.67:0/4054637988 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdcbc19a110 con 0x7fdcbc1023a0 2024-05-03T21:18:05.834 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:05.830+0000 7fdcbbfff700 1 --2- 172.21.15.67:0/4054637988 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdcbc1023a0 0x7fdcbc19f7a0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fdcac00c9b0 tx=0x7fdcac00ccc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T21:18:05.834 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:05.830+0000 7fdcb97fa700 1 -- 172.21.15.67:0/4054637988 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdcac007b00 con 0x7fdcbc1023a0 2024-05-03T21:18:05.835 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:05.830+0000 7fdcc29a7700 1 -- 172.21.15.67:0/4054637988 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdcbc19a400 con 0x7fdcbc1023a0 2024-05-03T21:18:05.835 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:05.830+0000 7fdcc29a7700 1 -- 172.21.15.67:0/4054637988 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdcbc1a6670 con 0x7fdcbc1023a0 2024-05-03T21:18:05.835 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:05.830+0000 7fdcb97fa700 1 -- 172.21.15.67:0/4054637988 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(28 keys) ==== 1170+0+0 (secure 0 0 0) 0x7fdcac007ca0 con 0x7fdcbc1023a0 2024-05-03T21:18:05.835 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:05.830+0000 7fdcb97fa700 1 -- 172.21.15.67:0/4054637988 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdcac018780 con 0x7fdcbc1023a0 2024-05-03T21:18:05.835 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:05.831+0000 7fdcc29a7700 1 -- 172.21.15.67:0/4054637988 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdcbc06a190 con 0x7fdcbc1023a0 2024-05-03T21:18:05.835 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:05.833+0000 7fdcb97fa700 1 -- 172.21.15.67:0/4054637988 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 16) ==== 149652+0+0 (secure 0 0 0) 0x7fdcac00f450 con 0x7fdcbc1023a0 2024-05-03T21:18:05.836 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:05.834+0000 7fdcb97fa700 1 --2- 172.21.15.67:0/4054637988 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7fdca40b2880 0x7fdca40b4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:18:05.836 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:05.834+0000 7fdcbb7fe700 1 --2- 172.21.15.67:0/4054637988 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7fdca40b2880 0x7fdca40b4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:18:05.836 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:05.834+0000 7fdcb97fa700 1 -- 172.21.15.67:0/4054637988 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(75..75 src has 1..75) ==== 7127+0+0 (secure 0 0 0) 0x7fdcac0e1fa0 con 0x7fdcbc1023a0 2024-05-03T21:18:05.836 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:05.834+0000 7fdcbb7fe700 1 --2- 172.21.15.67:0/4054637988 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7fdca40b2880 0x7fdca40b4d40 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fdcb0005ea0 tx=0x7fdcb0005e30 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T21:18:05.840 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:05.838+0000 7fdcb97fa700 1 -- 172.21.15.67:0/4054637988 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191839 (secure 0 0 0) 0x7fdcac014030 con 0x7fdcbc1023a0 2024-05-03T21:18:06.115 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:06.113+0000 7fdcc29a7700 1 -- 172.21.15.67:0/4054637988 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fdcbc19ae90 con 0x7fdcbc1023a0 2024-05-03T21:18:06.115 INFO:journalctl@ceph.mds.d.smithi067.stdout:May 03 21:18:06 smithi067 podman[67971]: 2024-05-03 21:18:06.006702633 +0000 UTC m=+0.150897050 container create 62837578d30949856a931a27ca9a9e8b4d7ff72bc95197aa16ad1b09ca97e3b7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:68e15ae152ef5c805c733e76da8fc1d3dd4ee669d5ba025f457e05725baa24dd, name=ceph-713bc57a-0991-11ef-bc97-c7b262605968-mds-d, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240430, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, ceph=True, GIT_CLEAN=True, RELEASE=wip-rishabh-testing-20240501.193033-9a047fe, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image) 2024-05-03T21:18:06.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:06.113+0000 7fdcb97fa700 1 -- 172.21.15.67:0/4054637988 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 4 v4) ==== 93+0+3046 (secure 0 0 0) 0x7fdcbc19ae90 con 0x7fdcbc1023a0 2024-05-03T21:18:06.116 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-05-03T21:18:06.116 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":4,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14541,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6835/4224826546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":4224826546},{"type":"v1","addr":"172.21.15.67:6835","nonce":4224826546}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24376,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.138:6833/3028022602","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6832","nonce":3028022602},{"type":"v1","addr":"172.21.15.138:6833","nonce":3028022602}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24407,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.82:6833/1127287628","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6832","nonce":1127287628},{"type":"v1","addr":"172.21.15.82:6833","nonce":1127287628}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[]} 2024-05-03T21:18:06.118 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:06.115+0000 7fdcc29a7700 1 -- 172.21.15.67:0/4054637988 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7fdca40b2880 msgr2=0x7fdca40b4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:18:06.118 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:06.116+0000 7fdcc29a7700 1 --2- 172.21.15.67:0/4054637988 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7fdca40b2880 0x7fdca40b4d40 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fdcb0005ea0 tx=0x7fdcb0005e30 comp rx=0 tx=0).stop 2024-05-03T21:18:06.118 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:06.116+0000 7fdcc29a7700 1 -- 172.21.15.67:0/4054637988 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdcbc1023a0 msgr2=0x7fdcbc19f7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:18:06.118 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:06.116+0000 7fdcc29a7700 1 --2- 172.21.15.67:0/4054637988 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdcbc1023a0 0x7fdcbc19f7a0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fdcac00c9b0 tx=0x7fdcac00ccc0 comp rx=0 tx=0).stop 2024-05-03T21:18:06.118 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:06.116+0000 7fdcc29a7700 1 -- 172.21.15.67:0/4054637988 shutdown_connections 2024-05-03T21:18:06.118 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:06.116+0000 7fdcc29a7700 1 --2- 172.21.15.67:0/4054637988 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7fdca40b2880 0x7fdca40b4d40 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:18:06.118 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:06.116+0000 7fdcc29a7700 1 --2- 172.21.15.67:0/4054637988 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdcbc1023a0 0x7fdcbc19f7a0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:18:06.118 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:06.116+0000 7fdcc29a7700 1 --2- 172.21.15.67:0/4054637988 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fdcbc108410 0x7fdcbc1999b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:18:06.118 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:06.116+0000 7fdcc29a7700 1 --2- 172.21.15.67:0/4054637988 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fdcbc102d60 0x7fdcbc19fce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:18:06.119 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:06.116+0000 7fdcc29a7700 1 -- 172.21.15.67:0/4054637988 >> 172.21.15.67:0/4054637988 conn(0x7fdcbc079350 msgr2=0x7fdcbc1013f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T21:18:06.119 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:06.116+0000 7fdcc29a7700 1 -- 172.21.15.67:0/4054637988 shutdown_connections 2024-05-03T21:18:06.119 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:06.116+0000 7fdcc29a7700 1 -- 172.21.15.67:0/4054637988 wait complete. 2024-05-03T21:18:06.119 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 4 2024-05-03T21:18:06.445 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:18:06 smithi067 ceph-mon[29730]: from='client.? 172.21.15.67:0/4054637988' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-03T21:18:06.495 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:18:06 smithi082 ceph-mon[32933]: from='client.? 172.21.15.67:0/4054637988' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-03T21:18:06.574 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:18:06 smithi138 ceph-mon[33110]: from='client.? 172.21.15.67:0/4054637988' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-03T21:18:06.721 INFO:journalctl@ceph.mds.d.smithi067.stdout:May 03 21:18:06 smithi067 podman[67971]: 2024-05-03 21:18:06.444113394 +0000 UTC m=+0.588307812 container init 62837578d30949856a931a27ca9a9e8b4d7ff72bc95197aa16ad1b09ca97e3b7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:68e15ae152ef5c805c733e76da8fc1d3dd4ee669d5ba025f457e05725baa24dd, name=ceph-713bc57a-0991-11ef-bc97-c7b262605968-mds-d, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, RELEASE=wip-rishabh-testing-20240501.193033-9a047fe, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240430, io.buildah.version=1.33.5, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS) 2024-05-03T21:18:06.721 INFO:journalctl@ceph.mds.d.smithi067.stdout:May 03 21:18:06 smithi067 podman[67971]: 2024-05-03 21:18:06.446988507 +0000 UTC m=+0.591182927 container start 62837578d30949856a931a27ca9a9e8b4d7ff72bc95197aa16ad1b09ca97e3b7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:68e15ae152ef5c805c733e76da8fc1d3dd4ee669d5ba025f457e05725baa24dd, name=ceph-713bc57a-0991-11ef-bc97-c7b262605968-mds-d, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, ceph=True, org.label-schema.vendor=CentOS, RELEASE=wip-rishabh-testing-20240501.193033-9a047fe, GIT_CLEAN=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.build-date=20240430) 2024-05-03T21:18:06.721 INFO:journalctl@ceph.mds.d.smithi067.stdout:May 03 21:18:06 smithi067 ceph-mds[68003]: -- 172.21.15.67:0/2821330459 <== mon.0 v2:172.21.15.67:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55678fa41a40 con 0x55678fa14c00 2024-05-03T21:18:06.721 INFO:journalctl@ceph.mds.d.smithi067.stdout:May 03 21:18:06 smithi067 ceph-713bc57a-0991-11ef-bc97-c7b262605968-mds-d[67994]: starting mds.d at 2024-05-03T21:18:06.722 INFO:journalctl@ceph.mds.d.smithi067.stdout:May 03 21:18:06 smithi067 bash[67971]: 62837578d30949856a931a27ca9a9e8b4d7ff72bc95197aa16ad1b09ca97e3b7 2024-05-03T21:18:06.722 INFO:journalctl@ceph.mds.d.smithi067.stdout:May 03 21:18:06 smithi067 systemd[1]: Started Ceph mds.d for 713bc57a-0991-11ef-bc97-c7b262605968. 2024-05-03T21:18:07.405 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:18:07 smithi082 ceph-mon[32933]: pgmap v261: 1 pgs: 1 active+clean; 577 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-03T21:18:07.405 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:18:07 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:18:07.405 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:18:07 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:18:07.405 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:18:07 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:18:07.405 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:18:07 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-03T21:18:07.405 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:18:07 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-03T21:18:07.405 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:18:07 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T21:18:07.464 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:18:07 smithi067 ceph-mon[29730]: pgmap v261: 1 pgs: 1 active+clean; 577 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-03T21:18:07.464 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:18:07 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:18:07.464 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:18:07 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:18:07.465 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:18:07 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:18:07.465 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:18:07 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-03T21:18:07.465 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:18:07 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-03T21:18:07.465 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:18:07 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T21:18:07.574 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:18:07 smithi138 ceph-mon[33110]: pgmap v261: 1 pgs: 1 active+clean; 577 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-03T21:18:07.575 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:18:07 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:18:07.575 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:18:07 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:18:07.575 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:18:07 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:18:07.575 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:18:07 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-03T21:18:07.575 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:18:07 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-03T21:18:07.575 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:18:07 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T21:18:08.420 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:18:08 smithi082 ceph-mon[32933]: Deploying daemon mds.e on smithi082 2024-05-03T21:18:08.420 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:18:08 smithi082 ceph-mon[32933]: mds.? [v2:172.21.15.67:6836/2485231621,v1:172.21.15.67:6837/2485231621] up:boot 2024-05-03T21:18:08.421 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:18:08 smithi082 ceph-mon[32933]: fsmap 4 up:standby 2024-05-03T21:18:08.421 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:18:08 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-05-03T21:18:08.471 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:18:08 smithi067 ceph-mon[29730]: Deploying daemon mds.e on smithi082 2024-05-03T21:18:08.471 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:18:08 smithi067 ceph-mon[29730]: mds.? [v2:172.21.15.67:6836/2485231621,v1:172.21.15.67:6837/2485231621] up:boot 2024-05-03T21:18:08.471 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:18:08 smithi067 ceph-mon[29730]: fsmap 4 up:standby 2024-05-03T21:18:08.471 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:18:08 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-05-03T21:18:08.574 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:18:08 smithi138 ceph-mon[33110]: Deploying daemon mds.e on smithi082 2024-05-03T21:18:08.574 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:18:08 smithi138 ceph-mon[33110]: mds.? [v2:172.21.15.67:6836/2485231621,v1:172.21.15.67:6837/2485231621] up:boot 2024-05-03T21:18:08.574 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:18:08 smithi138 ceph-mon[33110]: fsmap 4 up:standby 2024-05-03T21:18:08.575 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:18:08 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-05-03T21:18:08.995 INFO:journalctl@ceph.mds.e.smithi082.stdout:May 03 21:18:08 smithi082 systemd[1]: Starting Ceph mds.e for 713bc57a-0991-11ef-bc97-c7b262605968... 2024-05-03T21:18:09.104 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9a047fe7599a549042d06a3fb5b6813a203eabea shell --fsid 713bc57a-0991-11ef-bc97-c7b262605968 -- ceph fs dump --format=json 2024-05-03T21:18:09.367 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:18:09 smithi082 ceph-mon[32933]: pgmap v262: 1 pgs: 1 active+clean; 577 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-03T21:18:09.367 INFO:journalctl@ceph.mds.e.smithi082.stdout:May 03 21:18:09 smithi082 podman[62711]: 2024-05-03 21:18:09.113191353 +0000 UTC m=+0.106715787 container create 0e80ad566fa7b1e9425fb74127087884563fb3251d408de7b32de6bf4d9e56dc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:68e15ae152ef5c805c733e76da8fc1d3dd4ee669d5ba025f457e05725baa24dd, name=ceph-713bc57a-0991-11ef-bc97-c7b262605968-mds-e, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, ceph=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.build-date=20240430, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-rishabh-testing-20240501.193033-9a047fe) 2024-05-03T21:18:09.367 INFO:journalctl@ceph.mds.e.smithi082.stdout:May 03 21:18:09 smithi082 podman[62711]: 2024-05-03 21:18:09.024329641 +0000 UTC m=+0.017854098 image pull ee4d846202496ba6096593e8e8becc0700a5eb7ebd95536031187f5f337d4012 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:68e15ae152ef5c805c733e76da8fc1d3dd4ee669d5ba025f457e05725baa24dd 2024-05-03T21:18:09.367 INFO:journalctl@ceph.mds.e.smithi082.stdout:May 03 21:18:09 smithi082 podman[62711]: 2024-05-03 21:18:09.340680808 +0000 UTC m=+0.334205256 container init 0e80ad566fa7b1e9425fb74127087884563fb3251d408de7b32de6bf4d9e56dc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:68e15ae152ef5c805c733e76da8fc1d3dd4ee669d5ba025f457e05725baa24dd, name=ceph-713bc57a-0991-11ef-bc97-c7b262605968-mds-e, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, RELEASE=wip-rishabh-testing-20240501.193033-9a047fe, maintainer=Guillaume Abrioux , ceph=True, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.build-date=20240430, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True) 2024-05-03T21:18:09.367 INFO:journalctl@ceph.mds.e.smithi082.stdout:May 03 21:18:09 smithi082 podman[62711]: 2024-05-03 21:18:09.343144916 +0000 UTC m=+0.336669356 container start 0e80ad566fa7b1e9425fb74127087884563fb3251d408de7b32de6bf4d9e56dc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:68e15ae152ef5c805c733e76da8fc1d3dd4ee669d5ba025f457e05725baa24dd, name=ceph-713bc57a-0991-11ef-bc97-c7b262605968-mds-e, CEPH_POINT_RELEASE=, RELEASE=wip-rishabh-testing-20240501.193033-9a047fe, GIT_CLEAN=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, io.buildah.version=1.33.5, org.label-schema.build-date=20240430, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, ceph=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2) 2024-05-03T21:18:09.368 INFO:journalctl@ceph.mds.e.smithi082.stdout:May 03 21:18:09 smithi082 ceph-mds[62730]: -- 172.21.15.82:0/658900062 <== mon.2 v2:172.21.15.82:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5574240f1a40 con 0x5574240c5000 2024-05-03T21:18:09.369 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/713bc57a-0991-11ef-bc97-c7b262605968/mon.a/config 2024-05-03T21:18:09.471 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:18:09 smithi067 ceph-mon[29730]: pgmap v262: 1 pgs: 1 active+clean; 577 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-03T21:18:09.574 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:18:09 smithi138 ceph-mon[33110]: pgmap v262: 1 pgs: 1 active+clean; 577 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-03T21:18:09.692 INFO:journalctl@ceph.mds.e.smithi082.stdout:May 03 21:18:09 smithi082 ceph-713bc57a-0991-11ef-bc97-c7b262605968-mds-e[62726]: starting mds.e at 2024-05-03T21:18:09.693 INFO:journalctl@ceph.mds.e.smithi082.stdout:May 03 21:18:09 smithi082 bash[62711]: 0e80ad566fa7b1e9425fb74127087884563fb3251d408de7b32de6bf4d9e56dc 2024-05-03T21:18:09.693 INFO:journalctl@ceph.mds.e.smithi082.stdout:May 03 21:18:09 smithi082 systemd[1]: Started Ceph mds.e for 713bc57a-0991-11ef-bc97-c7b262605968. 2024-05-03T21:18:10.808 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:18:10 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:18:10.808 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:18:10 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:18:10.809 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:18:10 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:18:10.809 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:18:10 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-03T21:18:10.809 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:18:10 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-03T21:18:10.809 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:18:10 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T21:18:10.809 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:18:10 smithi067 ceph-mon[29730]: Deploying daemon mds.f on smithi138 2024-05-03T21:18:10.809 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:18:10 smithi067 ceph-mon[29730]: pgmap v263: 1 pgs: 1 active+clean; 577 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-03T21:18:10.809 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:18:10 smithi067 ceph-mon[29730]: mds.? [v2:172.21.15.82:6834/280518761,v1:172.21.15.82:6835/280518761] up:boot 2024-05-03T21:18:10.809 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:18:10 smithi067 ceph-mon[29730]: fsmap 5 up:standby 2024-05-03T21:18:10.809 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:18:10 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-05-03T21:18:10.824 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:18:10 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:18:10.824 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:18:10 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:18:10.825 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:18:10 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:18:10.825 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:18:10 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-03T21:18:10.825 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:18:10 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-03T21:18:10.825 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:18:10 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T21:18:10.825 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:18:10 smithi138 ceph-mon[33110]: Deploying daemon mds.f on smithi138 2024-05-03T21:18:10.825 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:18:10 smithi138 ceph-mon[33110]: pgmap v263: 1 pgs: 1 active+clean; 577 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-03T21:18:10.825 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:18:10 smithi138 ceph-mon[33110]: mds.? [v2:172.21.15.82:6834/280518761,v1:172.21.15.82:6835/280518761] up:boot 2024-05-03T21:18:10.825 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:18:10 smithi138 ceph-mon[33110]: fsmap 5 up:standby 2024-05-03T21:18:10.825 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:18:10 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-05-03T21:18:10.996 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:18:10 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:18:10.996 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:18:10 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:18:10.996 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:18:10 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:18:10.996 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:18:10 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-03T21:18:10.996 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:18:10 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-03T21:18:10.996 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:18:10 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T21:18:10.996 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:18:10 smithi082 ceph-mon[32933]: Deploying daemon mds.f on smithi138 2024-05-03T21:18:10.996 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:18:10 smithi082 ceph-mon[32933]: pgmap v263: 1 pgs: 1 active+clean; 577 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-03T21:18:10.996 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:18:10 smithi082 ceph-mon[32933]: mds.? [v2:172.21.15.82:6834/280518761,v1:172.21.15.82:6835/280518761] up:boot 2024-05-03T21:18:10.996 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:18:10 smithi082 ceph-mon[32933]: fsmap 5 up:standby 2024-05-03T21:18:10.997 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:18:10 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-05-03T21:18:11.074 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:11.071+0000 7efc4bc1f700 1 -- 172.21.15.67:0/2365019115 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efc44107700 msgr2=0x7efc4410f2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:18:11.074 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:11.071+0000 7efc4bc1f700 1 --2- 172.21.15.67:0/2365019115 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efc44107700 0x7efc4410f2d0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7efc3c00b0d0 tx=0x7efc3c00b3e0 comp rx=0 tx=0).stop 2024-05-03T21:18:11.074 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:11.071+0000 7efc4bc1f700 1 -- 172.21.15.67:0/2365019115 shutdown_connections 2024-05-03T21:18:11.074 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:11.071+0000 7efc4bc1f700 1 --2- 172.21.15.67:0/2365019115 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efc44107700 0x7efc4410f2d0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:18:11.074 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:11.071+0000 7efc4bc1f700 1 --2- 172.21.15.67:0/2365019115 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7efc44102620 0x7efc44102a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:18:11.074 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:11.071+0000 7efc4bc1f700 1 --2- 172.21.15.67:0/2365019115 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7efc44102fd0 0x7efc44107020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:18:11.074 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:11.071+0000 7efc4bc1f700 1 -- 172.21.15.67:0/2365019115 >> 172.21.15.67:0/2365019115 conn(0x7efc44079440 msgr2=0x7efc44079850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T21:18:11.074 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:11.071+0000 7efc4bc1f700 1 -- 172.21.15.67:0/2365019115 shutdown_connections 2024-05-03T21:18:11.074 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:11.071+0000 7efc4bc1f700 1 -- 172.21.15.67:0/2365019115 wait complete. 2024-05-03T21:18:11.075 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:11.072+0000 7efc4bc1f700 1 Processor -- start 2024-05-03T21:18:11.075 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:11.072+0000 7efc4bc1f700 1 -- start start 2024-05-03T21:18:11.076 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:11.072+0000 7efc4bc1f700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efc44102620 0x7efc441a0700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:18:11.076 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:11.072+0000 7efc4bc1f700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7efc44102fd0 0x7efc441a0c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:18:11.076 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:11.072+0000 7efc4bc1f700 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7efc44107700 0x7efc441a4bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:18:11.077 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:11.072+0000 7efc4bc1f700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7efc44112850 con 0x7efc44102620 2024-05-03T21:18:11.077 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:11.072+0000 7efc4bc1f700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7efc441126d0 con 0x7efc44102fd0 2024-05-03T21:18:11.077 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:11.072+0000 7efc4bc1f700 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7efc441129d0 con 0x7efc44107700 2024-05-03T21:18:11.077 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:11.072+0000 7efc499bb700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efc44102620 0x7efc441a0700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:18:11.077 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:11.072+0000 7efc499bb700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efc44102620 0x7efc441a0700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:59444/0 (socket says 172.21.15.67:59444) 2024-05-03T21:18:11.077 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:11.072+0000 7efc491ba700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7efc44102fd0 0x7efc441a0c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:18:11.077 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:11.072+0000 7efc499bb700 1 -- 172.21.15.67:0/3883751734 learned_addr learned my addr 172.21.15.67:0/3883751734 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-03T21:18:11.077 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:11.072+0000 7efc4a1bc700 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7efc44107700 0x7efc441a4bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:18:11.077 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:11.072+0000 7efc499bb700 1 -- 172.21.15.67:0/3883751734 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7efc44107700 msgr2=0x7efc441a4bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:18:11.077 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:11.072+0000 7efc499bb700 1 --2- 172.21.15.67:0/3883751734 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7efc44107700 0x7efc441a4bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:18:11.077 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:11.072+0000 7efc499bb700 1 -- 172.21.15.67:0/3883751734 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7efc44102fd0 msgr2=0x7efc441a0c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:18:11.078 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:11.072+0000 7efc499bb700 1 --2- 172.21.15.67:0/3883751734 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7efc44102fd0 0x7efc441a0c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:18:11.078 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:11.072+0000 7efc499bb700 1 -- 172.21.15.67:0/3883751734 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efc441a5300 con 0x7efc44102620 2024-05-03T21:18:11.078 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:11.073+0000 7efc4a1bc700 1 --2- 172.21.15.67:0/3883751734 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7efc44107700 0x7efc441a4bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-03T21:18:11.078 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:11.073+0000 7efc499bb700 1 --2- 172.21.15.67:0/3883751734 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efc44102620 0x7efc441a0700 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7efc3400d980 tx=0x7efc3400dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T21:18:11.078 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:11.073+0000 7efc3affd700 1 -- 172.21.15.67:0/3883751734 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc34017070 con 0x7efc44102620 2024-05-03T21:18:11.078 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:11.073+0000 7efc4bc1f700 1 -- 172.21.15.67:0/3883751734 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efc441a55f0 con 0x7efc44102620 2024-05-03T21:18:11.078 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:11.073+0000 7efc3affd700 1 -- 172.21.15.67:0/3883751734 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(28 keys) ==== 1170+0+0 (secure 0 0 0) 0x7efc34005020 con 0x7efc44102620 2024-05-03T21:18:11.078 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:11.073+0000 7efc3affd700 1 -- 172.21.15.67:0/3883751734 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc3400fc60 con 0x7efc44102620 2024-05-03T21:18:11.078 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:11.073+0000 7efc4bc1f700 1 -- 172.21.15.67:0/3883751734 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efc441ace70 con 0x7efc44102620 2024-05-03T21:18:11.079 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:11.074+0000 7efc4bc1f700 1 -- 172.21.15.67:0/3883751734 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efc4406a190 con 0x7efc44102620 2024-05-03T21:18:11.079 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:11.076+0000 7efc3affd700 1 -- 172.21.15.67:0/3883751734 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 16) ==== 149652+0+0 (secure 0 0 0) 0x7efc340051d0 con 0x7efc44102620 2024-05-03T21:18:11.080 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:11.076+0000 7efc3affd700 1 --2- 172.21.15.67:0/3883751734 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7efc300b2880 0x7efc300b4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:18:11.080 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:11.076+0000 7efc491ba700 1 --2- 172.21.15.67:0/3883751734 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7efc300b2880 0x7efc300b4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:18:11.080 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:11.076+0000 7efc3affd700 1 -- 172.21.15.67:0/3883751734 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(75..75 src has 1..75) ==== 7127+0+0 (secure 0 0 0) 0x7efc340e1bc0 con 0x7efc44102620 2024-05-03T21:18:11.080 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:11.077+0000 7efc491ba700 1 --2- 172.21.15.67:0/3883751734 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7efc300b2880 0x7efc300b4d40 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7efc44106050 tx=0x7efc4000b410 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T21:18:11.083 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:11.081+0000 7efc3affd700 1 -- 172.21.15.67:0/3883751734 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191839 (secure 0 0 0) 0x7efc34020020 con 0x7efc44102620 2024-05-03T21:18:11.343 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:11.340+0000 7efc4bc1f700 1 -- 172.21.15.67:0/3883751734 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7efc441ad040 con 0x7efc44102620 2024-05-03T21:18:11.343 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:11.341+0000 7efc3affd700 1 -- 172.21.15.67:0/3883751734 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 6 v6) ==== 93+0+4693 (secure 0 0 0) 0x7efc340a9d80 con 0x7efc44102620 2024-05-03T21:18:11.344 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-05-03T21:18:11.344 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":6,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14541,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6835/4224826546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":4224826546},{"type":"v1","addr":"172.21.15.67:6835","nonce":4224826546}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14553,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6837/2485231621","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6836","nonce":2485231621},{"type":"v1","addr":"172.21.15.67:6837","nonce":2485231621}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24376,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.138:6833/3028022602","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6832","nonce":3028022602},{"type":"v1","addr":"172.21.15.138:6833","nonce":3028022602}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24407,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.82:6833/1127287628","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6832","nonce":1127287628},{"type":"v1","addr":"172.21.15.82:6833","nonce":1127287628}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24425,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.82:6835/280518761","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6834","nonce":280518761},{"type":"v1","addr":"172.21.15.82:6835","nonce":280518761}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6}],"filesystems":[]} 2024-05-03T21:18:11.346 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:11.343+0000 7efc4bc1f700 1 -- 172.21.15.67:0/3883751734 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7efc300b2880 msgr2=0x7efc300b4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:18:11.346 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:11.343+0000 7efc4bc1f700 1 --2- 172.21.15.67:0/3883751734 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7efc300b2880 0x7efc300b4d40 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7efc44106050 tx=0x7efc4000b410 comp rx=0 tx=0).stop 2024-05-03T21:18:11.346 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:11.343+0000 7efc4bc1f700 1 -- 172.21.15.67:0/3883751734 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efc44102620 msgr2=0x7efc441a0700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:18:11.346 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:11.343+0000 7efc4bc1f700 1 --2- 172.21.15.67:0/3883751734 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efc44102620 0x7efc441a0700 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7efc3400d980 tx=0x7efc3400dc90 comp rx=0 tx=0).stop 2024-05-03T21:18:11.346 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:11.343+0000 7efc4bc1f700 1 -- 172.21.15.67:0/3883751734 shutdown_connections 2024-05-03T21:18:11.346 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:11.343+0000 7efc4bc1f700 1 --2- 172.21.15.67:0/3883751734 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7efc300b2880 0x7efc300b4d40 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:18:11.346 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:11.343+0000 7efc4bc1f700 1 --2- 172.21.15.67:0/3883751734 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efc44102620 0x7efc441a0700 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:18:11.346 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:11.343+0000 7efc4bc1f700 1 --2- 172.21.15.67:0/3883751734 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7efc44107700 0x7efc441a4bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:18:11.346 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:11.343+0000 7efc4bc1f700 1 --2- 172.21.15.67:0/3883751734 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7efc44102fd0 0x7efc441a0c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:18:11.347 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:11.343+0000 7efc4bc1f700 1 -- 172.21.15.67:0/3883751734 >> 172.21.15.67:0/3883751734 conn(0x7efc44079440 msgr2=0x7efc4406bbd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T21:18:11.347 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:11.343+0000 7efc4bc1f700 1 -- 172.21.15.67:0/3883751734 shutdown_connections 2024-05-03T21:18:11.347 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:11.344+0000 7efc4bc1f700 1 -- 172.21.15.67:0/3883751734 wait complete. 2024-05-03T21:18:11.347 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 6 2024-05-03T21:18:11.718 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:18:11 smithi067 ceph-mon[29730]: from='client.? 172.21.15.67:0/3883751734' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-03T21:18:11.752 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:18:11 smithi138 ceph-mon[33110]: from='client.? 172.21.15.67:0/3883751734' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-03T21:18:11.753 INFO:journalctl@ceph.mds.f.smithi138.stdout:May 03 21:18:11 smithi138 systemd[1]: Starting Ceph mds.f for 713bc57a-0991-11ef-bc97-c7b262605968... 2024-05-03T21:18:11.995 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:18:11 smithi082 ceph-mon[32933]: from='client.? 172.21.15.67:0/3883751734' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-03T21:18:12.075 INFO:journalctl@ceph.mds.f.smithi138.stdout:May 03 21:18:11 smithi138 podman[63108]: 2024-05-03 21:18:11.653361208 +0000 UTC m=+0.018522958 image pull ee4d846202496ba6096593e8e8becc0700a5eb7ebd95536031187f5f337d4012 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:68e15ae152ef5c805c733e76da8fc1d3dd4ee669d5ba025f457e05725baa24dd 2024-05-03T21:18:12.075 INFO:journalctl@ceph.mds.f.smithi138.stdout:May 03 21:18:11 smithi138 podman[63108]: 2024-05-03 21:18:11.754720904 +0000 UTC m=+0.119882692 container create 753ae948ece540c91b168cc1ed3ee732144027f044fc393e5da50d4ab292f72d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:68e15ae152ef5c805c733e76da8fc1d3dd4ee669d5ba025f457e05725baa24dd, name=ceph-713bc57a-0991-11ef-bc97-c7b262605968-mds-f, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.license=GPLv2, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240430, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, RELEASE=wip-rishabh-testing-20240501.193033-9a047fe, CEPH_POINT_RELEASE=) 2024-05-03T21:18:12.075 INFO:journalctl@ceph.mds.f.smithi138.stdout:May 03 21:18:11 smithi138 podman[63108]: 2024-05-03 21:18:11.965511302 +0000 UTC m=+0.330673049 container init 753ae948ece540c91b168cc1ed3ee732144027f044fc393e5da50d4ab292f72d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:68e15ae152ef5c805c733e76da8fc1d3dd4ee669d5ba025f457e05725baa24dd, name=ceph-713bc57a-0991-11ef-bc97-c7b262605968-mds-f, RELEASE=wip-rishabh-testing-20240501.193033-9a047fe, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240430, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image) 2024-05-03T21:18:12.075 INFO:journalctl@ceph.mds.f.smithi138.stdout:May 03 21:18:11 smithi138 podman[63108]: 2024-05-03 21:18:11.968707157 +0000 UTC m=+0.333868904 container start 753ae948ece540c91b168cc1ed3ee732144027f044fc393e5da50d4ab292f72d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:68e15ae152ef5c805c733e76da8fc1d3dd4ee669d5ba025f457e05725baa24dd, name=ceph-713bc57a-0991-11ef-bc97-c7b262605968-mds-f, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, RELEASE=wip-rishabh-testing-20240501.193033-9a047fe, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.build-date=20240430) 2024-05-03T21:18:12.075 INFO:journalctl@ceph.mds.f.smithi138.stdout:May 03 21:18:11 smithi138 ceph-mds[63127]: -- 172.21.15.138:0/1233326444 <== mon.1 v2:172.21.15.138:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55dea96bba40 con 0x55dea968ec00 2024-05-03T21:18:12.075 INFO:journalctl@ceph.mds.f.smithi138.stdout:May 03 21:18:11 smithi138 ceph-713bc57a-0991-11ef-bc97-c7b262605968-mds-f[63123]: starting mds.f at 2024-05-03T21:18:12.574 INFO:journalctl@ceph.mds.f.smithi138.stdout:May 03 21:18:12 smithi138 bash[63108]: 753ae948ece540c91b168cc1ed3ee732144027f044fc393e5da50d4ab292f72d 2024-05-03T21:18:12.575 INFO:journalctl@ceph.mds.f.smithi138.stdout:May 03 21:18:12 smithi138 systemd[1]: Started Ceph mds.f for 713bc57a-0991-11ef-bc97-c7b262605968. 2024-05-03T21:18:13.471 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:18:13 smithi067 ceph-mon[29730]: pgmap v264: 1 pgs: 1 active+clean; 577 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-03T21:18:13.471 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:18:13 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:18:13.471 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:18:13 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:18:13.471 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:18:13 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:18:13.471 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:18:13 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-03T21:18:13.471 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:18:13 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-03T21:18:13.472 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:18:13 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T21:18:13.472 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:18:13 smithi067 ceph-mon[29730]: Deploying daemon mds.c on smithi138 2024-05-03T21:18:13.472 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:18:13 smithi067 ceph-mon[29730]: mds.? [v2:172.21.15.138:6834/3342864697,v1:172.21.15.138:6835/3342864697] up:boot 2024-05-03T21:18:13.472 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:18:13 smithi067 ceph-mon[29730]: fsmap 6 up:standby 2024-05-03T21:18:13.472 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:18:13 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-05-03T21:18:13.495 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:18:13 smithi082 ceph-mon[32933]: pgmap v264: 1 pgs: 1 active+clean; 577 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-03T21:18:13.496 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:18:13 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:18:13.496 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:18:13 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:18:13.496 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:18:13 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:18:13.496 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:18:13 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-03T21:18:13.496 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:18:13 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-03T21:18:13.496 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:18:13 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T21:18:13.496 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:18:13 smithi082 ceph-mon[32933]: Deploying daemon mds.c on smithi138 2024-05-03T21:18:13.496 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:18:13 smithi082 ceph-mon[32933]: mds.? [v2:172.21.15.138:6834/3342864697,v1:172.21.15.138:6835/3342864697] up:boot 2024-05-03T21:18:13.496 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:18:13 smithi082 ceph-mon[32933]: fsmap 6 up:standby 2024-05-03T21:18:13.496 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:18:13 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-05-03T21:18:13.574 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:18:13 smithi138 ceph-mon[33110]: pgmap v264: 1 pgs: 1 active+clean; 577 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-03T21:18:13.575 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:18:13 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:18:13.575 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:18:13 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:18:13.575 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:18:13 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:18:13.575 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:18:13 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-03T21:18:13.575 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:18:13 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-03T21:18:13.575 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:18:13 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T21:18:13.575 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:18:13 smithi138 ceph-mon[33110]: Deploying daemon mds.c on smithi138 2024-05-03T21:18:13.575 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:18:13 smithi138 ceph-mon[33110]: mds.? [v2:172.21.15.138:6834/3342864697,v1:172.21.15.138:6835/3342864697] up:boot 2024-05-03T21:18:13.575 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:18:13 smithi138 ceph-mon[33110]: fsmap 6 up:standby 2024-05-03T21:18:13.575 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:18:13 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-05-03T21:18:14.059 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9a047fe7599a549042d06a3fb5b6813a203eabea shell --fsid 713bc57a-0991-11ef-bc97-c7b262605968 -- ceph fs dump --format=json 2024-05-03T21:18:14.325 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/713bc57a-0991-11ef-bc97-c7b262605968/mon.a/config 2024-05-03T21:18:14.325 INFO:journalctl@ceph.mds.c.smithi138.stdout:May 03 21:18:14 smithi138 systemd[1]: Starting Ceph mds.c for 713bc57a-0991-11ef-bc97-c7b262605968... 2024-05-03T21:18:14.622 INFO:journalctl@ceph.mds.c.smithi138.stdout:May 03 21:18:14 smithi138 podman[63419]: 2024-05-03 21:18:14.269817366 +0000 UTC m=+0.018360745 image pull ee4d846202496ba6096593e8e8becc0700a5eb7ebd95536031187f5f337d4012 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:68e15ae152ef5c805c733e76da8fc1d3dd4ee669d5ba025f457e05725baa24dd 2024-05-03T21:18:14.623 INFO:journalctl@ceph.mds.c.smithi138.stdout:May 03 21:18:14 smithi138 podman[63419]: 2024-05-03 21:18:14.385497117 +0000 UTC m=+0.134040525 container create 34446d8bd98e59e5f9cedb5125dc87b14ef078c743820e4be7a2786c470966fe (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:68e15ae152ef5c805c733e76da8fc1d3dd4ee669d5ba025f457e05725baa24dd, name=ceph-713bc57a-0991-11ef-bc97-c7b262605968-mds-c, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, RELEASE=wip-rishabh-testing-20240501.193033-9a047fe, CEPH_POINT_RELEASE=, GIT_CLEAN=True, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.build-date=20240430) 2024-05-03T21:18:14.623 INFO:journalctl@ceph.mds.c.smithi138.stdout:May 03 21:18:14 smithi138 podman[63419]: 2024-05-03 21:18:14.621492919 +0000 UTC m=+0.370036282 container init 34446d8bd98e59e5f9cedb5125dc87b14ef078c743820e4be7a2786c470966fe (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:68e15ae152ef5c805c733e76da8fc1d3dd4ee669d5ba025f457e05725baa24dd, name=ceph-713bc57a-0991-11ef-bc97-c7b262605968-mds-c, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240430, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_CLEAN=True, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.vendor=CentOS, RELEASE=wip-rishabh-testing-20240501.193033-9a047fe, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b) 2024-05-03T21:18:15.074 INFO:journalctl@ceph.mds.c.smithi138.stdout:May 03 21:18:14 smithi138 podman[63419]: 2024-05-03 21:18:14.623999549 +0000 UTC m=+0.372542914 container start 34446d8bd98e59e5f9cedb5125dc87b14ef078c743820e4be7a2786c470966fe (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:68e15ae152ef5c805c733e76da8fc1d3dd4ee669d5ba025f457e05725baa24dd, name=ceph-713bc57a-0991-11ef-bc97-c7b262605968-mds-c, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240430, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, RELEASE=wip-rishabh-testing-20240501.193033-9a047fe, CEPH_POINT_RELEASE=, ceph=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0) 2024-05-03T21:18:15.075 INFO:journalctl@ceph.mds.c.smithi138.stdout:May 03 21:18:14 smithi138 ceph-mds[63437]: -- 172.21.15.138:0/700330900 <== mon.1 v2:172.21.15.138:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55a256929a40 con 0x55a2568fd000 2024-05-03T21:18:15.075 INFO:journalctl@ceph.mds.c.smithi138.stdout:May 03 21:18:14 smithi138 ceph-713bc57a-0991-11ef-bc97-c7b262605968-mds-c[63433]: starting mds.c at 2024-05-03T21:18:15.075 INFO:journalctl@ceph.mds.c.smithi138.stdout:May 03 21:18:14 smithi138 bash[63419]: 34446d8bd98e59e5f9cedb5125dc87b14ef078c743820e4be7a2786c470966fe 2024-05-03T21:18:15.075 INFO:journalctl@ceph.mds.c.smithi138.stdout:May 03 21:18:14 smithi138 systemd[1]: Started Ceph mds.c for 713bc57a-0991-11ef-bc97-c7b262605968. 2024-05-03T21:18:15.411 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:18:15 smithi067 ceph-mon[29730]: pgmap v265: 1 pgs: 1 active+clean; 577 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-03T21:18:15.412 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:18:15 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:18:15.412 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:18:15 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:18:15.412 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:18:15 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:18:15.412 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:18:15 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-03T21:18:15.412 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:18:15 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-03T21:18:15.412 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:18:15 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T21:18:15.469 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:18:15 smithi082 ceph-mon[32933]: pgmap v265: 1 pgs: 1 active+clean; 577 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-03T21:18:15.469 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:18:15 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:18:15.469 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:18:15 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:18:15.469 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:18:15 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:18:15.469 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:18:15 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-03T21:18:15.469 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:18:15 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-03T21:18:15.469 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:18:15 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T21:18:15.574 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:18:15 smithi138 ceph-mon[33110]: pgmap v265: 1 pgs: 1 active+clean; 577 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-03T21:18:15.575 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:18:15 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:18:15.575 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:18:15 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:18:15.575 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:18:15 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:18:15.575 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:18:15 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-03T21:18:15.575 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:18:15 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-03T21:18:15.575 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:18:15 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T21:18:15.936 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:15.932+0000 7f5f83098700 1 -- 172.21.15.67:0/1243110073 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5f7c102d60 msgr2=0x7f5f7c10f250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:18:15.936 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:15.932+0000 7f5f83098700 1 --2- 172.21.15.67:0/1243110073 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5f7c102d60 0x7f5f7c10f250 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f5f74009990 tx=0x7f5f74009ca0 comp rx=0 tx=0).stop 2024-05-03T21:18:15.936 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:15.932+0000 7f5f83098700 1 -- 172.21.15.67:0/1243110073 shutdown_connections 2024-05-03T21:18:15.936 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:15.932+0000 7f5f83098700 1 --2- 172.21.15.67:0/1243110073 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5f7c102d60 0x7f5f7c10f250 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:18:15.936 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:15.932+0000 7f5f83098700 1 --2- 172.21.15.67:0/1243110073 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f5f7c108410 0x7f5f7c1087f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:18:15.936 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:15.932+0000 7f5f83098700 1 --2- 172.21.15.67:0/1243110073 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f5f7c1023a0 0x7f5f7c102820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:18:15.936 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:15.932+0000 7f5f83098700 1 -- 172.21.15.67:0/1243110073 >> 172.21.15.67:0/1243110073 conn(0x7f5f7c079350 msgr2=0x7f5f7c079760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T21:18:15.936 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:15.932+0000 7f5f83098700 1 -- 172.21.15.67:0/1243110073 shutdown_connections 2024-05-03T21:18:15.936 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:15.932+0000 7f5f83098700 1 -- 172.21.15.67:0/1243110073 wait complete. 2024-05-03T21:18:15.936 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:15.932+0000 7f5f83098700 1 Processor -- start 2024-05-03T21:18:15.936 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:15.932+0000 7f5f83098700 1 -- start start 2024-05-03T21:18:15.936 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:15.933+0000 7f5f83098700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f5f7c1023a0 0x7f5f7c1a4f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:18:15.937 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:15.933+0000 7f5f83098700 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f5f7c102d60 0x7f5f7c1a5450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:18:15.937 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:15.933+0000 7f5f83098700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5f7c108410 0x7f5f7c1a04a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:18:15.937 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:15.933+0000 7f5f83098700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f5f7c1127c0 con 0x7f5f7c108410 2024-05-03T21:18:15.937 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:15.933+0000 7f5f83098700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f5f7c112640 con 0x7f5f7c1023a0 2024-05-03T21:18:15.937 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:15.933+0000 7f5f83098700 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f5f7c112940 con 0x7f5f7c102d60 2024-05-03T21:18:15.937 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:15.933+0000 7f5f81635700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5f7c108410 0x7f5f7c1a04a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:18:15.937 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:15.933+0000 7f5f80e34700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f5f7c1023a0 0x7f5f7c1a4f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:18:15.937 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:15.933+0000 7f5f81635700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5f7c108410 0x7f5f7c1a04a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:47622/0 (socket says 172.21.15.67:47622) 2024-05-03T21:18:15.937 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:15.933+0000 7f5f80e34700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f5f7c1023a0 0x7f5f7c1a4f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.67:60200/0 (socket says 172.21.15.67:60200) 2024-05-03T21:18:15.937 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:15.933+0000 7f5f7bfff700 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f5f7c102d60 0x7f5f7c1a5450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:18:15.937 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:15.933+0000 7f5f81635700 1 -- 172.21.15.67:0/1550151703 learned_addr learned my addr 172.21.15.67:0/1550151703 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-03T21:18:15.937 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:15.933+0000 7f5f80e34700 1 -- 172.21.15.67:0/1550151703 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f5f7c102d60 msgr2=0x7f5f7c1a5450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:18:15.937 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:15.933+0000 7f5f80e34700 1 --2- 172.21.15.67:0/1550151703 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f5f7c102d60 0x7f5f7c1a5450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:18:15.938 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:15.933+0000 7f5f80e34700 1 -- 172.21.15.67:0/1550151703 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5f7c108410 msgr2=0x7f5f7c1a04a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:18:15.938 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:15.933+0000 7f5f80e34700 1 --2- 172.21.15.67:0/1550151703 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5f7c108410 0x7f5f7c1a04a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:18:15.938 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:15.933+0000 7f5f80e34700 1 -- 172.21.15.67:0/1550151703 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5f7c1a0c30 con 0x7f5f7c1023a0 2024-05-03T21:18:15.938 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:15.933+0000 7f5f81635700 1 --2- 172.21.15.67:0/1550151703 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5f7c108410 0x7f5f7c1a04a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-03T21:18:15.938 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:15.933+0000 7f5f7bfff700 1 --2- 172.21.15.67:0/1550151703 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f5f7c102d60 0x7f5f7c1a5450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-03T21:18:15.938 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:15.933+0000 7f5f80e34700 1 --2- 172.21.15.67:0/1550151703 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f5f7c1023a0 0x7f5f7c1a4f10 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f5f6c00ef10 tx=0x7f5f6c00c3a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T21:18:15.938 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:15.934+0000 7f5f79ffb700 1 -- 172.21.15.67:0/1550151703 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f6c00c9c0 con 0x7f5f7c1023a0 2024-05-03T21:18:15.938 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:15.934+0000 7f5f83098700 1 -- 172.21.15.67:0/1550151703 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5f7c1a0ec0 con 0x7f5f7c1023a0 2024-05-03T21:18:15.938 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:15.934+0000 7f5f79ffb700 1 -- 172.21.15.67:0/1550151703 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(28 keys) ==== 1170+0+0 (secure 0 0 0) 0x7f5f6c019070 con 0x7f5f7c1023a0 2024-05-03T21:18:15.938 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:15.934+0000 7f5f79ffb700 1 -- 172.21.15.67:0/1550151703 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f6c0146e0 con 0x7f5f7c1023a0 2024-05-03T21:18:15.939 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:15.934+0000 7f5f83098700 1 -- 172.21.15.67:0/1550151703 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5f7c1ace40 con 0x7f5f7c1023a0 2024-05-03T21:18:15.939 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:15.935+0000 7f5f83098700 1 -- 172.21.15.67:0/1550151703 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5f7c06a190 con 0x7f5f7c1023a0 2024-05-03T21:18:15.939 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:15.936+0000 7f5f79ffb700 1 -- 172.21.15.67:0/1550151703 <== mon.2 v2:172.21.15.82:3300/0 4 ==== mgrmap(e 16) ==== 149652+0+0 (secure 0 0 0) 0x7f5f6c012050 con 0x7f5f7c1023a0 2024-05-03T21:18:15.939 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:15.937+0000 7f5f79ffb700 1 --2- 172.21.15.67:0/1550151703 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f5f680b2880 0x7f5f680b4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:18:15.939 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:15.937+0000 7f5f7bfff700 1 --2- 172.21.15.67:0/1550151703 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f5f680b2880 0x7f5f680b4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:18:15.939 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:15.937+0000 7f5f79ffb700 1 -- 172.21.15.67:0/1550151703 <== mon.2 v2:172.21.15.82:3300/0 5 ==== osd_map(75..75 src has 1..75) ==== 7127+0+0 (secure 0 0 0) 0x7f5f6c0e1c90 con 0x7f5f7c1023a0 2024-05-03T21:18:15.940 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:15.938+0000 7f5f7bfff700 1 --2- 172.21.15.67:0/1550151703 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f5f680b2880 0x7f5f680b4d40 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f5f7c10f5c0 tx=0x7f5f640073d0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T21:18:15.948 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:15.946+0000 7f5f79ffb700 1 -- 172.21.15.67:0/1550151703 <== mon.2 v2:172.21.15.82:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191839 (secure 0 0 0) 0x7f5f6c010040 con 0x7f5f7c1023a0 2024-05-03T21:18:16.217 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:16.214+0000 7f5f83098700 1 -- 172.21.15.67:0/1550151703 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f5f7c1ad010 con 0x7f5f7c1023a0 2024-05-03T21:18:16.217 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:18:16 smithi067 ceph-mon[29730]: Deploying daemon mds.h on smithi082 2024-05-03T21:18:16.217 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:18:16 smithi067 ceph-mon[29730]: mds.? [v2:172.21.15.138:6836/2203273249,v1:172.21.15.138:6837/2203273249] up:boot 2024-05-03T21:18:16.217 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:18:16 smithi067 ceph-mon[29730]: fsmap 7 up:standby 2024-05-03T21:18:16.217 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:18:16 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-05-03T21:18:16.218 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:16.215+0000 7f5f79ffb700 1 -- 172.21.15.67:0/1550151703 <== mon.2 v2:172.21.15.82:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) ==== 93+0+6349 (secure 0 0 0) 0x7f5f6c0a9e30 con 0x7f5f7c1023a0 2024-05-03T21:18:16.218 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-05-03T21:18:16.218 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":8,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14541,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6835/4224826546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":4224826546},{"type":"v1","addr":"172.21.15.67:6835","nonce":4224826546}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14553,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6837/2485231621","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6836","nonce":2485231621},{"type":"v1","addr":"172.21.15.67:6837","nonce":2485231621}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24376,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.138:6833/3028022602","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6832","nonce":3028022602},{"type":"v1","addr":"172.21.15.138:6833","nonce":3028022602}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24400,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.138:6835/3342864697","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6834","nonce":3342864697},{"type":"v1","addr":"172.21.15.138:6835","nonce":3342864697}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24406,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.138:6837/2203273249","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6836","nonce":2203273249},{"type":"v1","addr":"172.21.15.138:6837","nonce":2203273249}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24407,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.82:6833/1127287628","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6832","nonce":1127287628},{"type":"v1","addr":"172.21.15.82:6833","nonce":1127287628}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24425,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.82:6835/280518761","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6834","nonce":280518761},{"type":"v1","addr":"172.21.15.82:6835","nonce":280518761}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6}],"filesystems":[]} 2024-05-03T21:18:16.220 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:16.217+0000 7f5f83098700 1 -- 172.21.15.67:0/1550151703 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f5f680b2880 msgr2=0x7f5f680b4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:18:16.220 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:16.217+0000 7f5f83098700 1 --2- 172.21.15.67:0/1550151703 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f5f680b2880 0x7f5f680b4d40 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f5f7c10f5c0 tx=0x7f5f640073d0 comp rx=0 tx=0).stop 2024-05-03T21:18:16.220 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:16.217+0000 7f5f83098700 1 -- 172.21.15.67:0/1550151703 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f5f7c1023a0 msgr2=0x7f5f7c1a4f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:18:16.220 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:16.217+0000 7f5f83098700 1 --2- 172.21.15.67:0/1550151703 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f5f7c1023a0 0x7f5f7c1a4f10 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f5f6c00ef10 tx=0x7f5f6c00c3a0 comp rx=0 tx=0).stop 2024-05-03T21:18:16.220 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:16.217+0000 7f5f83098700 1 -- 172.21.15.67:0/1550151703 shutdown_connections 2024-05-03T21:18:16.220 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:16.217+0000 7f5f83098700 1 --2- 172.21.15.67:0/1550151703 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f5f680b2880 0x7f5f680b4d40 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:18:16.220 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:16.217+0000 7f5f83098700 1 --2- 172.21.15.67:0/1550151703 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5f7c108410 0x7f5f7c1a04a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:18:16.220 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:16.217+0000 7f5f83098700 1 --2- 172.21.15.67:0/1550151703 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f5f7c1023a0 0x7f5f7c1a4f10 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:18:16.220 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:16.217+0000 7f5f83098700 1 --2- 172.21.15.67:0/1550151703 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f5f7c102d60 0x7f5f7c1a5450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:18:16.221 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:16.217+0000 7f5f83098700 1 -- 172.21.15.67:0/1550151703 >> 172.21.15.67:0/1550151703 conn(0x7f5f7c079350 msgr2=0x7f5f7c100c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T21:18:16.221 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:16.218+0000 7f5f83098700 1 -- 172.21.15.67:0/1550151703 shutdown_connections 2024-05-03T21:18:16.221 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:16.218+0000 7f5f83098700 1 -- 172.21.15.67:0/1550151703 wait complete. 2024-05-03T21:18:16.221 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 8 2024-05-03T21:18:16.324 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:18:16 smithi138 ceph-mon[33110]: Deploying daemon mds.h on smithi082 2024-05-03T21:18:16.324 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:18:16 smithi138 ceph-mon[33110]: mds.? [v2:172.21.15.138:6836/2203273249,v1:172.21.15.138:6837/2203273249] up:boot 2024-05-03T21:18:16.324 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:18:16 smithi138 ceph-mon[33110]: fsmap 7 up:standby 2024-05-03T21:18:16.325 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:18:16 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-05-03T21:18:16.473 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:18:16 smithi082 ceph-mon[32933]: Deploying daemon mds.h on smithi082 2024-05-03T21:18:16.473 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:18:16 smithi082 ceph-mon[32933]: mds.? [v2:172.21.15.138:6836/2203273249,v1:172.21.15.138:6837/2203273249] up:boot 2024-05-03T21:18:16.473 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:18:16 smithi082 ceph-mon[32933]: fsmap 7 up:standby 2024-05-03T21:18:16.473 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:18:16 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-05-03T21:18:17.150 INFO:journalctl@ceph.mds.h.smithi082.stdout:May 03 21:18:16 smithi082 systemd[1]: Starting Ceph mds.h for 713bc57a-0991-11ef-bc97-c7b262605968... 2024-05-03T21:18:17.324 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:18:17 smithi138 ceph-mon[33110]: pgmap v266: 1 pgs: 1 active+clean; 577 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-03T21:18:17.324 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:18:17 smithi138 ceph-mon[33110]: from='client.? 172.21.15.67:0/1550151703' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-03T21:18:17.411 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:18:17 smithi082 ceph-mon[32933]: pgmap v266: 1 pgs: 1 active+clean; 577 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-03T21:18:17.411 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:18:17 smithi082 ceph-mon[32933]: from='client.? 172.21.15.67:0/1550151703' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-03T21:18:17.412 INFO:journalctl@ceph.mds.h.smithi082.stdout:May 03 21:18:17 smithi082 podman[63035]: 2024-05-03 21:18:17.05138433 +0000 UTC m=+0.018239371 image pull ee4d846202496ba6096593e8e8becc0700a5eb7ebd95536031187f5f337d4012 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:68e15ae152ef5c805c733e76da8fc1d3dd4ee669d5ba025f457e05725baa24dd 2024-05-03T21:18:17.412 INFO:journalctl@ceph.mds.h.smithi082.stdout:May 03 21:18:17 smithi082 podman[63035]: 2024-05-03 21:18:17.174386333 +0000 UTC m=+0.141241373 container create 27462ab2a791dfc6474de794265842ea5b137f9f32670f255094a984d736ce49 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:68e15ae152ef5c805c733e76da8fc1d3dd4ee669d5ba025f457e05725baa24dd, name=ceph-713bc57a-0991-11ef-bc97-c7b262605968-mds-h, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.build-date=20240430, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, RELEASE=wip-rishabh-testing-20240501.193033-9a047fe, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD) 2024-05-03T21:18:17.412 INFO:journalctl@ceph.mds.h.smithi082.stdout:May 03 21:18:17 smithi082 podman[63035]: 2024-05-03 21:18:17.410370243 +0000 UTC m=+0.377225283 container init 27462ab2a791dfc6474de794265842ea5b137f9f32670f255094a984d736ce49 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:68e15ae152ef5c805c733e76da8fc1d3dd4ee669d5ba025f457e05725baa24dd, name=ceph-713bc57a-0991-11ef-bc97-c7b262605968-mds-h, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, ceph=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240430, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, RELEASE=wip-rishabh-testing-20240501.193033-9a047fe) 2024-05-03T21:18:17.471 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:18:17 smithi067 ceph-mon[29730]: pgmap v266: 1 pgs: 1 active+clean; 577 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-03T21:18:17.471 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:18:17 smithi067 ceph-mon[29730]: from='client.? 172.21.15.67:0/1550151703' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-03T21:18:17.746 INFO:journalctl@ceph.mds.h.smithi082.stdout:May 03 21:18:17 smithi082 podman[63035]: 2024-05-03 21:18:17.412929947 +0000 UTC m=+0.379784989 container start 27462ab2a791dfc6474de794265842ea5b137f9f32670f255094a984d736ce49 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:68e15ae152ef5c805c733e76da8fc1d3dd4ee669d5ba025f457e05725baa24dd, name=ceph-713bc57a-0991-11ef-bc97-c7b262605968-mds-h, org.label-schema.build-date=20240430, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, RELEASE=wip-rishabh-testing-20240501.193033-9a047fe, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_CLEAN=True, ceph=True) 2024-05-03T21:18:17.746 INFO:journalctl@ceph.mds.h.smithi082.stdout:May 03 21:18:17 smithi082 ceph-mds[63054]: -- 172.21.15.82:0/3643471406 <== mon.2 v2:172.21.15.82:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55b39ef53a40 con 0x55b39ef26c00 2024-05-03T21:18:17.746 INFO:journalctl@ceph.mds.h.smithi082.stdout:May 03 21:18:17 smithi082 ceph-713bc57a-0991-11ef-bc97-c7b262605968-mds-h[63050]: starting mds.h at 2024-05-03T21:18:17.746 INFO:journalctl@ceph.mds.h.smithi082.stdout:May 03 21:18:17 smithi082 bash[63035]: 27462ab2a791dfc6474de794265842ea5b137f9f32670f255094a984d736ce49 2024-05-03T21:18:17.746 INFO:journalctl@ceph.mds.h.smithi082.stdout:May 03 21:18:17 smithi082 systemd[1]: Started Ceph mds.h for 713bc57a-0991-11ef-bc97-c7b262605968. 2024-05-03T21:18:18.824 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:18:18 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:18:18.824 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:18:18 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:18:18.825 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:18:18 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:18:18.825 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:18:18 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-03T21:18:18.825 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:18:18 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-03T21:18:18.825 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:18:18 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T21:18:18.825 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:18:18 smithi138 ceph-mon[33110]: Deploying daemon mds.i on smithi138 2024-05-03T21:18:18.825 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:18:18 smithi138 ceph-mon[33110]: pgmap v267: 1 pgs: 1 active+clean; 577 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-03T21:18:18.825 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:18:18 smithi138 ceph-mon[33110]: mds.? [v2:172.21.15.82:6836/1275987852,v1:172.21.15.82:6837/1275987852] up:boot 2024-05-03T21:18:18.825 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:18:18 smithi138 ceph-mon[33110]: fsmap 8 up:standby 2024-05-03T21:18:18.825 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:18:18 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-05-03T21:18:18.866 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9a047fe7599a549042d06a3fb5b6813a203eabea shell --fsid 713bc57a-0991-11ef-bc97-c7b262605968 -- ceph fs dump --format=json 2024-05-03T21:18:18.894 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:18:18 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:18:18.894 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:18:18 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:18:18.894 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:18:18 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:18:18.894 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:18:18 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-03T21:18:18.894 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:18:18 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-03T21:18:18.894 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:18:18 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T21:18:18.894 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:18:18 smithi067 ceph-mon[29730]: Deploying daemon mds.i on smithi138 2024-05-03T21:18:18.894 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:18:18 smithi067 ceph-mon[29730]: pgmap v267: 1 pgs: 1 active+clean; 577 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-03T21:18:18.894 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:18:18 smithi067 ceph-mon[29730]: mds.? [v2:172.21.15.82:6836/1275987852,v1:172.21.15.82:6837/1275987852] up:boot 2024-05-03T21:18:18.894 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:18:18 smithi067 ceph-mon[29730]: fsmap 8 up:standby 2024-05-03T21:18:18.895 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:18:18 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-05-03T21:18:18.995 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:18:18 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:18:18.996 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:18:18 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:18:18.996 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:18:18 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:18:18.996 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:18:18 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-03T21:18:18.996 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:18:18 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-03T21:18:18.996 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:18:18 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T21:18:18.996 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:18:18 smithi082 ceph-mon[32933]: Deploying daemon mds.i on smithi138 2024-05-03T21:18:18.996 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:18:18 smithi082 ceph-mon[32933]: pgmap v267: 1 pgs: 1 active+clean; 577 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-03T21:18:18.996 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:18:18 smithi082 ceph-mon[32933]: mds.? [v2:172.21.15.82:6836/1275987852,v1:172.21.15.82:6837/1275987852] up:boot 2024-05-03T21:18:18.996 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:18:18 smithi082 ceph-mon[32933]: fsmap 8 up:standby 2024-05-03T21:18:18.996 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:18:18 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-05-03T21:18:19.126 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/713bc57a-0991-11ef-bc97-c7b262605968/mon.a/config 2024-05-03T21:18:19.734 INFO:journalctl@ceph.mds.i.smithi138.stdout:May 03 21:18:19 smithi138 systemd[1]: Starting Ceph mds.i for 713bc57a-0991-11ef-bc97-c7b262605968... 2024-05-03T21:18:20.075 INFO:journalctl@ceph.mds.i.smithi138.stdout:May 03 21:18:19 smithi138 podman[63739]: 2024-05-03 21:18:19.73220189 +0000 UTC m=+0.115443312 container create c4cc8b05b928e730689fa57d76ec19a549bb0669f2dfbc35e20ae2a0e8ee107e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:68e15ae152ef5c805c733e76da8fc1d3dd4ee669d5ba025f457e05725baa24dd, name=ceph-713bc57a-0991-11ef-bc97-c7b262605968-mds-i, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, RELEASE=wip-rishabh-testing-20240501.193033-9a047fe, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240430) 2024-05-03T21:18:20.075 INFO:journalctl@ceph.mds.i.smithi138.stdout:May 03 21:18:19 smithi138 podman[63739]: 2024-05-03 21:18:19.635484524 +0000 UTC m=+0.018725966 image pull ee4d846202496ba6096593e8e8becc0700a5eb7ebd95536031187f5f337d4012 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:68e15ae152ef5c805c733e76da8fc1d3dd4ee669d5ba025f457e05725baa24dd 2024-05-03T21:18:20.075 INFO:journalctl@ceph.mds.i.smithi138.stdout:May 03 21:18:19 smithi138 podman[63739]: 2024-05-03 21:18:19.942932961 +0000 UTC m=+0.326174382 container init c4cc8b05b928e730689fa57d76ec19a549bb0669f2dfbc35e20ae2a0e8ee107e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:68e15ae152ef5c805c733e76da8fc1d3dd4ee669d5ba025f457e05725baa24dd, name=ceph-713bc57a-0991-11ef-bc97-c7b262605968-mds-i, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, RELEASE=wip-rishabh-testing-20240501.193033-9a047fe, ceph=True, org.label-schema.build-date=20240430, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image) 2024-05-03T21:18:20.075 INFO:journalctl@ceph.mds.i.smithi138.stdout:May 03 21:18:19 smithi138 podman[63739]: 2024-05-03 21:18:19.945836165 +0000 UTC m=+0.329077592 container start c4cc8b05b928e730689fa57d76ec19a549bb0669f2dfbc35e20ae2a0e8ee107e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:68e15ae152ef5c805c733e76da8fc1d3dd4ee669d5ba025f457e05725baa24dd, name=ceph-713bc57a-0991-11ef-bc97-c7b262605968-mds-i, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240430, RELEASE=wip-rishabh-testing-20240501.193033-9a047fe, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=) 2024-05-03T21:18:20.075 INFO:journalctl@ceph.mds.i.smithi138.stdout:May 03 21:18:19 smithi138 ceph-mds[63758]: -- 172.21.15.138:0/3359739287 <== mon.1 v2:172.21.15.138:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x562ae379fa40 con 0x562ae3772800 2024-05-03T21:18:20.075 INFO:journalctl@ceph.mds.i.smithi138.stdout:May 03 21:18:19 smithi138 ceph-713bc57a-0991-11ef-bc97-c7b262605968-mds-i[63754]: starting mds.i at 2024-05-03T21:18:20.574 INFO:journalctl@ceph.mds.i.smithi138.stdout:May 03 21:18:20 smithi138 bash[63739]: c4cc8b05b928e730689fa57d76ec19a549bb0669f2dfbc35e20ae2a0e8ee107e 2024-05-03T21:18:20.574 INFO:journalctl@ceph.mds.i.smithi138.stdout:May 03 21:18:20 smithi138 systemd[1]: Started Ceph mds.i for 713bc57a-0991-11ef-bc97-c7b262605968. 2024-05-03T21:18:20.601 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:20.599+0000 7fa9c198f700 1 -- 172.21.15.67:0/1197264971 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa9bc1023a0 msgr2=0x7fa9bc102820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:18:20.602 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:20.599+0000 7fa9c198f700 1 --2- 172.21.15.67:0/1197264971 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa9bc1023a0 0x7fa9bc102820 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fa9b0009990 tx=0x7fa9b0009ca0 comp rx=0 tx=0).stop 2024-05-03T21:18:20.602 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:20.599+0000 7fa9c198f700 1 -- 172.21.15.67:0/1197264971 shutdown_connections 2024-05-03T21:18:20.602 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:20.599+0000 7fa9c198f700 1 --2- 172.21.15.67:0/1197264971 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fa9bc102d60 0x7fa9bc10f250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:18:20.602 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:20.599+0000 7fa9c198f700 1 --2- 172.21.15.67:0/1197264971 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fa9bc108410 0x7fa9bc1087f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:18:20.602 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:20.599+0000 7fa9c198f700 1 --2- 172.21.15.67:0/1197264971 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa9bc1023a0 0x7fa9bc102820 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:18:20.602 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:20.599+0000 7fa9c198f700 1 -- 172.21.15.67:0/1197264971 >> 172.21.15.67:0/1197264971 conn(0x7fa9bc079350 msgr2=0x7fa9bc079760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T21:18:20.602 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:20.599+0000 7fa9c198f700 1 -- 172.21.15.67:0/1197264971 shutdown_connections 2024-05-03T21:18:20.602 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:20.599+0000 7fa9c198f700 1 -- 172.21.15.67:0/1197264971 wait complete. 2024-05-03T21:18:20.604 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:20.600+0000 7fa9c198f700 1 Processor -- start 2024-05-03T21:18:20.604 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:20.600+0000 7fa9c198f700 1 -- start start 2024-05-03T21:18:20.604 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:20.600+0000 7fa9c198f700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fa9bc1023a0 0x7fa9bc1042b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:18:20.604 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:20.600+0000 7fa9c198f700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa9bc102d60 0x7fa9bc1047f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:18:20.604 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:20.601+0000 7fa9c198f700 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fa9bc108410 0x7fa9bc104d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:18:20.604 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:20.601+0000 7fa9c198f700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fa9bc1127c0 con 0x7fa9bc102d60 2024-05-03T21:18:20.604 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:20.601+0000 7fa9c198f700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7fa9bc112640 con 0x7fa9bc1023a0 2024-05-03T21:18:20.604 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:20.601+0000 7fa9c198f700 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7fa9bc112940 con 0x7fa9bc108410 2024-05-03T21:18:20.604 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:20.601+0000 7fa9ba7fc700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa9bc102d60 0x7fa9bc1047f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:18:20.604 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:20.601+0000 7fa9ba7fc700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa9bc102d60 0x7fa9bc1047f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:47636/0 (socket says 172.21.15.67:47636) 2024-05-03T21:18:20.605 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:20.601+0000 7fa9ba7fc700 1 -- 172.21.15.67:0/4028775669 learned_addr learned my addr 172.21.15.67:0/4028775669 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-03T21:18:20.605 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:20.601+0000 7fa9bb7fe700 1 --2- 172.21.15.67:0/4028775669 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fa9bc108410 0x7fa9bc104d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:18:20.605 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:20.601+0000 7fa9baffd700 1 --2- 172.21.15.67:0/4028775669 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fa9bc1023a0 0x7fa9bc1042b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:18:20.605 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:20.601+0000 7fa9ba7fc700 1 -- 172.21.15.67:0/4028775669 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fa9bc108410 msgr2=0x7fa9bc104d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:18:20.605 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:20.601+0000 7fa9ba7fc700 1 --2- 172.21.15.67:0/4028775669 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fa9bc108410 0x7fa9bc104d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:18:20.605 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:20.601+0000 7fa9ba7fc700 1 -- 172.21.15.67:0/4028775669 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fa9bc1023a0 msgr2=0x7fa9bc1042b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:18:20.605 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:20.601+0000 7fa9ba7fc700 1 --2- 172.21.15.67:0/4028775669 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fa9bc1023a0 0x7fa9bc1042b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:18:20.605 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:20.601+0000 7fa9ba7fc700 1 -- 172.21.15.67:0/4028775669 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa9bc1a8260 con 0x7fa9bc102d60 2024-05-03T21:18:20.605 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:20.601+0000 7fa9baffd700 1 --2- 172.21.15.67:0/4028775669 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fa9bc1023a0 0x7fa9bc1042b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-03T21:18:20.605 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:20.601+0000 7fa9ba7fc700 1 --2- 172.21.15.67:0/4028775669 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa9bc102d60 0x7fa9bc1047f0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fa9b0003b40 tx=0x7fa9b0003b70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T21:18:20.605 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:20.602+0000 7fa9c098d700 1 -- 172.21.15.67:0/4028775669 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9b0003e40 con 0x7fa9bc102d60 2024-05-03T21:18:20.605 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:20.602+0000 7fa9c098d700 1 -- 172.21.15.67:0/4028775669 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(28 keys) ==== 1170+0+0 (secure 0 0 0) 0x7fa9b000f460 con 0x7fa9bc102d60 2024-05-03T21:18:20.605 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:20.602+0000 7fa9c098d700 1 -- 172.21.15.67:0/4028775669 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9b0017650 con 0x7fa9bc102d60 2024-05-03T21:18:20.605 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:20.602+0000 7fa9c198f700 1 -- 172.21.15.67:0/4028775669 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa9bc1a84f0 con 0x7fa9bc102d60 2024-05-03T21:18:20.605 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:20.602+0000 7fa9c198f700 1 -- 172.21.15.67:0/4028775669 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa9bc1a8890 con 0x7fa9bc102d60 2024-05-03T21:18:20.606 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:20.603+0000 7fa9c198f700 1 -- 172.21.15.67:0/4028775669 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa9bc06a190 con 0x7fa9bc102d60 2024-05-03T21:18:20.607 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:20.605+0000 7fa9c098d700 1 -- 172.21.15.67:0/4028775669 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 16) ==== 149652+0+0 (secure 0 0 0) 0x7fa9b000f610 con 0x7fa9bc102d60 2024-05-03T21:18:20.608 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:20.606+0000 7fa9c098d700 1 --2- 172.21.15.67:0/4028775669 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7fa9a40b2880 0x7fa9a40b4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:18:20.608 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:20.606+0000 7fa9baffd700 1 --2- 172.21.15.67:0/4028775669 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7fa9a40b2880 0x7fa9a40b4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:18:20.608 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:20.606+0000 7fa9c098d700 1 -- 172.21.15.67:0/4028775669 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(75..75 src has 1..75) ==== 7127+0+0 (secure 0 0 0) 0x7fa9b00e2d50 con 0x7fa9bc102d60 2024-05-03T21:18:20.608 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:20.606+0000 7fa9baffd700 1 --2- 172.21.15.67:0/4028775669 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7fa9a40b2880 0x7fa9a40b4d40 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fa9ac005e10 tx=0x7fa9ac005d60 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T21:18:20.611 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:20.610+0000 7fa9c098d700 1 -- 172.21.15.67:0/4028775669 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191839 (secure 0 0 0) 0x7fa9b0013070 con 0x7fa9bc102d60 2024-05-03T21:18:20.889 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:20.887+0000 7fa9c198f700 1 -- 172.21.15.67:0/4028775669 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa9bc1a8b20 con 0x7fa9bc102d60 2024-05-03T21:18:20.889 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:20.887+0000 7fa9c098d700 1 -- 172.21.15.67:0/4028775669 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) ==== 94+0+8001 (secure 0 0 0) 0x7fa9b00aafa0 con 0x7fa9bc102d60 2024-05-03T21:18:20.890 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-05-03T21:18:20.890 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":10,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14541,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6835/4224826546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":4224826546},{"type":"v1","addr":"172.21.15.67:6835","nonce":4224826546}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14553,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6837/2485231621","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6836","nonce":2485231621},{"type":"v1","addr":"172.21.15.67:6837","nonce":2485231621}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24376,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.138:6833/3028022602","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6832","nonce":3028022602},{"type":"v1","addr":"172.21.15.138:6833","nonce":3028022602}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24400,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.138:6835/3342864697","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6834","nonce":3342864697},{"type":"v1","addr":"172.21.15.138:6835","nonce":3342864697}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24406,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.138:6837/2203273249","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6836","nonce":2203273249},{"type":"v1","addr":"172.21.15.138:6837","nonce":2203273249}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24407,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.82:6833/1127287628","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6832","nonce":1127287628},{"type":"v1","addr":"172.21.15.82:6833","nonce":1127287628}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24418,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.138:6839/334382443","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6838","nonce":334382443},{"type":"v1","addr":"172.21.15.138:6839","nonce":334382443}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10},{"gid":24425,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.82:6835/280518761","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6834","nonce":280518761},{"type":"v1","addr":"172.21.15.82:6835","nonce":280518761}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24449,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.82:6837/1275987852","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6836","nonce":1275987852},{"type":"v1","addr":"172.21.15.82:6837","nonce":1275987852}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9}],"filesystems":[]} 2024-05-03T21:18:20.892 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:20.889+0000 7fa9c198f700 1 -- 172.21.15.67:0/4028775669 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7fa9a40b2880 msgr2=0x7fa9a40b4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:18:20.892 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:20.889+0000 7fa9c198f700 1 --2- 172.21.15.67:0/4028775669 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7fa9a40b2880 0x7fa9a40b4d40 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fa9ac005e10 tx=0x7fa9ac005d60 comp rx=0 tx=0).stop 2024-05-03T21:18:20.892 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:20.889+0000 7fa9c198f700 1 -- 172.21.15.67:0/4028775669 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa9bc102d60 msgr2=0x7fa9bc1047f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:18:20.892 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:20.889+0000 7fa9c198f700 1 --2- 172.21.15.67:0/4028775669 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa9bc102d60 0x7fa9bc1047f0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fa9b0003b40 tx=0x7fa9b0003b70 comp rx=0 tx=0).stop 2024-05-03T21:18:20.892 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:20.889+0000 7fa9c198f700 1 -- 172.21.15.67:0/4028775669 shutdown_connections 2024-05-03T21:18:20.892 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:20.889+0000 7fa9c198f700 1 --2- 172.21.15.67:0/4028775669 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7fa9a40b2880 0x7fa9a40b4d40 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:18:20.892 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:20.889+0000 7fa9c198f700 1 --2- 172.21.15.67:0/4028775669 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fa9bc108410 0x7fa9bc104d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:18:20.892 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:20.889+0000 7fa9c198f700 1 --2- 172.21.15.67:0/4028775669 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fa9bc1023a0 0x7fa9bc1042b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:18:20.892 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:20.889+0000 7fa9c198f700 1 --2- 172.21.15.67:0/4028775669 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa9bc102d60 0x7fa9bc1047f0 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:18:20.892 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:20.889+0000 7fa9c198f700 1 -- 172.21.15.67:0/4028775669 >> 172.21.15.67:0/4028775669 conn(0x7fa9bc079350 msgr2=0x7fa9bc106450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T21:18:20.892 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:20.890+0000 7fa9c198f700 1 -- 172.21.15.67:0/4028775669 shutdown_connections 2024-05-03T21:18:20.892 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:20.890+0000 7fa9c198f700 1 -- 172.21.15.67:0/4028775669 wait complete. 2024-05-03T21:18:20.893 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 10 2024-05-03T21:18:21.471 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:18:21 smithi067 ceph-mon[29730]: pgmap v268: 1 pgs: 1 active+clean; 577 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-03T21:18:21.471 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:18:21 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:18:21.471 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:18:21 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:18:21.471 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:18:21 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:18:21.472 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:18:21 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-03T21:18:21.472 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:18:21 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-03T21:18:21.472 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:18:21 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T21:18:21.472 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:18:21 smithi067 ceph-mon[29730]: Deploying daemon mds.g on smithi067 2024-05-03T21:18:21.472 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:18:21 smithi067 ceph-mon[29730]: mds.? [v2:172.21.15.138:6838/334382443,v1:172.21.15.138:6839/334382443] up:boot 2024-05-03T21:18:21.472 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:18:21 smithi067 ceph-mon[29730]: fsmap 9 up:standby 2024-05-03T21:18:21.472 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:18:21 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-05-03T21:18:21.472 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:18:21 smithi067 ceph-mon[29730]: from='client.? 172.21.15.67:0/4028775669' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-03T21:18:21.495 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:18:21 smithi082 ceph-mon[32933]: pgmap v268: 1 pgs: 1 active+clean; 577 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-03T21:18:21.495 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:18:21 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:18:21.495 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:18:21 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:18:21.496 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:18:21 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:18:21.496 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:18:21 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-03T21:18:21.496 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:18:21 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-03T21:18:21.496 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:18:21 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T21:18:21.496 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:18:21 smithi082 ceph-mon[32933]: Deploying daemon mds.g on smithi067 2024-05-03T21:18:21.496 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:18:21 smithi082 ceph-mon[32933]: mds.? [v2:172.21.15.138:6838/334382443,v1:172.21.15.138:6839/334382443] up:boot 2024-05-03T21:18:21.496 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:18:21 smithi082 ceph-mon[32933]: fsmap 9 up:standby 2024-05-03T21:18:21.496 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:18:21 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-05-03T21:18:21.496 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:18:21 smithi082 ceph-mon[32933]: from='client.? 172.21.15.67:0/4028775669' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-03T21:18:21.574 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:18:21 smithi138 ceph-mon[33110]: pgmap v268: 1 pgs: 1 active+clean; 577 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-03T21:18:21.575 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:18:21 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:18:21.575 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:18:21 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:18:21.575 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:18:21 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:18:21.575 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:18:21 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-03T21:18:21.575 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:18:21 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-03T21:18:21.575 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:18:21 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T21:18:21.575 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:18:21 smithi138 ceph-mon[33110]: Deploying daemon mds.g on smithi067 2024-05-03T21:18:21.575 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:18:21 smithi138 ceph-mon[33110]: mds.? [v2:172.21.15.138:6838/334382443,v1:172.21.15.138:6839/334382443] up:boot 2024-05-03T21:18:21.575 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:18:21 smithi138 ceph-mon[33110]: fsmap 9 up:standby 2024-05-03T21:18:21.575 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:18:21 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-05-03T21:18:21.575 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:18:21 smithi138 ceph-mon[33110]: from='client.? 172.21.15.67:0/4028775669' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-03T21:18:22.972 INFO:journalctl@ceph.mds.g.smithi067.stdout:May 03 21:18:22 smithi067 systemd[1]: Starting Ceph mds.g for 713bc57a-0991-11ef-bc97-c7b262605968... 2024-05-03T21:18:23.270 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:18:23 smithi067 ceph-mon[29730]: pgmap v269: 1 pgs: 1 active+clean; 577 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-03T21:18:23.270 INFO:journalctl@ceph.mds.g.smithi067.stdout:May 03 21:18:23 smithi067 podman[68780]: 2024-05-03 21:18:22.91744757 +0000 UTC m=+0.018632732 image pull ee4d846202496ba6096593e8e8becc0700a5eb7ebd95536031187f5f337d4012 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:68e15ae152ef5c805c733e76da8fc1d3dd4ee669d5ba025f457e05725baa24dd 2024-05-03T21:18:23.270 INFO:journalctl@ceph.mds.g.smithi067.stdout:May 03 21:18:23 smithi067 podman[68780]: 2024-05-03 21:18:23.022689641 +0000 UTC m=+0.123874802 container create 2b1a495821545b649dd7ae1995d82a10bc496ecd57f4a6b727fd69948d14a982 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:68e15ae152ef5c805c733e76da8fc1d3dd4ee669d5ba025f457e05725baa24dd, name=ceph-713bc57a-0991-11ef-bc97-c7b262605968-mds-g, org.label-schema.license=GPLv2, RELEASE=wip-rishabh-testing-20240501.193033-9a047fe, maintainer=Guillaume Abrioux , GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240430, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image) 2024-05-03T21:18:23.270 INFO:journalctl@ceph.mds.g.smithi067.stdout:May 03 21:18:23 smithi067 podman[68780]: 2024-05-03 21:18:23.241897274 +0000 UTC m=+0.343082436 container init 2b1a495821545b649dd7ae1995d82a10bc496ecd57f4a6b727fd69948d14a982 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:68e15ae152ef5c805c733e76da8fc1d3dd4ee669d5ba025f457e05725baa24dd, name=ceph-713bc57a-0991-11ef-bc97-c7b262605968-mds-g, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, RELEASE=wip-rishabh-testing-20240501.193033-9a047fe, org.label-schema.build-date=20240430, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, io.buildah.version=1.33.5, ceph=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b) 2024-05-03T21:18:23.270 INFO:journalctl@ceph.mds.g.smithi067.stdout:May 03 21:18:23 smithi067 podman[68780]: 2024-05-03 21:18:23.24494402 +0000 UTC m=+0.346129182 container start 2b1a495821545b649dd7ae1995d82a10bc496ecd57f4a6b727fd69948d14a982 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:68e15ae152ef5c805c733e76da8fc1d3dd4ee669d5ba025f457e05725baa24dd, name=ceph-713bc57a-0991-11ef-bc97-c7b262605968-mds-g, org.label-schema.vendor=CentOS, ceph=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_CLEAN=True, RELEASE=wip-rishabh-testing-20240501.193033-9a047fe, org.label-schema.build-date=20240430, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux ) 2024-05-03T21:18:23.495 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:18:23 smithi082 ceph-mon[32933]: pgmap v269: 1 pgs: 1 active+clean; 577 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-03T21:18:23.505 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9a047fe7599a549042d06a3fb5b6813a203eabea shell --fsid 713bc57a-0991-11ef-bc97-c7b262605968 -- ceph fs dump --format=json 2024-05-03T21:18:23.527 INFO:journalctl@ceph.mds.g.smithi067.stdout:May 03 21:18:23 smithi067 ceph-mds[68799]: -- 172.21.15.67:0/445290775 <== mon.0 v2:172.21.15.67:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x562c24cefa40 con 0x562c24cc2c00 2024-05-03T21:18:23.527 INFO:journalctl@ceph.mds.g.smithi067.stdout:May 03 21:18:23 smithi067 ceph-713bc57a-0991-11ef-bc97-c7b262605968-mds-g[68795]: starting mds.g at 2024-05-03T21:18:23.527 INFO:journalctl@ceph.mds.g.smithi067.stdout:May 03 21:18:23 smithi067 bash[68780]: 2b1a495821545b649dd7ae1995d82a10bc496ecd57f4a6b727fd69948d14a982 2024-05-03T21:18:23.527 INFO:journalctl@ceph.mds.g.smithi067.stdout:May 03 21:18:23 smithi067 systemd[1]: Started Ceph mds.g for 713bc57a-0991-11ef-bc97-c7b262605968. 2024-05-03T21:18:23.574 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:18:23 smithi138 ceph-mon[33110]: pgmap v269: 1 pgs: 1 active+clean; 577 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-03T21:18:23.763 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/713bc57a-0991-11ef-bc97-c7b262605968/mon.a/config 2024-05-03T21:18:24.471 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:18:24 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:18:24.471 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:18:24 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:18:24.471 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:18:24 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:18:24.471 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:18:24 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-03T21:18:24.471 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:18:24 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-03T21:18:24.472 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:18:24 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T21:18:24.472 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:18:24 smithi067 ceph-mon[29730]: Deploying daemon mds.a on smithi067 2024-05-03T21:18:24.472 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:18:24 smithi067 ceph-mon[29730]: pgmap v270: 1 pgs: 1 active+clean; 577 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-03T21:18:24.472 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:18:24 smithi067 ceph-mon[29730]: mds.? [v2:172.21.15.67:6838/2106078262,v1:172.21.15.67:6839/2106078262] up:boot 2024-05-03T21:18:24.472 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:18:24 smithi067 ceph-mon[29730]: fsmap 10 up:standby 2024-05-03T21:18:24.472 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:18:24 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-05-03T21:18:24.745 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:18:24 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:18:24.746 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:18:24 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:18:24.746 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:18:24 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:18:24.746 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:18:24 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-03T21:18:24.746 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:18:24 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-03T21:18:24.746 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:18:24 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T21:18:24.746 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:18:24 smithi082 ceph-mon[32933]: Deploying daemon mds.a on smithi067 2024-05-03T21:18:24.746 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:18:24 smithi082 ceph-mon[32933]: pgmap v270: 1 pgs: 1 active+clean; 577 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-03T21:18:24.746 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:18:24 smithi082 ceph-mon[32933]: mds.? [v2:172.21.15.67:6838/2106078262,v1:172.21.15.67:6839/2106078262] up:boot 2024-05-03T21:18:24.746 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:18:24 smithi082 ceph-mon[32933]: fsmap 10 up:standby 2024-05-03T21:18:24.746 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:18:24 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-05-03T21:18:24.824 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:18:24 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:18:24.825 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:18:24 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:18:24.825 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:18:24 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:18:24.825 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:18:24 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-03T21:18:24.825 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:18:24 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-03T21:18:24.825 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:18:24 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T21:18:24.825 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:18:24 smithi138 ceph-mon[33110]: Deploying daemon mds.a on smithi067 2024-05-03T21:18:24.825 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:18:24 smithi138 ceph-mon[33110]: pgmap v270: 1 pgs: 1 active+clean; 577 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-03T21:18:24.825 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:18:24 smithi138 ceph-mon[33110]: mds.? [v2:172.21.15.67:6838/2106078262,v1:172.21.15.67:6839/2106078262] up:boot 2024-05-03T21:18:24.825 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:18:24 smithi138 ceph-mon[33110]: fsmap 10 up:standby 2024-05-03T21:18:24.825 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:18:24 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-05-03T21:18:26.493 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:26.489+0000 7f2c4a4de700 1 -- 172.21.15.67:0/3462648076 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2c44109520 msgr2=0x7f2c44109900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:18:26.493 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:26.489+0000 7f2c4a4de700 1 --2- 172.21.15.67:0/3462648076 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2c44109520 0x7f2c44109900 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f2c34009990 tx=0x7f2c34009ca0 comp rx=0 tx=0).stop 2024-05-03T21:18:26.493 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:26.489+0000 7f2c4a4de700 1 -- 172.21.15.67:0/3462648076 shutdown_connections 2024-05-03T21:18:26.493 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:26.489+0000 7f2c4a4de700 1 --2- 172.21.15.67:0/3462648076 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2c44109520 0x7f2c44109900 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:18:26.493 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:26.489+0000 7f2c4a4de700 1 --2- 172.21.15.67:0/3462648076 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f2c44102450 0x7f2c44110340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:18:26.493 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:26.489+0000 7f2c4a4de700 1 --2- 172.21.15.67:0/3462648076 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f2c44101a90 0x7f2c44101f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:18:26.494 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:26.489+0000 7f2c4a4de700 1 -- 172.21.15.67:0/3462648076 >> 172.21.15.67:0/3462648076 conn(0x7f2c44078a10 msgr2=0x7f2c440ffdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T21:18:26.494 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:26.489+0000 7f2c4a4de700 1 -- 172.21.15.67:0/3462648076 shutdown_connections 2024-05-03T21:18:26.494 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:26.489+0000 7f2c4a4de700 1 -- 172.21.15.67:0/3462648076 wait complete. 2024-05-03T21:18:26.494 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:26.490+0000 7f2c4a4de700 1 Processor -- start 2024-05-03T21:18:26.494 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:26.490+0000 7f2c4a4de700 1 -- start start 2024-05-03T21:18:26.494 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:26.491+0000 7f2c4a4de700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f2c44101a90 0x7f2c4419fa50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:18:26.494 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:26.491+0000 7f2c4a4de700 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f2c44102450 0x7f2c4419ff90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:18:26.494 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:26.491+0000 7f2c4a4de700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2c44109520 0x7f2c44199b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:18:26.494 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:26.491+0000 7f2c4a4de700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f2c44113940 con 0x7f2c44109520 2024-05-03T21:18:26.494 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:26.491+0000 7f2c4a4de700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f2c441137c0 con 0x7f2c44101a90 2024-05-03T21:18:26.494 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:26.491+0000 7f2c4a4de700 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f2c44113ac0 con 0x7f2c44102450 2024-05-03T21:18:26.494 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:26.491+0000 7f2c48a7b700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2c44109520 0x7f2c44199b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:18:26.494 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:26.491+0000 7f2c48a7b700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2c44109520 0x7f2c44199b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:36984/0 (socket says 172.21.15.67:36984) 2024-05-03T21:18:26.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:26.491+0000 7f2c48a7b700 1 -- 172.21.15.67:0/3966780114 learned_addr learned my addr 172.21.15.67:0/3966780114 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-03T21:18:26.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:26.491+0000 7f2c43fff700 1 --2- 172.21.15.67:0/3966780114 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f2c44101a90 0x7f2c4419fa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:18:26.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:26.491+0000 7f2c437fe700 1 --2- 172.21.15.67:0/3966780114 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f2c44102450 0x7f2c4419ff90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:18:26.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:26.491+0000 7f2c48a7b700 1 -- 172.21.15.67:0/3966780114 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f2c44102450 msgr2=0x7f2c4419ff90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:18:26.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:26.491+0000 7f2c48a7b700 1 --2- 172.21.15.67:0/3966780114 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f2c44102450 0x7f2c4419ff90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:18:26.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:26.491+0000 7f2c48a7b700 1 -- 172.21.15.67:0/3966780114 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f2c44101a90 msgr2=0x7f2c4419fa50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:18:26.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:26.491+0000 7f2c48a7b700 1 --2- 172.21.15.67:0/3966780114 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f2c44101a90 0x7f2c4419fa50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:18:26.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:26.491+0000 7f2c48a7b700 1 -- 172.21.15.67:0/3966780114 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2c4419a400 con 0x7f2c44109520 2024-05-03T21:18:26.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:26.491+0000 7f2c48a7b700 1 --2- 172.21.15.67:0/3966780114 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2c44109520 0x7f2c44199b20 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f2c3c00b970 tx=0x7f2c3c00bc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T21:18:26.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:26.492+0000 7f2c417fa700 1 -- 172.21.15.67:0/3966780114 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c3c00d860 con 0x7f2c44109520 2024-05-03T21:18:26.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:26.492+0000 7f2c4a4de700 1 -- 172.21.15.67:0/3966780114 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2c4419a6f0 con 0x7f2c44109520 2024-05-03T21:18:26.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:26.492+0000 7f2c417fa700 1 -- 172.21.15.67:0/3966780114 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(28 keys) ==== 1170+0+0 (secure 0 0 0) 0x7f2c3c00e430 con 0x7f2c44109520 2024-05-03T21:18:26.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:26.492+0000 7f2c417fa700 1 -- 172.21.15.67:0/3966780114 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c3c017640 con 0x7f2c44109520 2024-05-03T21:18:26.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:26.492+0000 7f2c4a4de700 1 -- 172.21.15.67:0/3966780114 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2c441a69d0 con 0x7f2c44109520 2024-05-03T21:18:26.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:26.493+0000 7f2c4a4de700 1 -- 172.21.15.67:0/3966780114 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2c4410ba00 con 0x7f2c44109520 2024-05-03T21:18:26.497 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:26.494+0000 7f2c417fa700 1 -- 172.21.15.67:0/3966780114 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 16) ==== 149652+0+0 (secure 0 0 0) 0x7f2c3c00e5e0 con 0x7f2c44109520 2024-05-03T21:18:26.497 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:26.495+0000 7f2c417fa700 1 --2- 172.21.15.67:0/3966780114 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f2c2c0b2980 0x7f2c2c0b4e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:18:26.498 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:26.495+0000 7f2c43fff700 1 --2- 172.21.15.67:0/3966780114 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f2c2c0b2980 0x7f2c2c0b4e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:18:26.498 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:26.495+0000 7f2c417fa700 1 -- 172.21.15.67:0/3966780114 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(75..75 src has 1..75) ==== 7127+0+0 (secure 0 0 0) 0x7f2c3c0e1ba0 con 0x7f2c44109520 2024-05-03T21:18:26.498 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:26.496+0000 7f2c43fff700 1 --2- 172.21.15.67:0/3966780114 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f2c2c0b2980 0x7f2c2c0b4e40 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f2c340098e0 tx=0x7f2c34005960 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T21:18:26.502 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:26.500+0000 7f2c417fa700 1 -- 172.21.15.67:0/3966780114 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191839 (secure 0 0 0) 0x7f2c3c013030 con 0x7f2c44109520 2024-05-03T21:18:26.794 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:26.791+0000 7f2c4a4de700 1 -- 172.21.15.67:0/3966780114 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f2c4419afe0 con 0x7f2c44109520 2024-05-03T21:18:26.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:26.793+0000 7f2c417fa700 1 -- 172.21.15.67:0/3966780114 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 11 v11) ==== 94+0+8827 (secure 0 0 0) 0x7f2c3c0a9d40 con 0x7f2c44109520 2024-05-03T21:18:26.795 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-05-03T21:18:26.795 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":11,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14541,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6835/4224826546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":4224826546},{"type":"v1","addr":"172.21.15.67:6835","nonce":4224826546}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14553,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6837/2485231621","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6836","nonce":2485231621},{"type":"v1","addr":"172.21.15.67:6837","nonce":2485231621}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14598,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6839/2106078262","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6838","nonce":2106078262},{"type":"v1","addr":"172.21.15.67:6839","nonce":2106078262}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":24376,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.138:6833/3028022602","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6832","nonce":3028022602},{"type":"v1","addr":"172.21.15.138:6833","nonce":3028022602}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24400,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.138:6835/3342864697","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6834","nonce":3342864697},{"type":"v1","addr":"172.21.15.138:6835","nonce":3342864697}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24406,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.138:6837/2203273249","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6836","nonce":2203273249},{"type":"v1","addr":"172.21.15.138:6837","nonce":2203273249}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24407,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.82:6833/1127287628","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6832","nonce":1127287628},{"type":"v1","addr":"172.21.15.82:6833","nonce":1127287628}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24418,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.138:6839/334382443","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6838","nonce":334382443},{"type":"v1","addr":"172.21.15.138:6839","nonce":334382443}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10},{"gid":24425,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.82:6835/280518761","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6834","nonce":280518761},{"type":"v1","addr":"172.21.15.82:6835","nonce":280518761}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24449,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.82:6837/1275987852","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6836","nonce":1275987852},{"type":"v1","addr":"172.21.15.82:6837","nonce":1275987852}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9}],"filesystems":[]} 2024-05-03T21:18:26.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:26.795+0000 7f2c4a4de700 1 -- 172.21.15.67:0/3966780114 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f2c2c0b2980 msgr2=0x7f2c2c0b4e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:18:26.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:26.795+0000 7f2c4a4de700 1 --2- 172.21.15.67:0/3966780114 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f2c2c0b2980 0x7f2c2c0b4e40 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f2c340098e0 tx=0x7f2c34005960 comp rx=0 tx=0).stop 2024-05-03T21:18:26.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:26.795+0000 7f2c4a4de700 1 -- 172.21.15.67:0/3966780114 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2c44109520 msgr2=0x7f2c44199b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:18:26.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:26.795+0000 7f2c4a4de700 1 --2- 172.21.15.67:0/3966780114 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2c44109520 0x7f2c44199b20 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f2c3c00b970 tx=0x7f2c3c00bc80 comp rx=0 tx=0).stop 2024-05-03T21:18:26.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:26.796+0000 7f2c4a4de700 1 -- 172.21.15.67:0/3966780114 shutdown_connections 2024-05-03T21:18:26.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:26.796+0000 7f2c4a4de700 1 --2- 172.21.15.67:0/3966780114 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f2c2c0b2980 0x7f2c2c0b4e40 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:18:26.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:26.796+0000 7f2c4a4de700 1 --2- 172.21.15.67:0/3966780114 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2c44109520 0x7f2c44199b20 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:18:26.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:26.796+0000 7f2c4a4de700 1 --2- 172.21.15.67:0/3966780114 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f2c44101a90 0x7f2c4419fa50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:18:26.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:26.796+0000 7f2c4a4de700 1 --2- 172.21.15.67:0/3966780114 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f2c44102450 0x7f2c4419ff90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:18:26.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:26.796+0000 7f2c4a4de700 1 -- 172.21.15.67:0/3966780114 >> 172.21.15.67:0/3966780114 conn(0x7f2c44078a10 msgr2=0x7f2c440fff70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T21:18:26.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:26.796+0000 7f2c4a4de700 1 -- 172.21.15.67:0/3966780114 shutdown_connections 2024-05-03T21:18:26.799 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:26.796+0000 7f2c4a4de700 1 -- 172.21.15.67:0/3966780114 wait complete. 2024-05-03T21:18:26.799 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 11 2024-05-03T21:18:27.168 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:18:26 smithi067 ceph-mon[29730]: pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-03T21:18:27.168 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:18:26 smithi067 ceph-mon[29730]: from='client.? 172.21.15.67:0/3966780114' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-03T21:18:27.245 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:18:26 smithi082 ceph-mon[32933]: pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-03T21:18:27.245 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:18:26 smithi082 ceph-mon[32933]: from='client.? 172.21.15.67:0/3966780114' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-03T21:18:27.324 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:18:26 smithi138 ceph-mon[33110]: pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-03T21:18:27.324 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:18:26 smithi138 ceph-mon[33110]: from='client.? 172.21.15.67:0/3966780114' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-03T21:18:27.471 INFO:journalctl@ceph.mds.a.smithi067.stdout:May 03 21:18:27 smithi067 podman[69251]: 2024-05-03 21:18:27.17041199 +0000 UTC m=+0.077571628 image pull ee4d846202496ba6096593e8e8becc0700a5eb7ebd95536031187f5f337d4012 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:68e15ae152ef5c805c733e76da8fc1d3dd4ee669d5ba025f457e05725baa24dd 2024-05-03T21:18:27.755 INFO:journalctl@ceph.mds.a.smithi067.stdout:May 03 21:18:27 smithi067 podman[69251]: 2024-05-03 21:18:27.617564405 +0000 UTC m=+0.524724018 container create 5d2acf9b2666fa730cb4db415372fddb085ad7d5a2421cfb05eb3319bf7a4d4e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:68e15ae152ef5c805c733e76da8fc1d3dd4ee669d5ba025f457e05725baa24dd, name=ceph-713bc57a-0991-11ef-bc97-c7b262605968-mds-a, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.build-date=20240430, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-rishabh-testing-20240501.193033-9a047fe, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=) 2024-05-03T21:18:28.221 INFO:journalctl@ceph.mds.a.smithi067.stdout:May 03 21:18:27 smithi067 podman[69251]: 2024-05-03 21:18:27.828756591 +0000 UTC m=+0.735916217 container init 5d2acf9b2666fa730cb4db415372fddb085ad7d5a2421cfb05eb3319bf7a4d4e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:68e15ae152ef5c805c733e76da8fc1d3dd4ee669d5ba025f457e05725baa24dd, name=ceph-713bc57a-0991-11ef-bc97-c7b262605968-mds-a, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240430, RELEASE=wip-rishabh-testing-20240501.193033-9a047fe, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.vendor=CentOS) 2024-05-03T21:18:28.222 INFO:journalctl@ceph.mds.a.smithi067.stdout:May 03 21:18:27 smithi067 podman[69251]: 2024-05-03 21:18:27.831588468 +0000 UTC m=+0.738748083 container start 5d2acf9b2666fa730cb4db415372fddb085ad7d5a2421cfb05eb3319bf7a4d4e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:68e15ae152ef5c805c733e76da8fc1d3dd4ee669d5ba025f457e05725baa24dd, name=ceph-713bc57a-0991-11ef-bc97-c7b262605968-mds-a, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, ceph=True, GIT_CLEAN=True, org.label-schema.build-date=20240430, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-rishabh-testing-20240501.193033-9a047fe) 2024-05-03T21:18:28.222 INFO:journalctl@ceph.mds.a.smithi067.stdout:May 03 21:18:27 smithi067 ceph-mds[69274]: -- 172.21.15.67:0/1886397850 <== mon.0 v2:172.21.15.67:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5581b0b05a40 con 0x5581b0ad9000 2024-05-03T21:18:28.222 INFO:journalctl@ceph.mds.a.smithi067.stdout:May 03 21:18:27 smithi067 ceph-713bc57a-0991-11ef-bc97-c7b262605968-mds-a[69269]: starting mds.a at 2024-05-03T21:18:28.222 INFO:journalctl@ceph.mds.a.smithi067.stdout:May 03 21:18:27 smithi067 bash[69251]: 5d2acf9b2666fa730cb4db415372fddb085ad7d5a2421cfb05eb3319bf7a4d4e 2024-05-03T21:18:28.222 INFO:journalctl@ceph.mds.a.smithi067.stdout:May 03 21:18:27 smithi067 systemd[1]: Started Ceph mds.a for 713bc57a-0991-11ef-bc97-c7b262605968. 2024-05-03T21:18:29.222 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:18:28 smithi067 ceph-mon[29730]: pgmap v272: 1 pgs: 1 active+clean; 577 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-03T21:18:29.222 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:18:28 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:18:29.222 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:18:28 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:18:29.222 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:18:28 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:18:29.222 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:18:28 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-03T21:18:29.222 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:18:28 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-03T21:18:29.222 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:18:28 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T21:18:29.222 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:18:28 smithi067 ceph-mon[29730]: Deploying daemon mds.b on smithi082 2024-05-03T21:18:29.222 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:18:28 smithi067 ceph-mon[29730]: mds.? [v2:172.21.15.67:6840/1559196215,v1:172.21.15.67:6841/1559196215] up:boot 2024-05-03T21:18:29.222 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:18:28 smithi067 ceph-mon[29730]: fsmap 11 up:standby 2024-05-03T21:18:29.223 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:18:28 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-05-03T21:18:29.245 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:18:28 smithi082 ceph-mon[32933]: pgmap v272: 1 pgs: 1 active+clean; 577 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-03T21:18:29.246 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:18:28 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:18:29.246 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:18:28 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:18:29.246 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:18:28 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:18:29.246 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:18:28 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-03T21:18:29.246 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:18:28 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-03T21:18:29.246 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:18:28 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T21:18:29.246 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:18:28 smithi082 ceph-mon[32933]: Deploying daemon mds.b on smithi082 2024-05-03T21:18:29.246 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:18:28 smithi082 ceph-mon[32933]: mds.? [v2:172.21.15.67:6840/1559196215,v1:172.21.15.67:6841/1559196215] up:boot 2024-05-03T21:18:29.246 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:18:28 smithi082 ceph-mon[32933]: fsmap 11 up:standby 2024-05-03T21:18:29.246 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:18:28 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-05-03T21:18:29.324 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:18:28 smithi138 ceph-mon[33110]: pgmap v272: 1 pgs: 1 active+clean; 577 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-03T21:18:29.324 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:18:28 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:18:29.324 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:18:28 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:18:29.325 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:18:28 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:18:29.325 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:18:28 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-03T21:18:29.325 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:18:28 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-03T21:18:29.325 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:18:28 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T21:18:29.325 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:18:28 smithi138 ceph-mon[33110]: Deploying daemon mds.b on smithi082 2024-05-03T21:18:29.325 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:18:28 smithi138 ceph-mon[33110]: mds.? [v2:172.21.15.67:6840/1559196215,v1:172.21.15.67:6841/1559196215] up:boot 2024-05-03T21:18:29.325 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:18:28 smithi138 ceph-mon[33110]: fsmap 11 up:standby 2024-05-03T21:18:29.325 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:18:28 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-05-03T21:18:29.522 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9a047fe7599a549042d06a3fb5b6813a203eabea shell --fsid 713bc57a-0991-11ef-bc97-c7b262605968 -- ceph fs dump --format=json 2024-05-03T21:18:29.788 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/713bc57a-0991-11ef-bc97-c7b262605968/mon.a/config 2024-05-03T21:18:30.246 INFO:journalctl@ceph.mds.b.smithi082.stdout:May 03 21:18:30 smithi082 systemd[1]: Starting Ceph mds.b for 713bc57a-0991-11ef-bc97-c7b262605968... 2024-05-03T21:18:30.588 INFO:journalctl@ceph.mds.b.smithi082.stdout:May 03 21:18:30 smithi082 podman[63346]: 2024-05-03 21:18:30.213696399 +0000 UTC m=+0.018156753 image pull ee4d846202496ba6096593e8e8becc0700a5eb7ebd95536031187f5f337d4012 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:68e15ae152ef5c805c733e76da8fc1d3dd4ee669d5ba025f457e05725baa24dd 2024-05-03T21:18:30.588 INFO:journalctl@ceph.mds.b.smithi082.stdout:May 03 21:18:30 smithi082 podman[63346]: 2024-05-03 21:18:30.318056032 +0000 UTC m=+0.122516373 container create 3b7d6f82f0cbea2c4ba60dfefdb9b1b21c6a1cc98c6d07c52c94432300813928 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:68e15ae152ef5c805c733e76da8fc1d3dd4ee669d5ba025f457e05725baa24dd, name=ceph-713bc57a-0991-11ef-bc97-c7b262605968-mds-b, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.build-date=20240430, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, RELEASE=wip-rishabh-testing-20240501.193033-9a047fe, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, ceph=True) 2024-05-03T21:18:30.588 INFO:journalctl@ceph.mds.b.smithi082.stdout:May 03 21:18:30 smithi082 podman[63346]: 2024-05-03 21:18:30.587296633 +0000 UTC m=+0.391756974 container init 3b7d6f82f0cbea2c4ba60dfefdb9b1b21c6a1cc98c6d07c52c94432300813928 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:68e15ae152ef5c805c733e76da8fc1d3dd4ee669d5ba025f457e05725baa24dd, name=ceph-713bc57a-0991-11ef-bc97-c7b262605968-mds-b, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240430, CEPH_POINT_RELEASE=, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, ceph=True, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, RELEASE=wip-rishabh-testing-20240501.193033-9a047fe) 2024-05-03T21:18:30.962 INFO:journalctl@ceph.mds.b.smithi082.stdout:May 03 21:18:30 smithi082 podman[63346]: 2024-05-03 21:18:30.590006813 +0000 UTC m=+0.394467155 container start 3b7d6f82f0cbea2c4ba60dfefdb9b1b21c6a1cc98c6d07c52c94432300813928 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:68e15ae152ef5c805c733e76da8fc1d3dd4ee669d5ba025f457e05725baa24dd, name=ceph-713bc57a-0991-11ef-bc97-c7b262605968-mds-b, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240430, org.label-schema.vendor=CentOS, GIT_CLEAN=True, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.schema-version=1.0, RELEASE=wip-rishabh-testing-20240501.193033-9a047fe, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.license=GPLv2) 2024-05-03T21:18:30.962 INFO:journalctl@ceph.mds.b.smithi082.stdout:May 03 21:18:30 smithi082 ceph-mds[63365]: -- 172.21.15.82:0/2511579575 <== mon.2 v2:172.21.15.82:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x559c06defa40 con 0x559c06dc2800 2024-05-03T21:18:30.962 INFO:journalctl@ceph.mds.b.smithi082.stdout:May 03 21:18:30 smithi082 ceph-713bc57a-0991-11ef-bc97-c7b262605968-mds-b[63361]: starting mds.b at 2024-05-03T21:18:30.962 INFO:journalctl@ceph.mds.b.smithi082.stdout:May 03 21:18:30 smithi082 bash[63346]: 3b7d6f82f0cbea2c4ba60dfefdb9b1b21c6a1cc98c6d07c52c94432300813928 2024-05-03T21:18:30.963 INFO:journalctl@ceph.mds.b.smithi082.stdout:May 03 21:18:30 smithi082 systemd[1]: Started Ceph mds.b for 713bc57a-0991-11ef-bc97-c7b262605968. 2024-05-03T21:18:31.161 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:18:30 smithi067 ceph-mon[29730]: pgmap v273: 1 pgs: 1 active+clean; 577 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-03T21:18:31.161 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:18:30 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:18:31.161 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:18:30 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:18:31.161 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:18:30 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:18:31.161 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:18:30 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:18:31.162 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:18:30 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:18:31.245 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:18:30 smithi082 ceph-mon[32933]: pgmap v273: 1 pgs: 1 active+clean; 577 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-03T21:18:31.245 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:18:30 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:18:31.245 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:18:30 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:18:31.245 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:18:30 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:18:31.245 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:18:30 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:18:31.246 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:18:30 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:18:31.324 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:18:30 smithi138 ceph-mon[33110]: pgmap v273: 1 pgs: 1 active+clean; 577 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-03T21:18:31.324 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:18:30 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:18:31.324 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:18:30 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:18:31.325 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:18:30 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:18:31.325 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:18:30 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:18:31.325 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:18:30 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:18:31.422 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:31.420+0000 7fc303de7700 1 -- 172.21.15.67:0/2610074754 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc2fc108400 msgr2=0x7fc2fc1087e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:18:31.423 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:31.420+0000 7fc303de7700 1 --2- 172.21.15.67:0/2610074754 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc2fc108400 0x7fc2fc1087e0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fc2f8009990 tx=0x7fc2f8009ca0 comp rx=0 tx=0).stop 2024-05-03T21:18:31.423 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:31.420+0000 7fc303de7700 1 -- 172.21.15.67:0/2610074754 shutdown_connections 2024-05-03T21:18:31.423 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:31.420+0000 7fc303de7700 1 --2- 172.21.15.67:0/2610074754 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc2fc108400 0x7fc2fc1087e0 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:18:31.423 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:31.420+0000 7fc303de7700 1 --2- 172.21.15.67:0/2610074754 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fc2fc102d50 0x7fc2fc10f240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:18:31.423 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:31.420+0000 7fc303de7700 1 --2- 172.21.15.67:0/2610074754 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fc2fc102390 0x7fc2fc102810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:18:31.423 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:31.420+0000 7fc303de7700 1 -- 172.21.15.67:0/2610074754 >> 172.21.15.67:0/2610074754 conn(0x7fc2fc079310 msgr2=0x7fc2fc079720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T21:18:31.423 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:31.420+0000 7fc303de7700 1 -- 172.21.15.67:0/2610074754 shutdown_connections 2024-05-03T21:18:31.423 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:31.420+0000 7fc303de7700 1 -- 172.21.15.67:0/2610074754 wait complete. 2024-05-03T21:18:31.425 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:31.421+0000 7fc303de7700 1 Processor -- start 2024-05-03T21:18:31.425 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:31.421+0000 7fc303de7700 1 -- start start 2024-05-03T21:18:31.425 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:31.421+0000 7fc303de7700 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fc2fc102390 0x7fc2fc19bee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:18:31.425 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:31.421+0000 7fc303de7700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc2fc102d50 0x7fc2fc19c420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:18:31.425 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:31.421+0000 7fc303de7700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fc2fc108400 0x7fc2fc1a07b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:18:31.425 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:31.421+0000 7fc303de7700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fc2fc1127e0 con 0x7fc2fc102d50 2024-05-03T21:18:31.425 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:31.421+0000 7fc303de7700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7fc2fc112660 con 0x7fc2fc108400 2024-05-03T21:18:31.425 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:31.421+0000 7fc303de7700 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7fc2fc112960 con 0x7fc2fc102390 2024-05-03T21:18:31.425 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:31.421+0000 7fc301b83700 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fc2fc102390 0x7fc2fc19bee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:18:31.425 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:31.421+0000 7fc301382700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc2fc102d50 0x7fc2fc19c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:18:31.425 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:31.421+0000 7fc301b83700 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fc2fc102390 0x7fc2fc19bee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3300/0 says I am v2:172.21.15.67:55996/0 (socket says 172.21.15.67:55996) 2024-05-03T21:18:31.426 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:31.421+0000 7fc301b83700 1 -- 172.21.15.67:0/3284630923 learned_addr learned my addr 172.21.15.67:0/3284630923 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-03T21:18:31.426 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:31.422+0000 7fc302384700 1 --2- 172.21.15.67:0/3284630923 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fc2fc108400 0x7fc2fc1a07b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:18:31.426 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:31.422+0000 7fc301382700 1 -- 172.21.15.67:0/3284630923 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fc2fc102390 msgr2=0x7fc2fc19bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:18:31.426 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:31.422+0000 7fc301382700 1 --2- 172.21.15.67:0/3284630923 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fc2fc102390 0x7fc2fc19bee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:18:31.426 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:31.422+0000 7fc301382700 1 -- 172.21.15.67:0/3284630923 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fc2fc108400 msgr2=0x7fc2fc1a07b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:18:31.426 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:31.422+0000 7fc301382700 1 --2- 172.21.15.67:0/3284630923 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fc2fc108400 0x7fc2fc1a07b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:18:31.426 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:31.422+0000 7fc301382700 1 -- 172.21.15.67:0/3284630923 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc2fc1a0ef0 con 0x7fc2fc102d50 2024-05-03T21:18:31.426 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:31.422+0000 7fc301b83700 1 --2- 172.21.15.67:0/3284630923 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fc2fc102390 0x7fc2fc19bee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-03T21:18:31.426 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:31.422+0000 7fc301382700 1 --2- 172.21.15.67:0/3284630923 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc2fc102d50 0x7fc2fc19c420 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fc2ec00e9f0 tx=0x7fc2ec00ed00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T21:18:31.426 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:31.422+0000 7fc2f2ffd700 1 -- 172.21.15.67:0/3284630923 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2ec00c9c0 con 0x7fc2fc102d50 2024-05-03T21:18:31.426 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:31.422+0000 7fc2f2ffd700 1 -- 172.21.15.67:0/3284630923 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(28 keys) ==== 1170+0+0 (secure 0 0 0) 0x7fc2ec00cb60 con 0x7fc2fc102d50 2024-05-03T21:18:31.426 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:31.422+0000 7fc303de7700 1 -- 172.21.15.67:0/3284630923 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc2fc1a11e0 con 0x7fc2fc102d50 2024-05-03T21:18:31.426 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:31.423+0000 7fc303de7700 1 -- 172.21.15.67:0/3284630923 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc2fc1a8a60 con 0x7fc2fc102d50 2024-05-03T21:18:31.427 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:31.423+0000 7fc2f2ffd700 1 -- 172.21.15.67:0/3284630923 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2ec018680 con 0x7fc2fc102d50 2024-05-03T21:18:31.427 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:31.424+0000 7fc303de7700 1 -- 172.21.15.67:0/3284630923 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc2fc06a190 con 0x7fc2fc102d50 2024-05-03T21:18:31.428 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:31.426+0000 7fc2f2ffd700 1 -- 172.21.15.67:0/3284630923 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 16) ==== 149652+0+0 (secure 0 0 0) 0x7fc2ec00cd10 con 0x7fc2fc102d50 2024-05-03T21:18:31.429 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:31.427+0000 7fc2f2ffd700 1 --2- 172.21.15.67:0/3284630923 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7fc2e80b2900 0x7fc2e80b4dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:18:31.429 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:31.427+0000 7fc301b83700 1 --2- 172.21.15.67:0/3284630923 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7fc2e80b2900 0x7fc2e80b4dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:18:31.430 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:31.427+0000 7fc2f2ffd700 1 -- 172.21.15.67:0/3284630923 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(75..75 src has 1..75) ==== 7127+0+0 (secure 0 0 0) 0x7fc2ec0e22c0 con 0x7fc2fc102d50 2024-05-03T21:18:31.430 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:31.428+0000 7fc301b83700 1 --2- 172.21.15.67:0/3284630923 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7fc2e80b2900 0x7fc2e80b4dc0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fc2f800b2e0 tx=0x7fc2f801a040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T21:18:31.432 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:31.430+0000 7fc2f2ffd700 1 -- 172.21.15.67:0/3284630923 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191839 (secure 0 0 0) 0x7fc2ec0aa480 con 0x7fc2fc102d50 2024-05-03T21:18:31.723 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:31.720+0000 7fc303de7700 1 -- 172.21.15.67:0/3284630923 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc2fc19d240 con 0x7fc2fc102d50 2024-05-03T21:18:31.723 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:31.721+0000 7fc2f2ffd700 1 -- 172.21.15.67:0/3284630923 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 13 v13) ==== 94+0+10479 (secure 0 0 0) 0x7fc2ec0a9d80 con 0x7fc2fc102d50 2024-05-03T21:18:31.723 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-05-03T21:18:31.724 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":13,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14541,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6835/4224826546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":4224826546},{"type":"v1","addr":"172.21.15.67:6835","nonce":4224826546}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14553,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6837/2485231621","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6836","nonce":2485231621},{"type":"v1","addr":"172.21.15.67:6837","nonce":2485231621}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14598,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6839/2106078262","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6838","nonce":2106078262},{"type":"v1","addr":"172.21.15.67:6839","nonce":2106078262}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14610,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6841/1559196215","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6840","nonce":1559196215},{"type":"v1","addr":"172.21.15.67:6841","nonce":1559196215}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24376,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.138:6833/3028022602","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6832","nonce":3028022602},{"type":"v1","addr":"172.21.15.138:6833","nonce":3028022602}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24400,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.138:6835/3342864697","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6834","nonce":3342864697},{"type":"v1","addr":"172.21.15.138:6835","nonce":3342864697}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24406,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.138:6837/2203273249","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6836","nonce":2203273249},{"type":"v1","addr":"172.21.15.138:6837","nonce":2203273249}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24407,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.82:6833/1127287628","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6832","nonce":1127287628},{"type":"v1","addr":"172.21.15.82:6833","nonce":1127287628}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24418,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.138:6839/334382443","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6838","nonce":334382443},{"type":"v1","addr":"172.21.15.138:6839","nonce":334382443}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10},{"gid":24425,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.82:6835/280518761","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6834","nonce":280518761},{"type":"v1","addr":"172.21.15.82:6835","nonce":280518761}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24449,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.82:6837/1275987852","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6836","nonce":1275987852},{"type":"v1","addr":"172.21.15.82:6837","nonce":1275987852}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24473,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.82:6839/3644499578","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6838","nonce":3644499578},{"type":"v1","addr":"172.21.15.82:6839","nonce":3644499578}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":13}],"filesystems":[]} 2024-05-03T21:18:31.725 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:31.723+0000 7fc303de7700 1 -- 172.21.15.67:0/3284630923 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7fc2e80b2900 msgr2=0x7fc2e80b4dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:18:31.725 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:31.723+0000 7fc303de7700 1 --2- 172.21.15.67:0/3284630923 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7fc2e80b2900 0x7fc2e80b4dc0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fc2f800b2e0 tx=0x7fc2f801a040 comp rx=0 tx=0).stop 2024-05-03T21:18:31.725 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:31.723+0000 7fc303de7700 1 -- 172.21.15.67:0/3284630923 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc2fc102d50 msgr2=0x7fc2fc19c420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:18:31.726 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:31.723+0000 7fc303de7700 1 --2- 172.21.15.67:0/3284630923 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc2fc102d50 0x7fc2fc19c420 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fc2ec00e9f0 tx=0x7fc2ec00ed00 comp rx=0 tx=0).stop 2024-05-03T21:18:31.726 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:31.723+0000 7fc303de7700 1 -- 172.21.15.67:0/3284630923 shutdown_connections 2024-05-03T21:18:31.726 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:31.723+0000 7fc303de7700 1 --2- 172.21.15.67:0/3284630923 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7fc2e80b2900 0x7fc2e80b4dc0 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:18:31.726 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:31.723+0000 7fc303de7700 1 --2- 172.21.15.67:0/3284630923 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fc2fc108400 0x7fc2fc1a07b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:18:31.726 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:31.723+0000 7fc303de7700 1 --2- 172.21.15.67:0/3284630923 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fc2fc102390 0x7fc2fc19bee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:18:31.726 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:31.723+0000 7fc303de7700 1 --2- 172.21.15.67:0/3284630923 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc2fc102d50 0x7fc2fc19c420 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:18:31.726 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:31.723+0000 7fc303de7700 1 -- 172.21.15.67:0/3284630923 >> 172.21.15.67:0/3284630923 conn(0x7fc2fc079310 msgr2=0x7fc2fc06ba40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T21:18:31.726 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:31.723+0000 7fc303de7700 1 -- 172.21.15.67:0/3284630923 shutdown_connections 2024-05-03T21:18:31.726 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:31.723+0000 7fc303de7700 1 -- 172.21.15.67:0/3284630923 wait complete. 2024-05-03T21:18:31.727 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 13 2024-05-03T21:18:31.972 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:18:31 smithi067 ceph-mon[29730]: mds.? [v2:172.21.15.82:6838/3644499578,v1:172.21.15.82:6839/3644499578] up:boot 2024-05-03T21:18:31.973 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:18:31 smithi067 ceph-mon[29730]: fsmap 12 up:standby 2024-05-03T21:18:31.973 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:18:31 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-05-03T21:18:31.973 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:18:31 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T21:18:31.973 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:18:31 smithi067 ceph-mon[29730]: from='client.? 172.21.15.67:0/3284630923' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-03T21:18:32.048 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:18:31 smithi082 ceph-mon[32933]: mds.? [v2:172.21.15.82:6838/3644499578,v1:172.21.15.82:6839/3644499578] up:boot 2024-05-03T21:18:32.048 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:18:31 smithi082 ceph-mon[32933]: fsmap 12 up:standby 2024-05-03T21:18:32.048 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:18:31 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-05-03T21:18:32.048 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:18:31 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T21:18:32.048 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:18:31 smithi082 ceph-mon[32933]: from='client.? 172.21.15.67:0/3284630923' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-03T21:18:32.074 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:18:31 smithi138 ceph-mon[33110]: mds.? [v2:172.21.15.82:6838/3644499578,v1:172.21.15.82:6839/3644499578] up:boot 2024-05-03T21:18:32.075 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:18:31 smithi138 ceph-mon[33110]: fsmap 12 up:standby 2024-05-03T21:18:32.075 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:18:31 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-05-03T21:18:32.075 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:18:31 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T21:18:32.075 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:18:31 smithi138 ceph-mon[33110]: from='client.? 172.21.15.67:0/3284630923' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-03T21:18:32.537 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9a047fe7599a549042d06a3fb5b6813a203eabea shell --fsid 713bc57a-0991-11ef-bc97-c7b262605968 -- ceph --format=json-pretty osd lspools 2024-05-03T21:18:33.128 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/713bc57a-0991-11ef-bc97-c7b262605968/mon.a/config 2024-05-03T21:18:33.246 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:18:32 smithi082 ceph-mon[32933]: pgmap v274: 1 pgs: 1 active+clean; 577 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-03T21:18:33.246 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:18:32 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:18:33.246 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:18:32 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:18:33.246 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:18:32 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:18:33.246 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:18:32 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:18:33.253 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:18:32 smithi067 ceph-mon[29730]: pgmap v274: 1 pgs: 1 active+clean; 577 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-03T21:18:33.253 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:18:32 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:18:33.253 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:18:32 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:18:33.253 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:18:32 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:18:33.253 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:18:32 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:18:33.324 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:18:32 smithi138 ceph-mon[33110]: pgmap v274: 1 pgs: 1 active+clean; 577 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-03T21:18:33.325 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:18:32 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:18:33.325 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:18:32 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:18:33.325 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:18:32 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:18:33.325 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:18:32 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:18:34.771 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:34.770+0000 7f34b0b4a700 1 -- 172.21.15.67:0/1801240874 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f34ac101b80 msgr2=0x7f34ac102000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:18:34.772 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:34.770+0000 7f34b0b4a700 1 --2- 172.21.15.67:0/1801240874 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f34ac101b80 0x7f34ac102000 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f34a0009990 tx=0x7f34a0009ca0 comp rx=0 tx=0).stop 2024-05-03T21:18:34.772 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:34.770+0000 7f34b0b4a700 1 -- 172.21.15.67:0/1801240874 shutdown_connections 2024-05-03T21:18:34.772 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:34.770+0000 7f34b0b4a700 1 --2- 172.21.15.67:0/1801240874 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f34ac102540 0x7f34ac10e9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:18:34.772 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:34.770+0000 7f34b0b4a700 1 --2- 172.21.15.67:0/1801240874 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f34ac107be0 0x7f34ac107fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:18:34.772 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:34.770+0000 7f34b0b4a700 1 --2- 172.21.15.67:0/1801240874 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f34ac101b80 0x7f34ac102000 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:18:34.772 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:34.770+0000 7f34b0b4a700 1 -- 172.21.15.67:0/1801240874 >> 172.21.15.67:0/1801240874 conn(0x7f34ac078a10 msgr2=0x7f34ac0ffdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T21:18:34.772 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:34.770+0000 7f34b0b4a700 1 -- 172.21.15.67:0/1801240874 shutdown_connections 2024-05-03T21:18:34.772 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:34.770+0000 7f34b0b4a700 1 -- 172.21.15.67:0/1801240874 wait complete. 2024-05-03T21:18:34.773 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:34.771+0000 7f34b0b4a700 1 Processor -- start 2024-05-03T21:18:34.774 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:34.771+0000 7f34b0b4a700 1 -- start start 2024-05-03T21:18:34.774 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:34.771+0000 7f34b0b4a700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f34ac101b80 0x7f34ac19e850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:18:34.774 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:34.771+0000 7f34b0b4a700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f34ac102540 0x7f34ac19ed90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:18:34.774 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:34.771+0000 7f34b0b4a700 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f34ac107be0 0x7f34ac198a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:18:34.774 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:34.771+0000 7f34b0b4a700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f34ac112870 con 0x7f34ac101b80 2024-05-03T21:18:34.774 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:34.771+0000 7f34b0b4a700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f34ac1126f0 con 0x7f34ac102540 2024-05-03T21:18:34.774 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:34.771+0000 7f34b0b4a700 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f34ac1129f0 con 0x7f34ac107be0 2024-05-03T21:18:34.774 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:34.771+0000 7f34aa59c700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f34ac101b80 0x7f34ac19e850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:18:34.774 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:34.771+0000 7f34aa59c700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f34ac101b80 0x7f34ac19e850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:37026/0 (socket says 172.21.15.67:37026) 2024-05-03T21:18:34.774 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:34.771+0000 7f34aa59c700 1 -- 172.21.15.67:0/3088989028 learned_addr learned my addr 172.21.15.67:0/3088989028 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-03T21:18:34.774 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:34.772+0000 7f34aad9d700 1 --2- 172.21.15.67:0/3088989028 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f34ac107be0 0x7f34ac198a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:18:34.775 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:34.772+0000 7f34a9d9b700 1 --2- 172.21.15.67:0/3088989028 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f34ac102540 0x7f34ac19ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:18:34.775 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:34.772+0000 7f34aa59c700 1 -- 172.21.15.67:0/3088989028 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f34ac107be0 msgr2=0x7f34ac198a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:18:34.775 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:34.772+0000 7f34aa59c700 1 --2- 172.21.15.67:0/3088989028 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f34ac107be0 0x7f34ac198a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:18:34.775 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:34.772+0000 7f34aa59c700 1 -- 172.21.15.67:0/3088989028 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f34ac102540 msgr2=0x7f34ac19ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:18:34.775 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:34.772+0000 7f34aa59c700 1 --2- 172.21.15.67:0/3088989028 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f34ac102540 0x7f34ac19ed90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:18:34.775 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:34.772+0000 7f34aa59c700 1 -- 172.21.15.67:0/3088989028 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f34ac199170 con 0x7f34ac101b80 2024-05-03T21:18:34.775 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:34.772+0000 7f34aa59c700 1 --2- 172.21.15.67:0/3088989028 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f34ac101b80 0x7f34ac19e850 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f349c00b530 tx=0x7f349c00b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T21:18:34.775 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:34.772+0000 7f349b7fe700 1 -- 172.21.15.67:0/3088989028 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f349c015020 con 0x7f34ac101b80 2024-05-03T21:18:34.776 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:34.772+0000 7f349b7fe700 1 -- 172.21.15.67:0/3088989028 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(28 keys) ==== 1170+0+0 (secure 0 0 0) 0x7f349c011d30 con 0x7f34ac101b80 2024-05-03T21:18:34.776 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:34.772+0000 7f349b7fe700 1 -- 172.21.15.67:0/3088989028 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f349c00f2e0 con 0x7f34ac101b80 2024-05-03T21:18:34.776 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:34.772+0000 7f34b0b4a700 1 -- 172.21.15.67:0/3088989028 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f34ac199460 con 0x7f34ac101b80 2024-05-03T21:18:34.776 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:34.772+0000 7f34b0b4a700 1 -- 172.21.15.67:0/3088989028 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f34ac1a5720 con 0x7f34ac101b80 2024-05-03T21:18:34.776 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:34.773+0000 7f34b0b4a700 1 -- 172.21.15.67:0/3088989028 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f34ac1031e0 con 0x7f34ac101b80 2024-05-03T21:18:34.776 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:34.775+0000 7f349b7fe700 1 -- 172.21.15.67:0/3088989028 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 16) ==== 149652+0+0 (secure 0 0 0) 0x7f349c018020 con 0x7f34ac101b80 2024-05-03T21:18:34.777 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:34.776+0000 7f349b7fe700 1 --2- 172.21.15.67:0/3088989028 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f34940b2880 0x7f34940b4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:18:34.777 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:34.776+0000 7f34a9d9b700 1 --2- 172.21.15.67:0/3088989028 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f34940b2880 0x7f34940b4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:18:34.778 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:34.777+0000 7f349b7fe700 1 -- 172.21.15.67:0/3088989028 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(75..75 src has 1..75) ==== 7127+0+0 (secure 0 0 0) 0x7f349c0e18d0 con 0x7f34ac101b80 2024-05-03T21:18:34.778 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:34.777+0000 7f34a9d9b700 1 --2- 172.21.15.67:0/3088989028 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f34940b2880 0x7f34940b4d40 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f34a000b5c0 tx=0x7f34a0005960 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T21:18:34.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:34.780+0000 7f349b7fe700 1 -- 172.21.15.67:0/3088989028 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191839 (secure 0 0 0) 0x7f349c01f020 con 0x7f34ac101b80 2024-05-03T21:18:35.006 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:35.005+0000 7f34b0b4a700 1 -- 172.21.15.67:0/3088989028 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) -- 0x7f34ac1a5950 con 0x7f34ac101b80 2024-05-03T21:18:35.007 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:35.005+0000 7f349b7fe700 1 -- 172.21.15.67:0/3088989028 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v75) ==== 84+0+65 (secure 0 0 0) 0x7f349c0a99f0 con 0x7f34ac101b80 2024-05-03T21:18:35.007 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-05-03T21:18:35.007 INFO:teuthology.orchestra.run.smithi067.stdout:[ 2024-05-03T21:18:35.007 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-05-03T21:18:35.007 INFO:teuthology.orchestra.run.smithi067.stdout: "poolnum": 1, 2024-05-03T21:18:35.007 INFO:teuthology.orchestra.run.smithi067.stdout: "poolname": ".mgr" 2024-05-03T21:18:35.007 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-05-03T21:18:35.007 INFO:teuthology.orchestra.run.smithi067.stdout:] 2024-05-03T21:18:35.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:35.008+0000 7f34b0b4a700 1 -- 172.21.15.67:0/3088989028 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f34940b2880 msgr2=0x7f34940b4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:18:35.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:35.008+0000 7f34b0b4a700 1 --2- 172.21.15.67:0/3088989028 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f34940b2880 0x7f34940b4d40 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f34a000b5c0 tx=0x7f34a0005960 comp rx=0 tx=0).stop 2024-05-03T21:18:35.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:35.008+0000 7f34b0b4a700 1 -- 172.21.15.67:0/3088989028 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f34ac101b80 msgr2=0x7f34ac19e850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:18:35.010 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:35.008+0000 7f34b0b4a700 1 --2- 172.21.15.67:0/3088989028 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f34ac101b80 0x7f34ac19e850 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f349c00b530 tx=0x7f349c00b840 comp rx=0 tx=0).stop 2024-05-03T21:18:35.010 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:35.008+0000 7f34b0b4a700 1 -- 172.21.15.67:0/3088989028 shutdown_connections 2024-05-03T21:18:35.010 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:35.008+0000 7f34b0b4a700 1 --2- 172.21.15.67:0/3088989028 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f34940b2880 0x7f34940b4d40 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:18:35.010 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:35.008+0000 7f34b0b4a700 1 --2- 172.21.15.67:0/3088989028 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f34ac101b80 0x7f34ac19e850 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:18:35.010 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:35.008+0000 7f34b0b4a700 1 --2- 172.21.15.67:0/3088989028 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f34ac107be0 0x7f34ac198a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:18:35.010 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:35.008+0000 7f34b0b4a700 1 --2- 172.21.15.67:0/3088989028 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f34ac102540 0x7f34ac19ed90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:18:35.010 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:35.008+0000 7f34b0b4a700 1 -- 172.21.15.67:0/3088989028 >> 172.21.15.67:0/3088989028 conn(0x7f34ac078a10 msgr2=0x7f34ac0ffe50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T21:18:35.010 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:35.008+0000 7f34b0b4a700 1 -- 172.21.15.67:0/3088989028 shutdown_connections 2024-05-03T21:18:35.010 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:35.008+0000 7f34b0b4a700 1 -- 172.21.15.67:0/3088989028 wait complete. 2024-05-03T21:18:35.021 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:18:34 smithi067 ceph-mon[29730]: pgmap v275: 1 pgs: 1 active+clean; 577 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-03T21:18:35.021 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:18:34 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:18:35.074 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:18:34 smithi138 ceph-mon[33110]: pgmap v275: 1 pgs: 1 active+clean; 577 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-03T21:18:35.074 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:18:34 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:18:35.245 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:18:34 smithi082 ceph-mon[32933]: pgmap v275: 1 pgs: 1 active+clean; 577 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-03T21:18:35.245 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:18:34 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:18:35.682 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-05-03T21:18:35.682 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9a047fe7599a549042d06a3fb5b6813a203eabea shell --fsid 713bc57a-0991-11ef-bc97-c7b262605968 -- ceph osd pool create cephfs_metadata --pg_num_min 64 2024-05-03T21:18:35.947 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/713bc57a-0991-11ef-bc97-c7b262605968/mon.a/config 2024-05-03T21:18:35.971 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:18:35 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:18:35.971 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:18:35 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:18:35.971 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:18:35 smithi067 ceph-mon[29730]: from='client.? 172.21.15.67:0/3088989028' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2024-05-03T21:18:36.245 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:18:35 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:18:36.245 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:18:35 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:18:36.245 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:18:35 smithi082 ceph-mon[32933]: from='client.? 172.21.15.67:0/3088989028' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2024-05-03T21:18:36.324 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:18:35 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:18:36.324 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:18:35 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:18:36.324 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:18:35 smithi138 ceph-mon[33110]: from='client.? 172.21.15.67:0/3088989028' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2024-05-03T21:18:37.071 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:18:36 smithi067 ceph-mon[29730]: pgmap v276: 1 pgs: 1 active+clean; 577 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-03T21:18:37.071 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:18:36 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T21:18:37.071 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:18:36 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T21:18:37.071 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:18:36 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:18:37.071 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:18:36 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T21:18:37.245 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:18:36 smithi082 ceph-mon[32933]: pgmap v276: 1 pgs: 1 active+clean; 577 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-03T21:18:37.246 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:18:36 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T21:18:37.246 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:18:36 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T21:18:37.246 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:18:36 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:18:37.246 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:18:36 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T21:18:37.324 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:18:36 smithi138 ceph-mon[33110]: pgmap v276: 1 pgs: 1 active+clean; 577 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-03T21:18:37.324 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:18:36 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T21:18:37.324 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:18:36 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T21:18:37.324 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:18:36 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:18:37.325 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:18:36 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T21:18:37.721 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:37.719+0000 7f74d27a6700 1 -- 172.21.15.67:0/1130824183 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f74cc108410 msgr2=0x7f74cc1087f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:18:37.722 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:37.719+0000 7f74d27a6700 1 --2- 172.21.15.67:0/1130824183 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f74cc108410 0x7f74cc1087f0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f74bc009990 tx=0x7f74bc009ca0 comp rx=0 tx=0).stop 2024-05-03T21:18:37.722 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:37.719+0000 7f74d27a6700 1 -- 172.21.15.67:0/1130824183 shutdown_connections 2024-05-03T21:18:37.722 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:37.719+0000 7f74d27a6700 1 --2- 172.21.15.67:0/1130824183 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f74cc108410 0x7f74cc1087f0 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:18:37.722 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:37.719+0000 7f74d27a6700 1 --2- 172.21.15.67:0/1130824183 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f74cc102d60 0x7f74cc10f250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:18:37.722 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:37.719+0000 7f74d27a6700 1 --2- 172.21.15.67:0/1130824183 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f74cc1023a0 0x7f74cc102820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:18:37.722 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:37.719+0000 7f74d27a6700 1 -- 172.21.15.67:0/1130824183 >> 172.21.15.67:0/1130824183 conn(0x7f74cc079350 msgr2=0x7f74cc079760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T21:18:37.722 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:37.719+0000 7f74d27a6700 1 -- 172.21.15.67:0/1130824183 shutdown_connections 2024-05-03T21:18:37.722 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:37.719+0000 7f74d27a6700 1 -- 172.21.15.67:0/1130824183 wait complete. 2024-05-03T21:18:37.722 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:37.720+0000 7f74d27a6700 1 Processor -- start 2024-05-03T21:18:37.722 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:37.720+0000 7f74d27a6700 1 -- start start 2024-05-03T21:18:37.724 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:37.720+0000 7f74d27a6700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f74cc1023a0 0x7f74cc19e860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:18:37.724 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:37.720+0000 7f74d27a6700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f74cc102d60 0x7f74cc19eda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:18:37.724 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:37.720+0000 7f74d27a6700 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f74cc108410 0x7f74cc198a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:18:37.724 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:37.720+0000 7f74d27a6700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f74cc1127c0 con 0x7f74cc102d60 2024-05-03T21:18:37.724 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:37.720+0000 7f74d27a6700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f74cc112640 con 0x7f74cc1023a0 2024-05-03T21:18:37.725 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:37.720+0000 7f74d27a6700 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f74cc112940 con 0x7f74cc108410 2024-05-03T21:18:37.725 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:37.720+0000 7f74cb7fe700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f74cc102d60 0x7f74cc19eda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:18:37.725 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:37.720+0000 7f74cb7fe700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f74cc102d60 0x7f74cc19eda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:44004/0 (socket says 172.21.15.67:44004) 2024-05-03T21:18:37.725 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:37.720+0000 7f74cb7fe700 1 -- 172.21.15.67:0/1878692247 learned_addr learned my addr 172.21.15.67:0/1878692247 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-03T21:18:37.725 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:37.720+0000 7f74cbfff700 1 --2- 172.21.15.67:0/1878692247 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f74cc1023a0 0x7f74cc19e860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:18:37.725 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:37.720+0000 7f74d0d43700 1 --2- 172.21.15.67:0/1878692247 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f74cc108410 0x7f74cc198a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:18:37.725 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:37.720+0000 7f74cbfff700 1 -- 172.21.15.67:0/1878692247 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f74cc108410 msgr2=0x7f74cc198a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:18:37.725 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:37.720+0000 7f74cbfff700 1 --2- 172.21.15.67:0/1878692247 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f74cc108410 0x7f74cc198a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:18:37.725 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:37.720+0000 7f74cbfff700 1 -- 172.21.15.67:0/1878692247 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f74cc102d60 msgr2=0x7f74cc19eda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:18:37.725 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:37.720+0000 7f74cbfff700 1 --2- 172.21.15.67:0/1878692247 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f74cc102d60 0x7f74cc19eda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:18:37.725 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:37.720+0000 7f74cbfff700 1 -- 172.21.15.67:0/1878692247 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f74cc199180 con 0x7f74cc1023a0 2024-05-03T21:18:37.725 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:37.720+0000 7f74cb7fe700 1 --2- 172.21.15.67:0/1878692247 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f74cc102d60 0x7f74cc19eda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-03T21:18:37.725 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:37.721+0000 7f74cbfff700 1 --2- 172.21.15.67:0/1878692247 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f74cc1023a0 0x7f74cc19e860 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f74bc006010 tx=0x7f74bc0057c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T21:18:37.726 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:37.721+0000 7f74c97fa700 1 -- 172.21.15.67:0/1878692247 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74bc027070 con 0x7f74cc1023a0 2024-05-03T21:18:37.726 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:37.721+0000 7f74d27a6700 1 -- 172.21.15.67:0/1878692247 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f74cc199410 con 0x7f74cc1023a0 2024-05-03T21:18:37.726 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:37.721+0000 7f74c97fa700 1 -- 172.21.15.67:0/1878692247 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(28 keys) ==== 1170+0+0 (secure 0 0 0) 0x7f74bc022470 con 0x7f74cc1023a0 2024-05-03T21:18:37.726 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:37.721+0000 7f74c97fa700 1 -- 172.21.15.67:0/1878692247 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74bc017650 con 0x7f74cc1023a0 2024-05-03T21:18:37.726 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:37.721+0000 7f74d27a6700 1 -- 172.21.15.67:0/1878692247 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f74cc06b180 con 0x7f74cc1023a0 2024-05-03T21:18:37.726 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:37.722+0000 7f74d27a6700 1 -- 172.21.15.67:0/1878692247 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f74cc06a190 con 0x7f74cc1023a0 2024-05-03T21:18:37.727 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:37.723+0000 7f74c97fa700 1 -- 172.21.15.67:0/1878692247 <== mon.2 v2:172.21.15.82:3300/0 4 ==== mgrmap(e 16) ==== 149652+0+0 (secure 0 0 0) 0x7f74bc02a070 con 0x7f74cc1023a0 2024-05-03T21:18:37.727 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:37.724+0000 7f74c97fa700 1 --2- 172.21.15.67:0/1878692247 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f74b40b2880 0x7f74b40b4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:18:37.727 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:37.724+0000 7f74cb7fe700 1 --2- 172.21.15.67:0/1878692247 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f74b40b2880 0x7f74b40b4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:18:37.727 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:37.724+0000 7f74c97fa700 1 -- 172.21.15.67:0/1878692247 <== mon.2 v2:172.21.15.82:3300/0 5 ==== osd_map(75..75 src has 1..75) ==== 7127+0+0 (secure 0 0 0) 0x7f74bc0e2ba0 con 0x7f74cc1023a0 2024-05-03T21:18:37.727 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:37.724+0000 7f74cb7fe700 1 --2- 172.21.15.67:0/1878692247 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f74b40b2880 0x7f74b40b4d40 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f74cc079950 tx=0x7f74c0007480 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T21:18:37.730 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:37.728+0000 7f74c97fa700 1 -- 172.21.15.67:0/1878692247 <== mon.2 v2:172.21.15.82:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191839 (secure 0 0 0) 0x7f74bc013070 con 0x7f74cc1023a0 2024-05-03T21:18:37.951 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:37.948+0000 7f74d27a6700 1 -- 172.21.15.67:0/1878692247 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) -- 0x7f74cc199f50 con 0x7f74cc1023a0 2024-05-03T21:18:38.782 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:38.779+0000 7f74c97fa700 1 -- 172.21.15.67:0/1878692247 <== mon.2 v2:172.21.15.82:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v76) ==== 138+0+0 (secure 0 0 0) 0x7f74bc0aad50 con 0x7f74cc1023a0 2024-05-03T21:18:38.787 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:38.783+0000 7f74d27a6700 1 -- 172.21.15.67:0/1878692247 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f74b40b2880 msgr2=0x7f74b40b4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:18:38.787 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:38.783+0000 7f74d27a6700 1 --2- 172.21.15.67:0/1878692247 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f74b40b2880 0x7f74b40b4d40 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f74cc079950 tx=0x7f74c0007480 comp rx=0 tx=0).stop 2024-05-03T21:18:38.787 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:38.784+0000 7f74d27a6700 1 -- 172.21.15.67:0/1878692247 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f74cc1023a0 msgr2=0x7f74cc19e860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:18:38.787 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:38.784+0000 7f74d27a6700 1 --2- 172.21.15.67:0/1878692247 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f74cc1023a0 0x7f74cc19e860 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f74bc006010 tx=0x7f74bc0057c0 comp rx=0 tx=0).stop 2024-05-03T21:18:38.787 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:38.784+0000 7f74d27a6700 1 -- 172.21.15.67:0/1878692247 shutdown_connections 2024-05-03T21:18:38.787 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:38.784+0000 7f74d27a6700 1 --2- 172.21.15.67:0/1878692247 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f74b40b2880 0x7f74b40b4d40 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:18:38.787 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:38.784+0000 7f74d27a6700 1 --2- 172.21.15.67:0/1878692247 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f74cc108410 0x7f74cc198a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:18:38.787 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:38.784+0000 7f74d27a6700 1 --2- 172.21.15.67:0/1878692247 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f74cc1023a0 0x7f74cc19e860 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:18:38.787 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:38.784+0000 7f74d27a6700 1 --2- 172.21.15.67:0/1878692247 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f74cc102d60 0x7f74cc19eda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:18:38.787 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:38.784+0000 7f74d27a6700 1 -- 172.21.15.67:0/1878692247 >> 172.21.15.67:0/1878692247 conn(0x7f74cc079350 msgr2=0x7f74cc101020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T21:18:38.787 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:38.784+0000 7f74d27a6700 1 -- 172.21.15.67:0/1878692247 shutdown_connections 2024-05-03T21:18:38.787 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:38.784+0000 7f74d27a6700 1 -- 172.21.15.67:0/1878692247 wait complete. 2024-05-03T21:18:38.788 INFO:teuthology.orchestra.run.smithi067.stderr:pool 'cephfs_metadata' created 2024-05-03T21:18:38.971 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:18:38 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:18:38.971 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:18:38 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:18:38.971 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:18:38 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:18:38.971 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:18:38 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:18:38.971 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:18:38 smithi067 ceph-mon[29730]: pgmap v277: 1 pgs: 1 active+clean; 577 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-03T21:18:38.971 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:18:38 smithi067 ceph-mon[29730]: from='client.? 172.21.15.67:0/1878692247' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-05-03T21:18:38.971 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:18:38 smithi067 ceph-mon[29730]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-05-03T21:18:38.995 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:18:38 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:18:38.995 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:18:38 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:18:38.996 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:18:38 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:18:38.996 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:18:38 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:18:38.996 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:18:38 smithi082 ceph-mon[32933]: pgmap v277: 1 pgs: 1 active+clean; 577 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-03T21:18:38.996 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:18:38 smithi082 ceph-mon[32933]: from='client.? 172.21.15.67:0/1878692247' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-05-03T21:18:38.996 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:18:38 smithi082 ceph-mon[32933]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-05-03T21:18:39.074 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:18:38 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:18:39.075 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:18:38 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:18:39.075 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:18:38 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:18:39.075 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:18:38 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:18:39.075 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:18:38 smithi138 ceph-mon[33110]: pgmap v277: 1 pgs: 1 active+clean; 577 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-03T21:18:39.075 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:18:38 smithi138 ceph-mon[33110]: from='client.? 172.21.15.67:0/1878692247' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-05-03T21:18:39.075 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:18:38 smithi138 ceph-mon[33110]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-05-03T21:18:39.408 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9a047fe7599a549042d06a3fb5b6813a203eabea shell --fsid 713bc57a-0991-11ef-bc97-c7b262605968 -- ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-05-03T21:18:39.671 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/713bc57a-0991-11ef-bc97-c7b262605968/mon.a/config 2024-05-03T21:18:40.075 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:18:39 smithi138 ceph-mon[33110]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2024-05-03T21:18:40.075 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:18:39 smithi138 ceph-mon[33110]: osdmap e76: 12 total, 12 up, 12 in 2024-05-03T21:18:40.075 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:18:39 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:18:40.075 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:18:39 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:18:40.075 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:18:39 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T21:18:40.075 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:18:39 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T21:18:40.075 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:18:39 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:18:40.078 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:18:39 smithi067 ceph-mon[29730]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2024-05-03T21:18:40.079 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:18:39 smithi067 ceph-mon[29730]: osdmap e76: 12 total, 12 up, 12 in 2024-05-03T21:18:40.079 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:18:39 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:18:40.079 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:18:39 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:18:40.079 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:18:39 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T21:18:40.079 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:18:39 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T21:18:40.079 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:18:39 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:18:40.245 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:18:39 smithi082 ceph-mon[32933]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2024-05-03T21:18:40.245 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:18:39 smithi082 ceph-mon[32933]: osdmap e76: 12 total, 12 up, 12 in 2024-05-03T21:18:40.245 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:18:39 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:18:40.246 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:18:39 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:18:40.246 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:18:39 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T21:18:40.246 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:18:39 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T21:18:40.246 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:18:39 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:18:40.806 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:18:40 smithi067 ceph-mon[29730]: osdmap e77: 12 total, 12 up, 12 in 2024-05-03T21:18:40.806 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:18:40 smithi067 ceph-mon[29730]: pgmap v280: 33 pgs: 32 unknown, 1 active+clean; 577 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-03T21:18:41.074 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:18:40 smithi138 ceph-mon[33110]: osdmap e77: 12 total, 12 up, 12 in 2024-05-03T21:18:41.074 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:18:40 smithi138 ceph-mon[33110]: pgmap v280: 33 pgs: 32 unknown, 1 active+clean; 577 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-03T21:18:41.245 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:18:40 smithi082 ceph-mon[32933]: osdmap e77: 12 total, 12 up, 12 in 2024-05-03T21:18:41.246 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:18:40 smithi082 ceph-mon[32933]: pgmap v280: 33 pgs: 32 unknown, 1 active+clean; 577 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-03T21:18:41.347 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:41.343+0000 7fa51f0a6700 1 -- 172.21.15.67:0/4024882980 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa518108850 msgr2=0x7fa5181103d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:18:41.347 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:41.343+0000 7fa51f0a6700 1 --2- 172.21.15.67:0/4024882980 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa518108850 0x7fa5181103d0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fa51000b0d0 tx=0x7fa51000b3e0 comp rx=0 tx=0).stop 2024-05-03T21:18:41.347 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:41.343+0000 7fa51f0a6700 1 -- 172.21.15.67:0/4024882980 shutdown_connections 2024-05-03T21:18:41.347 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:41.343+0000 7fa51f0a6700 1 --2- 172.21.15.67:0/4024882980 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa518108850 0x7fa5181103d0 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:18:41.347 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:41.343+0000 7fa51f0a6700 1 --2- 172.21.15.67:0/4024882980 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fa518103750 0x7fa518103b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:18:41.347 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:41.343+0000 7fa51f0a6700 1 --2- 172.21.15.67:0/4024882980 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fa518104100 0x7fa518108120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:18:41.347 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:41.343+0000 7fa51f0a6700 1 -- 172.21.15.67:0/4024882980 >> 172.21.15.67:0/4024882980 conn(0x7fa51807a600 msgr2=0x7fa51807aa10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T21:18:41.347 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:41.343+0000 7fa51f0a6700 1 -- 172.21.15.67:0/4024882980 shutdown_connections 2024-05-03T21:18:41.348 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:41.343+0000 7fa51f0a6700 1 -- 172.21.15.67:0/4024882980 wait complete. 2024-05-03T21:18:41.348 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:41.343+0000 7fa51f0a6700 1 Processor -- start 2024-05-03T21:18:41.348 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:41.343+0000 7fa51f0a6700 1 -- start start 2024-05-03T21:18:41.348 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:41.344+0000 7fa51f0a6700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fa518103750 0x7fa5180757a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:18:41.348 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:41.344+0000 7fa51f0a6700 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fa518104100 0x7fa518075ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:18:41.348 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:41.344+0000 7fa51f0a6700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa518108850 0x7fa518076220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:18:41.348 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:41.344+0000 7fa51f0a6700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fa5181138b0 con 0x7fa518108850 2024-05-03T21:18:41.348 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:41.344+0000 7fa51f0a6700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7fa518113730 con 0x7fa518103750 2024-05-03T21:18:41.348 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:41.344+0000 7fa51f0a6700 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7fa518113a30 con 0x7fa518104100 2024-05-03T21:18:41.348 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:41.344+0000 7fa51d643700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa518108850 0x7fa518076220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:18:41.348 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:41.344+0000 7fa51d643700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa518108850 0x7fa518076220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:44018/0 (socket says 172.21.15.67:44018) 2024-05-03T21:18:41.348 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:41.344+0000 7fa51d643700 1 -- 172.21.15.67:0/66121442 learned_addr learned my addr 172.21.15.67:0/66121442 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-03T21:18:41.348 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:41.344+0000 7fa51ce42700 1 --2- 172.21.15.67:0/66121442 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fa518103750 0x7fa5180757a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:18:41.349 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:41.344+0000 7fa517fff700 1 --2- 172.21.15.67:0/66121442 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fa518104100 0x7fa518075ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:18:41.349 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:41.344+0000 7fa51d643700 1 -- 172.21.15.67:0/66121442 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fa518104100 msgr2=0x7fa518075ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:18:41.349 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:41.344+0000 7fa51d643700 1 --2- 172.21.15.67:0/66121442 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fa518104100 0x7fa518075ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:18:41.349 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:41.344+0000 7fa51d643700 1 -- 172.21.15.67:0/66121442 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fa518103750 msgr2=0x7fa5180757a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:18:41.349 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:41.344+0000 7fa51d643700 1 --2- 172.21.15.67:0/66121442 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fa518103750 0x7fa5180757a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:18:41.349 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:41.344+0000 7fa51d643700 1 -- 172.21.15.67:0/66121442 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa5181a80a0 con 0x7fa518108850 2024-05-03T21:18:41.349 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:41.344+0000 7fa517fff700 1 --2- 172.21.15.67:0/66121442 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fa518104100 0x7fa518075ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-03T21:18:41.349 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:41.344+0000 7fa51d643700 1 --2- 172.21.15.67:0/66121442 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa518108850 0x7fa518076220 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fa510007a20 tx=0x7fa510007a50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T21:18:41.349 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:41.345+0000 7fa515ffb700 1 -- 172.21.15.67:0/66121442 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa51001f070 con 0x7fa518108850 2024-05-03T21:18:41.349 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:41.345+0000 7fa515ffb700 1 -- 172.21.15.67:0/66121442 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(28 keys) ==== 1170+0+0 (secure 0 0 0) 0x7fa510003c00 con 0x7fa518108850 2024-05-03T21:18:41.349 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:41.345+0000 7fa51f0a6700 1 -- 172.21.15.67:0/66121442 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa5181a8330 con 0x7fa518108850 2024-05-03T21:18:41.349 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:41.345+0000 7fa515ffb700 1 -- 172.21.15.67:0/66121442 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa51001b8a0 con 0x7fa518108850 2024-05-03T21:18:41.349 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:41.345+0000 7fa51f0a6700 1 -- 172.21.15.67:0/66121442 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa5181a8780 con 0x7fa518108850 2024-05-03T21:18:41.351 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:41.346+0000 7fa51f0a6700 1 -- 172.21.15.67:0/66121442 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa51806a190 con 0x7fa518108850 2024-05-03T21:18:41.351 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:41.347+0000 7fa515ffb700 1 -- 172.21.15.67:0/66121442 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 16) ==== 149652+0+0 (secure 0 0 0) 0x7fa510022070 con 0x7fa518108850 2024-05-03T21:18:41.351 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:41.348+0000 7fa515ffb700 1 --2- 172.21.15.67:0/66121442 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7fa5080b2880 0x7fa5080b4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:18:41.351 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:41.348+0000 7fa51ce42700 1 --2- 172.21.15.67:0/66121442 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7fa5080b2880 0x7fa5080b4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:18:41.351 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:41.348+0000 7fa515ffb700 1 -- 172.21.15.67:0/66121442 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 7497+0+0 (secure 0 0 0) 0x7fa51000f020 con 0x7fa518108850 2024-05-03T21:18:41.351 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:41.349+0000 7fa51ce42700 1 --2- 172.21.15.67:0/66121442 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7fa5080b2880 0x7fa5080b4d40 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fa50c00ac50 tx=0x7fa50c00a380 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T21:18:41.354 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:41.352+0000 7fa515ffb700 1 -- 172.21.15.67:0/66121442 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191839 (secure 0 0 0) 0x7fa510029020 con 0x7fa518108850 2024-05-03T21:18:41.569 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:41.566+0000 7fa51f0a6700 1 -- 172.21.15.67:0/66121442 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) -- 0x7fa51802e510 con 0x7fa518108850 2024-05-03T21:18:41.792 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:41.789+0000 7fa515ffb700 1 -- 172.21.15.67:0/66121442 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v79) ==== 170+0+0 (secure 0 0 0) 0x7fa5100ab650 con 0x7fa518108850 2024-05-03T21:18:41.794 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:41.792+0000 7fa51f0a6700 1 -- 172.21.15.67:0/66121442 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7fa5080b2880 msgr2=0x7fa5080b4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:18:41.794 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:41.792+0000 7fa51f0a6700 1 --2- 172.21.15.67:0/66121442 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7fa5080b2880 0x7fa5080b4d40 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fa50c00ac50 tx=0x7fa50c00a380 comp rx=0 tx=0).stop 2024-05-03T21:18:41.794 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:41.792+0000 7fa51f0a6700 1 -- 172.21.15.67:0/66121442 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa518108850 msgr2=0x7fa518076220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:18:41.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:41.792+0000 7fa51f0a6700 1 --2- 172.21.15.67:0/66121442 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa518108850 0x7fa518076220 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fa510007a20 tx=0x7fa510007a50 comp rx=0 tx=0).stop 2024-05-03T21:18:41.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:41.793+0000 7fa51f0a6700 1 -- 172.21.15.67:0/66121442 shutdown_connections 2024-05-03T21:18:41.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:41.793+0000 7fa51f0a6700 1 --2- 172.21.15.67:0/66121442 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7fa5080b2880 0x7fa5080b4d40 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:18:41.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:41.793+0000 7fa51f0a6700 1 --2- 172.21.15.67:0/66121442 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa518108850 0x7fa518076220 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:18:41.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:41.793+0000 7fa51f0a6700 1 --2- 172.21.15.67:0/66121442 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fa518103750 0x7fa5180757a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:18:41.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:41.793+0000 7fa51f0a6700 1 --2- 172.21.15.67:0/66121442 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fa518104100 0x7fa518075ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:18:41.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:41.793+0000 7fa51f0a6700 1 -- 172.21.15.67:0/66121442 >> 172.21.15.67:0/66121442 conn(0x7fa51807a600 msgr2=0x7fa518102480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T21:18:41.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:41.793+0000 7fa51f0a6700 1 -- 172.21.15.67:0/66121442 shutdown_connections 2024-05-03T21:18:41.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:41.794+0000 7fa51f0a6700 1 -- 172.21.15.67:0/66121442 wait complete. 2024-05-03T21:18:41.799 INFO:teuthology.orchestra.run.smithi067.stderr:pool 'cephfs_data' created 2024-05-03T21:18:42.074 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:18:41 smithi138 ceph-mon[33110]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-05-03T21:18:42.074 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:18:41 smithi138 ceph-mon[33110]: osdmap e78: 12 total, 12 up, 12 in 2024-05-03T21:18:42.075 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:18:41 smithi138 ceph-mon[33110]: from='client.? 172.21.15.67:0/66121442' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-05-03T21:18:42.101 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:18:41 smithi067 ceph-mon[29730]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-05-03T21:18:42.102 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:18:41 smithi067 ceph-mon[29730]: osdmap e78: 12 total, 12 up, 12 in 2024-05-03T21:18:42.102 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:18:41 smithi067 ceph-mon[29730]: from='client.? 172.21.15.67:0/66121442' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-05-03T21:18:42.245 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:18:41 smithi082 ceph-mon[32933]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-05-03T21:18:42.245 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:18:41 smithi082 ceph-mon[32933]: osdmap e78: 12 total, 12 up, 12 in 2024-05-03T21:18:42.245 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:18:41 smithi082 ceph-mon[32933]: from='client.? 172.21.15.67:0/66121442' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-05-03T21:18:42.402 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9a047fe7599a549042d06a3fb5b6813a203eabea shell --fsid 713bc57a-0991-11ef-bc97-c7b262605968 -- ceph fs new cephfs cephfs_metadata cephfs_data 2024-05-03T21:18:42.664 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/713bc57a-0991-11ef-bc97-c7b262605968/mon.a/config 2024-05-03T21:18:42.962 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:18:42 smithi067 ceph-mon[29730]: from='client.? 172.21.15.67:0/66121442' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2024-05-03T21:18:42.962 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:18:42 smithi067 ceph-mon[29730]: osdmap e79: 12 total, 12 up, 12 in 2024-05-03T21:18:42.962 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:18:42 smithi067 ceph-mon[29730]: pgmap v283: 97 pgs: 17 active+clean, 80 unknown; 577 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-03T21:18:43.074 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:18:42 smithi138 ceph-mon[33110]: from='client.? 172.21.15.67:0/66121442' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2024-05-03T21:18:43.074 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:18:42 smithi138 ceph-mon[33110]: osdmap e79: 12 total, 12 up, 12 in 2024-05-03T21:18:43.074 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:18:42 smithi138 ceph-mon[33110]: pgmap v283: 97 pgs: 17 active+clean, 80 unknown; 577 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-03T21:18:43.245 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:18:42 smithi082 ceph-mon[32933]: from='client.? 172.21.15.67:0/66121442' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2024-05-03T21:18:43.246 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:18:42 smithi082 ceph-mon[32933]: osdmap e79: 12 total, 12 up, 12 in 2024-05-03T21:18:43.246 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:18:42 smithi082 ceph-mon[32933]: pgmap v283: 97 pgs: 17 active+clean, 80 unknown; 577 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-03T21:18:43.971 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:18:43 smithi067 ceph-mon[29730]: osdmap e80: 12 total, 12 up, 12 in 2024-05-03T21:18:43.971 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:18:43 smithi067 ceph-mon[29730]: osdmap e81: 12 total, 12 up, 12 in 2024-05-03T21:18:44.074 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:18:43 smithi138 ceph-mon[33110]: osdmap e80: 12 total, 12 up, 12 in 2024-05-03T21:18:44.074 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:18:43 smithi138 ceph-mon[33110]: osdmap e81: 12 total, 12 up, 12 in 2024-05-03T21:18:44.245 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:18:43 smithi082 ceph-mon[32933]: osdmap e80: 12 total, 12 up, 12 in 2024-05-03T21:18:44.245 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:18:43 smithi082 ceph-mon[32933]: osdmap e81: 12 total, 12 up, 12 in 2024-05-03T21:18:44.396 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:44.394+0000 7f2a22e75700 1 -- 172.21.15.67:0/360022522 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2a1c06bc70 msgr2=0x7f2a1c06c0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:18:44.396 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:44.394+0000 7f2a22e75700 1 --2- 172.21.15.67:0/360022522 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2a1c06bc70 0x7f2a1c06c0f0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f2a100099c0 tx=0x7f2a10009cd0 comp rx=0 tx=0).stop 2024-05-03T21:18:44.396 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:44.394+0000 7f2a22e75700 1 -- 172.21.15.67:0/360022522 shutdown_connections 2024-05-03T21:18:44.396 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:44.394+0000 7f2a22e75700 1 --2- 172.21.15.67:0/360022522 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f2a1c06c630 0x7f2a1c110360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:18:44.397 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:44.394+0000 7f2a22e75700 1 --2- 172.21.15.67:0/360022522 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f2a1c109590 0x7f2a1c109970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:18:44.397 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:44.394+0000 7f2a22e75700 1 --2- 172.21.15.67:0/360022522 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2a1c06bc70 0x7f2a1c06c0f0 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:18:44.397 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:44.394+0000 7f2a22e75700 1 -- 172.21.15.67:0/360022522 >> 172.21.15.67:0/360022522 conn(0x7f2a1c079440 msgr2=0x7f2a1c079850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T21:18:44.397 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:44.394+0000 7f2a22e75700 1 -- 172.21.15.67:0/360022522 shutdown_connections 2024-05-03T21:18:44.397 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:44.394+0000 7f2a22e75700 1 -- 172.21.15.67:0/360022522 wait complete. 2024-05-03T21:18:44.399 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:44.394+0000 7f2a22e75700 1 Processor -- start 2024-05-03T21:18:44.399 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:44.395+0000 7f2a22e75700 1 -- start start 2024-05-03T21:18:44.399 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:44.395+0000 7f2a22e75700 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f2a1c06bc70 0x7f2a1c19f900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:18:44.399 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:44.395+0000 7f2a22e75700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2a1c06c630 0x7f2a1c19fe40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:18:44.399 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:44.395+0000 7f2a22e75700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f2a1c109590 0x7f2a1c199b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:18:44.399 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:44.395+0000 7f2a22e75700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f2a1c113930 con 0x7f2a1c06c630 2024-05-03T21:18:44.400 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:44.395+0000 7f2a22e75700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f2a1c1137b0 con 0x7f2a1c109590 2024-05-03T21:18:44.400 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:44.395+0000 7f2a22e75700 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f2a1c113ab0 con 0x7f2a1c06bc70 2024-05-03T21:18:44.400 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:44.395+0000 7f2a1bfff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2a1c06c630 0x7f2a1c19fe40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:18:44.400 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:44.395+0000 7f2a1bfff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2a1c06c630 0x7f2a1c19fe40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:44028/0 (socket says 172.21.15.67:44028) 2024-05-03T21:18:44.400 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:44.395+0000 7f2a1bfff700 1 -- 172.21.15.67:0/3149204191 learned_addr learned my addr 172.21.15.67:0/3149204191 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-03T21:18:44.400 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:44.395+0000 7f2a20c11700 1 --2- 172.21.15.67:0/3149204191 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f2a1c06bc70 0x7f2a1c19f900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:18:44.400 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:44.395+0000 7f2a21412700 1 --2- 172.21.15.67:0/3149204191 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f2a1c109590 0x7f2a1c199b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:18:44.400 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:44.395+0000 7f2a1bfff700 1 -- 172.21.15.67:0/3149204191 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f2a1c06bc70 msgr2=0x7f2a1c19f900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:18:44.400 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:44.395+0000 7f2a1bfff700 1 --2- 172.21.15.67:0/3149204191 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f2a1c06bc70 0x7f2a1c19f900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:18:44.400 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:44.395+0000 7f2a1bfff700 1 -- 172.21.15.67:0/3149204191 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f2a1c109590 msgr2=0x7f2a1c199b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:18:44.400 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:44.395+0000 7f2a1bfff700 1 --2- 172.21.15.67:0/3149204191 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f2a1c109590 0x7f2a1c199b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:18:44.400 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:44.395+0000 7f2a1bfff700 1 -- 172.21.15.67:0/3149204191 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2a1c19a270 con 0x7f2a1c06c630 2024-05-03T21:18:44.400 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:44.395+0000 7f2a1bfff700 1 --2- 172.21.15.67:0/3149204191 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2a1c06c630 0x7f2a1c19fe40 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f2a100059d0 tx=0x7f2a1000bb00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T21:18:44.401 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:44.396+0000 7f2a19ffb700 1 -- 172.21.15.67:0/3149204191 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a10027070 con 0x7f2a1c06c630 2024-05-03T21:18:44.401 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:44.396+0000 7f2a19ffb700 1 -- 172.21.15.67:0/3149204191 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(28 keys) ==== 1170+0+0 (secure 0 0 0) 0x7f2a10003f70 con 0x7f2a1c06c630 2024-05-03T21:18:44.401 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:44.396+0000 7f2a22e75700 1 -- 172.21.15.67:0/3149204191 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2a1c19a500 con 0x7f2a1c06c630 2024-05-03T21:18:44.401 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:44.396+0000 7f2a19ffb700 1 -- 172.21.15.67:0/3149204191 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a1000f460 con 0x7f2a1c06c630 2024-05-03T21:18:44.401 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:44.396+0000 7f2a22e75700 1 -- 172.21.15.67:0/3149204191 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2a1c1a67d0 con 0x7f2a1c06c630 2024-05-03T21:18:44.401 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:44.397+0000 7f2a22e75700 1 -- 172.21.15.67:0/3149204191 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2a1c10ba20 con 0x7f2a1c06c630 2024-05-03T21:18:44.402 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:44.398+0000 7f2a19ffb700 1 -- 172.21.15.67:0/3149204191 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 16) ==== 149652+0+0 (secure 0 0 0) 0x7f2a1002a070 con 0x7f2a1c06c630 2024-05-03T21:18:44.402 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:44.399+0000 7f2a19ffb700 1 --2- 172.21.15.67:0/3149204191 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f2a040b2900 0x7f2a040b4dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:18:44.402 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:44.399+0000 7f2a20c11700 1 --2- 172.21.15.67:0/3149204191 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f2a040b2900 0x7f2a040b4dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:18:44.402 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:44.399+0000 7f2a19ffb700 1 -- 172.21.15.67:0/3149204191 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7879+0+0 (secure 0 0 0) 0x7f2a100e2d60 con 0x7f2a1c06c630 2024-05-03T21:18:44.402 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:44.400+0000 7f2a20c11700 1 --2- 172.21.15.67:0/3149204191 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f2a040b2900 0x7f2a040b4dc0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f2a1c100a60 tx=0x7f2a0c00b3c0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T21:18:44.405 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:44.403+0000 7f2a19ffb700 1 -- 172.21.15.67:0/3149204191 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191839 (secure 0 0 0) 0x7f2a10013070 con 0x7f2a1c06c630 2024-05-03T21:18:44.666 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:44.664+0000 7f2a22e75700 1 -- 172.21.15.67:0/3149204191 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) -- 0x7f2a1c19aff0 con 0x7f2a1c06c630 2024-05-03T21:18:44.815 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:44.813+0000 7f2a19ffb700 1 -- 172.21.15.67:0/3149204191 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 2 and data pool 3 v14) ==== 172+0+0 (secure 0 0 0) 0x7f2a100aacc0 con 0x7f2a1c06c630 2024-05-03T21:18:44.818 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:44.815+0000 7f2a22e75700 1 -- 172.21.15.67:0/3149204191 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f2a040b2900 msgr2=0x7f2a040b4dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:18:44.818 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:44.815+0000 7f2a22e75700 1 --2- 172.21.15.67:0/3149204191 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f2a040b2900 0x7f2a040b4dc0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f2a1c100a60 tx=0x7f2a0c00b3c0 comp rx=0 tx=0).stop 2024-05-03T21:18:44.818 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:44.815+0000 7f2a22e75700 1 -- 172.21.15.67:0/3149204191 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2a1c06c630 msgr2=0x7f2a1c19fe40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:18:44.818 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:44.815+0000 7f2a22e75700 1 --2- 172.21.15.67:0/3149204191 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2a1c06c630 0x7f2a1c19fe40 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f2a100059d0 tx=0x7f2a1000bb00 comp rx=0 tx=0).stop 2024-05-03T21:18:44.819 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:44.816+0000 7f2a22e75700 1 -- 172.21.15.67:0/3149204191 shutdown_connections 2024-05-03T21:18:44.819 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:44.816+0000 7f2a22e75700 1 --2- 172.21.15.67:0/3149204191 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f2a040b2900 0x7f2a040b4dc0 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:18:44.819 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:44.816+0000 7f2a22e75700 1 --2- 172.21.15.67:0/3149204191 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f2a1c109590 0x7f2a1c199b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:18:44.819 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:44.816+0000 7f2a22e75700 1 --2- 172.21.15.67:0/3149204191 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f2a1c06bc70 0x7f2a1c19f900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:18:44.819 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:44.816+0000 7f2a22e75700 1 --2- 172.21.15.67:0/3149204191 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2a1c06c630 0x7f2a1c19fe40 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:18:44.819 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:44.816+0000 7f2a22e75700 1 -- 172.21.15.67:0/3149204191 >> 172.21.15.67:0/3149204191 conn(0x7f2a1c079440 msgr2=0x7f2a1c100e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T21:18:44.819 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:44.816+0000 7f2a22e75700 1 -- 172.21.15.67:0/3149204191 shutdown_connections 2024-05-03T21:18:44.819 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:44.816+0000 7f2a22e75700 1 -- 172.21.15.67:0/3149204191 wait complete. 2024-05-03T21:18:44.819 INFO:teuthology.orchestra.run.smithi067.stderr:new fs with metadata pool 2 and data pool 3 2024-05-03T21:18:44.971 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:18:44 smithi067 ceph-mon[29730]: pgmap v285: 97 pgs: 13 creating+peering, 39 unknown, 45 active+clean; 577 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-03T21:18:44.971 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:18:44 smithi067 ceph-mon[29730]: from='client.? 172.21.15.67:0/3149204191' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-05-03T21:18:44.971 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:18:44 smithi067 ceph-mon[29730]: daemon mds.b assigned to filesystem cephfs as rank 0 2024-05-03T21:18:44.971 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:18:44 smithi067 ceph-mon[29730]: osdmap e82: 12 total, 12 up, 12 in 2024-05-03T21:18:45.074 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:18:44 smithi138 ceph-mon[33110]: pgmap v285: 97 pgs: 13 creating+peering, 39 unknown, 45 active+clean; 577 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-03T21:18:45.074 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:18:44 smithi138 ceph-mon[33110]: from='client.? 172.21.15.67:0/3149204191' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-05-03T21:18:45.074 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:18:44 smithi138 ceph-mon[33110]: daemon mds.b assigned to filesystem cephfs as rank 0 2024-05-03T21:18:45.075 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:18:44 smithi138 ceph-mon[33110]: osdmap e82: 12 total, 12 up, 12 in 2024-05-03T21:18:45.147 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:18:44 smithi082 ceph-mon[32933]: pgmap v285: 97 pgs: 13 creating+peering, 39 unknown, 45 active+clean; 577 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-03T21:18:45.147 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:18:44 smithi082 ceph-mon[32933]: from='client.? 172.21.15.67:0/3149204191' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-05-03T21:18:45.147 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:18:44 smithi082 ceph-mon[32933]: daemon mds.b assigned to filesystem cephfs as rank 0 2024-05-03T21:18:45.147 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:18:44 smithi082 ceph-mon[32933]: osdmap e82: 12 total, 12 up, 12 in 2024-05-03T21:18:45.421 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9a047fe7599a549042d06a3fb5b6813a203eabea shell --fsid 713bc57a-0991-11ef-bc97-c7b262605968 -- ceph osd dump --format=json 2024-05-03T21:18:45.671 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/713bc57a-0991-11ef-bc97-c7b262605968/mon.a/config 2024-05-03T21:18:46.128 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:18:45 smithi067 ceph-mon[29730]: from='client.? 172.21.15.67:0/3149204191' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2024-05-03T21:18:46.128 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:18:45 smithi067 ceph-mon[29730]: fsmap cephfs:1 {0=b=up:creating} 11 up:standby 2024-05-03T21:18:46.129 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:18:45 smithi067 ceph-mon[29730]: daemon mds.b is now active in filesystem cephfs as rank 0 2024-05-03T21:18:46.180 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:18:45 smithi138 ceph-mon[33110]: from='client.? 172.21.15.67:0/3149204191' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2024-05-03T21:18:46.180 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:18:45 smithi138 ceph-mon[33110]: fsmap cephfs:1 {0=b=up:creating} 11 up:standby 2024-05-03T21:18:46.181 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:18:45 smithi138 ceph-mon[33110]: daemon mds.b is now active in filesystem cephfs as rank 0 2024-05-03T21:18:46.245 INFO:journalctl@ceph.mds.b.smithi082.stdout:May 03 21:18:45 smithi082 ceph-713bc57a-0991-11ef-bc97-c7b262605968-mds-b[63361]: 2024-05-03T21:18:45.974+0000 7fef1f85a700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-05-03T21:18:46.246 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:18:45 smithi082 ceph-mon[32933]: from='client.? 172.21.15.67:0/3149204191' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2024-05-03T21:18:46.246 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:18:45 smithi082 ceph-mon[32933]: fsmap cephfs:1 {0=b=up:creating} 11 up:standby 2024-05-03T21:18:46.246 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:18:45 smithi082 ceph-mon[32933]: daemon mds.b is now active in filesystem cephfs as rank 0 2024-05-03T21:18:47.161 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:18:46 smithi067 ceph-mon[29730]: pgmap v288: 97 pgs: 13 creating+peering, 21 unknown, 63 active+clean; 578 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 765 B/s wr, 1 op/s 2024-05-03T21:18:47.161 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:18:46 smithi067 ceph-mon[29730]: Health check cleared: POOL_APP_NOT_ENABLED (was: 2 pool(s) do not have an application enabled) 2024-05-03T21:18:47.161 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:18:46 smithi067 ceph-mon[29730]: Cluster is now healthy 2024-05-03T21:18:47.161 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:18:46 smithi067 ceph-mon[29730]: mds.? [v2:172.21.15.82:6838/3644499578,v1:172.21.15.82:6839/3644499578] up:active 2024-05-03T21:18:47.161 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:18:46 smithi067 ceph-mon[29730]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-05-03T21:18:47.161 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:18:46 smithi067 ceph-mon[29730]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-05-03T21:18:47.245 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:18:46 smithi082 ceph-mon[32933]: pgmap v288: 97 pgs: 13 creating+peering, 21 unknown, 63 active+clean; 578 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 765 B/s wr, 1 op/s 2024-05-03T21:18:47.246 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:18:46 smithi082 ceph-mon[32933]: Health check cleared: POOL_APP_NOT_ENABLED (was: 2 pool(s) do not have an application enabled) 2024-05-03T21:18:47.246 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:18:46 smithi082 ceph-mon[32933]: Cluster is now healthy 2024-05-03T21:18:47.246 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:18:46 smithi082 ceph-mon[32933]: mds.? [v2:172.21.15.82:6838/3644499578,v1:172.21.15.82:6839/3644499578] up:active 2024-05-03T21:18:47.246 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:18:46 smithi082 ceph-mon[32933]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-05-03T21:18:47.246 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:18:46 smithi082 ceph-mon[32933]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-05-03T21:18:47.266 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:47.263+0000 7fa336da2700 1 -- 172.21.15.67:0/2149451584 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa330104f20 msgr2=0x7fa33010d410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:18:47.266 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:47.263+0000 7fa336da2700 1 --2- 172.21.15.67:0/2149451584 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa330104f20 0x7fa33010d410 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fa32c009990 tx=0x7fa32c009ca0 comp rx=0 tx=0).stop 2024-05-03T21:18:47.266 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:47.264+0000 7fa336da2700 1 -- 172.21.15.67:0/2149451584 shutdown_connections 2024-05-03T21:18:47.266 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:47.264+0000 7fa336da2700 1 --2- 172.21.15.67:0/2149451584 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fa33010da30 0x7fa33010fef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:18:47.266 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:47.264+0000 7fa336da2700 1 --2- 172.21.15.67:0/2149451584 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fa330104600 0x7fa3301049e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:18:47.266 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:47.264+0000 7fa336da2700 1 --2- 172.21.15.67:0/2149451584 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa330104f20 0x7fa33010d410 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:18:47.267 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:47.264+0000 7fa336da2700 1 -- 172.21.15.67:0/2149451584 >> 172.21.15.67:0/2149451584 conn(0x7fa3300fde50 msgr2=0x7fa330100270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T21:18:47.267 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:47.264+0000 7fa336da2700 1 -- 172.21.15.67:0/2149451584 shutdown_connections 2024-05-03T21:18:47.267 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:47.264+0000 7fa336da2700 1 -- 172.21.15.67:0/2149451584 wait complete. 2024-05-03T21:18:47.268 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:47.264+0000 7fa336da2700 1 Processor -- start 2024-05-03T21:18:47.269 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:47.264+0000 7fa336da2700 1 -- start start 2024-05-03T21:18:47.269 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:47.265+0000 7fa336da2700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa330104600 0x7fa330199e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:18:47.269 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:47.265+0000 7fa336da2700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fa330104f20 0x7fa33019a370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:18:47.269 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:47.265+0000 7fa336da2700 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fa33010da30 0x7fa33019e6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:18:47.269 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:47.265+0000 7fa336da2700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fa330113960 con 0x7fa330104600 2024-05-03T21:18:47.269 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:47.265+0000 7fa336da2700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7fa3301137e0 con 0x7fa330104f20 2024-05-03T21:18:47.269 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:47.265+0000 7fa336da2700 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7fa330113ae0 con 0x7fa33010da30 2024-05-03T21:18:47.269 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:47.265+0000 7fa335da0700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa330104600 0x7fa330199e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:18:47.269 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:47.265+0000 7fa335da0700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa330104600 0x7fa330199e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:40386/0 (socket says 172.21.15.67:40386) 2024-05-03T21:18:47.269 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:47.265+0000 7fa335da0700 1 -- 172.21.15.67:0/1351885054 learned_addr learned my addr 172.21.15.67:0/1351885054 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-03T21:18:47.269 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:47.265+0000 7fa33559f700 1 --2- 172.21.15.67:0/1351885054 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fa330104f20 0x7fa33019a370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:18:47.269 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:47.265+0000 7fa3365a1700 1 --2- 172.21.15.67:0/1351885054 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fa33010da30 0x7fa33019e6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:18:47.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:47.265+0000 7fa335da0700 1 -- 172.21.15.67:0/1351885054 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fa33010da30 msgr2=0x7fa33019e6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:18:47.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:47.265+0000 7fa335da0700 1 --2- 172.21.15.67:0/1351885054 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fa33010da30 0x7fa33019e6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:18:47.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:47.265+0000 7fa335da0700 1 -- 172.21.15.67:0/1351885054 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fa330104f20 msgr2=0x7fa33019a370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:18:47.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:47.265+0000 7fa335da0700 1 --2- 172.21.15.67:0/1351885054 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fa330104f20 0x7fa33019a370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:18:47.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:47.265+0000 7fa335da0700 1 -- 172.21.15.67:0/1351885054 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa33019edf0 con 0x7fa330104600 2024-05-03T21:18:47.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:47.265+0000 7fa3365a1700 1 --2- 172.21.15.67:0/1351885054 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fa33010da30 0x7fa33019e6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-03T21:18:47.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:47.265+0000 7fa335da0700 1 --2- 172.21.15.67:0/1351885054 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa330104600 0x7fa330199e30 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fa32400b530 tx=0x7fa32400b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T21:18:47.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:47.265+0000 7fa322ffd700 1 -- 172.21.15.67:0/1351885054 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa324015020 con 0x7fa330104600 2024-05-03T21:18:47.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:47.265+0000 7fa322ffd700 1 -- 172.21.15.67:0/1351885054 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(28 keys) ==== 1170+0+0 (secure 0 0 0) 0x7fa324011d30 con 0x7fa330104600 2024-05-03T21:18:47.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:47.265+0000 7fa336da2700 1 -- 172.21.15.67:0/1351885054 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa33019f0e0 con 0x7fa330104600 2024-05-03T21:18:47.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:47.265+0000 7fa322ffd700 1 -- 172.21.15.67:0/1351885054 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa32400f2e0 con 0x7fa330104600 2024-05-03T21:18:47.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:47.266+0000 7fa336da2700 1 -- 172.21.15.67:0/1351885054 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa3301a6960 con 0x7fa330104600 2024-05-03T21:18:47.271 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:47.267+0000 7fa336da2700 1 -- 172.21.15.67:0/1351885054 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa33006a190 con 0x7fa330104600 2024-05-03T21:18:47.271 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:47.268+0000 7fa322ffd700 1 -- 172.21.15.67:0/1351885054 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 16) ==== 149652+0+0 (secure 0 0 0) 0x7fa324018020 con 0x7fa330104600 2024-05-03T21:18:47.271 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:47.269+0000 7fa322ffd700 1 --2- 172.21.15.67:0/1351885054 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7fa31c0b2880 0x7fa31c0b4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:18:47.271 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:47.269+0000 7fa33559f700 1 --2- 172.21.15.67:0/1351885054 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7fa31c0b2880 0x7fa31c0b4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:18:47.272 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:47.269+0000 7fa322ffd700 1 -- 172.21.15.67:0/1351885054 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7979+0+0 (secure 0 0 0) 0x7fa3240e1f70 con 0x7fa330104600 2024-05-03T21:18:47.272 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:47.269+0000 7fa33559f700 1 --2- 172.21.15.67:0/1351885054 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7fa31c0b2880 0x7fa31c0b4d40 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fa330101a50 tx=0x7fa32c005fd0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T21:18:47.275 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:47.273+0000 7fa322ffd700 1 -- 172.21.15.67:0/1351885054 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191839 (secure 0 0 0) 0x7fa32401f020 con 0x7fa330104600 2024-05-03T21:18:47.324 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:18:46 smithi138 ceph-mon[33110]: pgmap v288: 97 pgs: 13 creating+peering, 21 unknown, 63 active+clean; 578 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 765 B/s wr, 1 op/s 2024-05-03T21:18:47.325 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:18:46 smithi138 ceph-mon[33110]: Health check cleared: POOL_APP_NOT_ENABLED (was: 2 pool(s) do not have an application enabled) 2024-05-03T21:18:47.325 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:18:46 smithi138 ceph-mon[33110]: Cluster is now healthy 2024-05-03T21:18:47.325 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:18:46 smithi138 ceph-mon[33110]: mds.? [v2:172.21.15.82:6838/3644499578,v1:172.21.15.82:6839/3644499578] up:active 2024-05-03T21:18:47.325 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:18:46 smithi138 ceph-mon[33110]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-05-03T21:18:47.325 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:18:46 smithi138 ceph-mon[33110]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-05-03T21:18:47.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:47.486+0000 7fa336da2700 1 -- 172.21.15.67:0/1351885054 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fa33019b140 con 0x7fa330104600 2024-05-03T21:18:47.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:47.488+0000 7fa322ffd700 1 -- 172.21.15.67:0/1351885054 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v82) ==== 74+0+22205 (secure 0 0 0) 0x7fa3240a9d40 con 0x7fa330104600 2024-05-03T21:18:47.490 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-05-03T21:18:47.491 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":82,"fsid":"713bc57a-0991-11ef-bc97-c7b262605968","created":"2024-05-03T21:10:34.070178+0000","modified":"2024-05-03T21:18:44.665863+0000","last_up_change":"2024-05-03T21:17:39.993919+0000","last_in_change":"2024-05-03T21:17:19.978727+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-03T21:13:59.810446+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-05-03T21:18:37.953919+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"82","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-05-03T21:18:41.571707+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"82","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7939f31c-6fb7-42bd-af96-88a448d11bd2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":3692483807},{"type":"v1","addr":"172.21.15.67:6803","nonce":3692483807}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":3692483807},{"type":"v1","addr":"172.21.15.67:6805","nonce":3692483807}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":3692483807},{"type":"v1","addr":"172.21.15.67:6809","nonce":3692483807}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":3692483807},{"type":"v1","addr":"172.21.15.67:6807","nonce":3692483807}]},"public_addr":"172.21.15.67:6803/3692483807","cluster_addr":"172.21.15.67:6805/3692483807","heartbeat_back_addr":"172.21.15.67:6809/3692483807","heartbeat_front_addr":"172.21.15.67:6807/3692483807","state":["exists","up"]},{"osd":1,"uuid":"71858193-05a6-4c2c-9bb7-bc3d811d634e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6800","nonce":1139376397},{"type":"v1","addr":"172.21.15.82:6801","nonce":1139376397}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6802","nonce":1139376397},{"type":"v1","addr":"172.21.15.82:6803","nonce":1139376397}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6806","nonce":1139376397},{"type":"v1","addr":"172.21.15.82:6807","nonce":1139376397}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6804","nonce":1139376397},{"type":"v1","addr":"172.21.15.82:6805","nonce":1139376397}]},"public_addr":"172.21.15.82:6801/1139376397","cluster_addr":"172.21.15.82:6803/1139376397","heartbeat_back_addr":"172.21.15.82:6807/1139376397","heartbeat_front_addr":"172.21.15.82:6805/1139376397","state":["exists","up"]},{"osd":2,"uuid":"f9ece78d-6a19-4d46-9022-6ed184074745","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6800","nonce":2774330123},{"type":"v1","addr":"172.21.15.138:6801","nonce":2774330123}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6802","nonce":2774330123},{"type":"v1","addr":"172.21.15.138:6803","nonce":2774330123}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6806","nonce":2774330123},{"type":"v1","addr":"172.21.15.138:6807","nonce":2774330123}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6804","nonce":2774330123},{"type":"v1","addr":"172.21.15.138:6805","nonce":2774330123}]},"public_addr":"172.21.15.138:6801/2774330123","cluster_addr":"172.21.15.138:6803/2774330123","heartbeat_back_addr":"172.21.15.138:6807/2774330123","heartbeat_front_addr":"172.21.15.138:6805/2774330123","state":["exists","up"]},{"osd":3,"uuid":"3616df71-e7b3-4687-aaba-37b0b7766a3c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":27,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":3915934652},{"type":"v1","addr":"172.21.15.67:6811","nonce":3915934652}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":3915934652},{"type":"v1","addr":"172.21.15.67:6813","nonce":3915934652}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":3915934652},{"type":"v1","addr":"172.21.15.67:6817","nonce":3915934652}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":3915934652},{"type":"v1","addr":"172.21.15.67:6815","nonce":3915934652}]},"public_addr":"172.21.15.67:6811/3915934652","cluster_addr":"172.21.15.67:6813/3915934652","heartbeat_back_addr":"172.21.15.67:6817/3915934652","heartbeat_front_addr":"172.21.15.67:6815/3915934652","state":["exists","up"]},{"osd":4,"uuid":"c3fd0ffd-e23f-4279-b679-9d95a144cf32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6808","nonce":4031818951},{"type":"v1","addr":"172.21.15.82:6809","nonce":4031818951}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6810","nonce":4031818951},{"type":"v1","addr":"172.21.15.82:6811","nonce":4031818951}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6814","nonce":4031818951},{"type":"v1","addr":"172.21.15.82:6815","nonce":4031818951}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6812","nonce":4031818951},{"type":"v1","addr":"172.21.15.82:6813","nonce":4031818951}]},"public_addr":"172.21.15.82:6809/4031818951","cluster_addr":"172.21.15.82:6811/4031818951","heartbeat_back_addr":"172.21.15.82:6815/4031818951","heartbeat_front_addr":"172.21.15.82:6813/4031818951","state":["exists","up"]},{"osd":5,"uuid":"540ef0d9-a6dc-43c6-b09e-e87340ba24cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6808","nonce":2521889499},{"type":"v1","addr":"172.21.15.138:6809","nonce":2521889499}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6810","nonce":2521889499},{"type":"v1","addr":"172.21.15.138:6811","nonce":2521889499}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6814","nonce":2521889499},{"type":"v1","addr":"172.21.15.138:6815","nonce":2521889499}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6812","nonce":2521889499},{"type":"v1","addr":"172.21.15.138:6813","nonce":2521889499}]},"public_addr":"172.21.15.138:6809/2521889499","cluster_addr":"172.21.15.138:6811/2521889499","heartbeat_back_addr":"172.21.15.138:6815/2521889499","heartbeat_front_addr":"172.21.15.138:6813/2521889499","state":["exists","up"]},{"osd":6,"uuid":"49a89fd0-958a-4be9-a8db-6022f946153e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6818","nonce":3269682731},{"type":"v1","addr":"172.21.15.67:6819","nonce":3269682731}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":3269682731},{"type":"v1","addr":"172.21.15.67:6821","nonce":3269682731}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":3269682731},{"type":"v1","addr":"172.21.15.67:6825","nonce":3269682731}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":3269682731},{"type":"v1","addr":"172.21.15.67:6823","nonce":3269682731}]},"public_addr":"172.21.15.67:6819/3269682731","cluster_addr":"172.21.15.67:6821/3269682731","heartbeat_back_addr":"172.21.15.67:6825/3269682731","heartbeat_front_addr":"172.21.15.67:6823/3269682731","state":["exists","up"]},{"osd":7,"uuid":"8fa4cd64-ca6b-48ae-8db0-d369ce6102ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":50,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6816","nonce":3008589976},{"type":"v1","addr":"172.21.15.82:6817","nonce":3008589976}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6818","nonce":3008589976},{"type":"v1","addr":"172.21.15.82:6819","nonce":3008589976}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6822","nonce":3008589976},{"type":"v1","addr":"172.21.15.82:6823","nonce":3008589976}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6820","nonce":3008589976},{"type":"v1","addr":"172.21.15.82:6821","nonce":3008589976}]},"public_addr":"172.21.15.82:6817/3008589976","cluster_addr":"172.21.15.82:6819/3008589976","heartbeat_back_addr":"172.21.15.82:6823/3008589976","heartbeat_front_addr":"172.21.15.82:6821/3008589976","state":["exists","up"]},{"osd":8,"uuid":"fae39087-6c38-4771-a2b9-3f4fecb08edd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":57,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6816","nonce":2223920070},{"type":"v1","addr":"172.21.15.138:6817","nonce":2223920070}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6818","nonce":2223920070},{"type":"v1","addr":"172.21.15.138:6819","nonce":2223920070}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6822","nonce":2223920070},{"type":"v1","addr":"172.21.15.138:6823","nonce":2223920070}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6820","nonce":2223920070},{"type":"v1","addr":"172.21.15.138:6821","nonce":2223920070}]},"public_addr":"172.21.15.138:6817/2223920070","cluster_addr":"172.21.15.138:6819/2223920070","heartbeat_back_addr":"172.21.15.138:6823/2223920070","heartbeat_front_addr":"172.21.15.138:6821/2223920070","state":["exists","up"]},{"osd":9,"uuid":"3d387dbf-0163-41b7-98af-4fc45ed73ad4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":63,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":3939417129},{"type":"v1","addr":"172.21.15.67:6827","nonce":3939417129}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":3939417129},{"type":"v1","addr":"172.21.15.67:6829","nonce":3939417129}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":3939417129},{"type":"v1","addr":"172.21.15.67:6833","nonce":3939417129}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":3939417129},{"type":"v1","addr":"172.21.15.67:6831","nonce":3939417129}]},"public_addr":"172.21.15.67:6827/3939417129","cluster_addr":"172.21.15.67:6829/3939417129","heartbeat_back_addr":"172.21.15.67:6833/3939417129","heartbeat_front_addr":"172.21.15.67:6831/3939417129","state":["exists","up"]},{"osd":10,"uuid":"db0305a0-a83b-474b-837a-fbc239ac1738","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6824","nonce":2505507029},{"type":"v1","addr":"172.21.15.82:6825","nonce":2505507029}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6826","nonce":2505507029},{"type":"v1","addr":"172.21.15.82:6827","nonce":2505507029}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6830","nonce":2505507029},{"type":"v1","addr":"172.21.15.82:6831","nonce":2505507029}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6828","nonce":2505507029},{"type":"v1","addr":"172.21.15.82:6829","nonce":2505507029}]},"public_addr":"172.21.15.82:6825/2505507029","cluster_addr":"172.21.15.82:6827/2505507029","heartbeat_back_addr":"172.21.15.82:6831/2505507029","heartbeat_front_addr":"172.21.15.82:6829/2505507029","state":["exists","up"]},{"osd":11,"uuid":"479c2747-f28e-4704-87e4-d0c31782e2de","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":74,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6824","nonce":2528685018},{"type":"v1","addr":"172.21.15.138:6825","nonce":2528685018}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6826","nonce":2528685018},{"type":"v1","addr":"172.21.15.138:6827","nonce":2528685018}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6830","nonce":2528685018},{"type":"v1","addr":"172.21.15.138:6831","nonce":2528685018}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6828","nonce":2528685018},{"type":"v1","addr":"172.21.15.138:6829","nonce":2528685018}]},"public_addr":"172.21.15.138:6825/2528685018","cluster_addr":"172.21.15.138:6827/2528685018","heartbeat_back_addr":"172.21.15.138:6831/2528685018","heartbeat_front_addr":"172.21.15.138:6829/2528685018","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-03T21:13:07.768773+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-03T21:13:31.870232+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-03T21:13:55.394206+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-03T21:14:20.540281+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-03T21:14:44.318523+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-03T21:15:08.490829+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-03T21:15:34.144734+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-03T21:15:58.717240+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-03T21:16:22.743013+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-03T21:16:49.099588+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-03T21:17:13.321126+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-03T21:17:38.001978+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.67:0/1239643865":"2024-05-04T21:11:33.721416+0000","172.21.15.67:6801/2576113038":"2024-05-04T21:11:33.721416+0000","172.21.15.67:6800/2576113038":"2024-05-04T21:11:33.721416+0000","172.21.15.67:6800/1698171509":"2024-05-04T21:11:07.333651+0000","172.21.15.67:0/2217893923":"2024-05-04T21:11:33.721416+0000","172.21.15.67:0/20864811":"2024-05-04T21:11:07.333651+0000","172.21.15.67:6801/1698171509":"2024-05-04T21:11:07.333651+0000","172.21.15.67:0/773376553":"2024-05-04T21:11:33.721416+0000","172.21.15.67:0/847549114":"2024-05-04T21:11:07.333651+0000","172.21.15.67:0/1291778035":"2024-05-04T21:11:07.333651+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-03T21:18:47.492 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:47.490+0000 7fa336da2700 1 -- 172.21.15.67:0/1351885054 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7fa31c0b2880 msgr2=0x7fa31c0b4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:18:47.493 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:47.490+0000 7fa336da2700 1 --2- 172.21.15.67:0/1351885054 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7fa31c0b2880 0x7fa31c0b4d40 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fa330101a50 tx=0x7fa32c005fd0 comp rx=0 tx=0).stop 2024-05-03T21:18:47.493 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:47.490+0000 7fa336da2700 1 -- 172.21.15.67:0/1351885054 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa330104600 msgr2=0x7fa330199e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:18:47.493 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:47.490+0000 7fa336da2700 1 --2- 172.21.15.67:0/1351885054 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa330104600 0x7fa330199e30 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fa32400b530 tx=0x7fa32400b840 comp rx=0 tx=0).stop 2024-05-03T21:18:47.493 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:47.490+0000 7fa336da2700 1 -- 172.21.15.67:0/1351885054 shutdown_connections 2024-05-03T21:18:47.493 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:47.490+0000 7fa336da2700 1 --2- 172.21.15.67:0/1351885054 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7fa31c0b2880 0x7fa31c0b4d40 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:18:47.493 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:47.490+0000 7fa336da2700 1 --2- 172.21.15.67:0/1351885054 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa330104600 0x7fa330199e30 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:18:47.493 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:47.490+0000 7fa336da2700 1 --2- 172.21.15.67:0/1351885054 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fa33010da30 0x7fa33019e6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:18:47.493 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:47.490+0000 7fa336da2700 1 --2- 172.21.15.67:0/1351885054 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fa330104f20 0x7fa33019a370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:18:47.493 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:47.490+0000 7fa336da2700 1 -- 172.21.15.67:0/1351885054 >> 172.21.15.67:0/1351885054 conn(0x7fa3300fde50 msgr2=0x7fa3300feac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T21:18:47.493 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:47.491+0000 7fa336da2700 1 -- 172.21.15.67:0/1351885054 shutdown_connections 2024-05-03T21:18:47.493 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:47.491+0000 7fa336da2700 1 -- 172.21.15.67:0/1351885054 wait complete. 2024-05-03T21:18:48.098 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:18:47 smithi067 ceph-mon[29730]: from='client.? 172.21.15.67:0/1351885054' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-03T21:18:48.132 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9a047fe7599a549042d06a3fb5b6813a203eabea shell --fsid 713bc57a-0991-11ef-bc97-c7b262605968 -- ceph osd dump --format=json 2024-05-03T21:18:48.245 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:18:47 smithi082 ceph-mon[32933]: from='client.? 172.21.15.67:0/1351885054' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-03T21:18:48.324 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:18:47 smithi138 ceph-mon[33110]: from='client.? 172.21.15.67:0/1351885054' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-03T21:18:48.386 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/713bc57a-0991-11ef-bc97-c7b262605968/mon.a/config 2024-05-03T21:18:49.221 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:18:48 smithi067 ceph-mon[29730]: pgmap v289: 97 pgs: 97 active+clean; 579 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s wr, 6 op/s 2024-05-03T21:18:49.245 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:18:48 smithi082 ceph-mon[32933]: pgmap v289: 97 pgs: 97 active+clean; 579 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s wr, 6 op/s 2024-05-03T21:18:49.324 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:18:48 smithi138 ceph-mon[33110]: pgmap v289: 97 pgs: 97 active+clean; 579 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s wr, 6 op/s 2024-05-03T21:18:49.913 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:49.911+0000 7f17449b7700 1 -- 172.21.15.67:0/1420223711 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1740104100 msgr2=0x7f1740108120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:18:49.914 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:49.911+0000 7f17449b7700 1 --2- 172.21.15.67:0/1420223711 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1740104100 0x7f1740108120 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f17340099c0 tx=0x7f1734009cd0 comp rx=0 tx=0).stop 2024-05-03T21:18:49.914 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:49.911+0000 7f17449b7700 1 -- 172.21.15.67:0/1420223711 shutdown_connections 2024-05-03T21:18:49.914 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:49.911+0000 7f17449b7700 1 --2- 172.21.15.67:0/1420223711 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f1740108850 0x7f17401103d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:18:49.914 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:49.911+0000 7f17449b7700 1 --2- 172.21.15.67:0/1420223711 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f1740103750 0x7f1740103b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:18:49.914 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:49.911+0000 7f17449b7700 1 --2- 172.21.15.67:0/1420223711 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1740104100 0x7f1740108120 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:18:49.914 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:49.911+0000 7f17449b7700 1 -- 172.21.15.67:0/1420223711 >> 172.21.15.67:0/1420223711 conn(0x7f174007a600 msgr2=0x7f174007aa10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T21:18:49.914 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:49.911+0000 7f17449b7700 1 -- 172.21.15.67:0/1420223711 shutdown_connections 2024-05-03T21:18:49.915 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:49.911+0000 7f17449b7700 1 -- 172.21.15.67:0/1420223711 wait complete. 2024-05-03T21:18:49.917 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:49.912+0000 7f17449b7700 1 Processor -- start 2024-05-03T21:18:49.918 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:49.912+0000 7f17449b7700 1 -- start start 2024-05-03T21:18:49.918 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:49.912+0000 7f17449b7700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1740103750 0x7f174019fa50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:18:49.918 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:49.912+0000 7f17449b7700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f1740104100 0x7f174019ff90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:18:49.918 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:49.912+0000 7f17449b7700 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f1740108850 0x7f1740199b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:18:49.918 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:49.912+0000 7f17449b7700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f1740113920 con 0x7f1740103750 2024-05-03T21:18:49.918 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:49.912+0000 7f17449b7700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f17401137a0 con 0x7f1740104100 2024-05-03T21:18:49.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:49.912+0000 7f17449b7700 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f1740113aa0 con 0x7f1740108850 2024-05-03T21:18:49.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:49.912+0000 7f173e59c700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1740103750 0x7f174019fa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:18:49.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:49.912+0000 7f173e59c700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1740103750 0x7f174019fa50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:40406/0 (socket says 172.21.15.67:40406) 2024-05-03T21:18:49.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:49.912+0000 7f173e59c700 1 -- 172.21.15.67:0/3065580613 learned_addr learned my addr 172.21.15.67:0/3065580613 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-03T21:18:49.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:49.912+0000 7f173ed9d700 1 --2- 172.21.15.67:0/3065580613 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f1740108850 0x7f1740199b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:18:49.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:49.912+0000 7f173dd9b700 1 --2- 172.21.15.67:0/3065580613 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f1740104100 0x7f174019ff90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:18:49.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:49.913+0000 7f173e59c700 1 -- 172.21.15.67:0/3065580613 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f1740108850 msgr2=0x7f1740199b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:18:49.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:49.913+0000 7f173e59c700 1 --2- 172.21.15.67:0/3065580613 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f1740108850 0x7f1740199b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:18:49.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:49.913+0000 7f173e59c700 1 -- 172.21.15.67:0/3065580613 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f1740104100 msgr2=0x7f174019ff90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:18:49.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:49.913+0000 7f173e59c700 1 --2- 172.21.15.67:0/3065580613 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f1740104100 0x7f174019ff90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:18:49.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:49.913+0000 7f173e59c700 1 -- 172.21.15.67:0/3065580613 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f174019a400 con 0x7f1740103750 2024-05-03T21:18:49.921 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:49.913+0000 7f173e59c700 1 --2- 172.21.15.67:0/3065580613 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1740103750 0x7f174019fa50 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f173000b530 tx=0x7f173000b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T21:18:49.921 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:49.913+0000 7f172f7fe700 1 -- 172.21.15.67:0/3065580613 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1730015020 con 0x7f1740103750 2024-05-03T21:18:49.921 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:49.913+0000 7f172f7fe700 1 -- 172.21.15.67:0/3065580613 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(28 keys) ==== 1170+0+0 (secure 0 0 0) 0x7f1730011d30 con 0x7f1740103750 2024-05-03T21:18:49.921 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:49.913+0000 7f17449b7700 1 -- 172.21.15.67:0/3065580613 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f174019a6f0 con 0x7f1740103750 2024-05-03T21:18:49.921 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:49.913+0000 7f172f7fe700 1 -- 172.21.15.67:0/3065580613 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f173000f2e0 con 0x7f1740103750 2024-05-03T21:18:49.921 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:49.913+0000 7f17449b7700 1 -- 172.21.15.67:0/3065580613 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f17401a69d0 con 0x7f1740103750 2024-05-03T21:18:49.921 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:49.914+0000 7f17449b7700 1 -- 172.21.15.67:0/3065580613 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f174010ba90 con 0x7f1740103750 2024-05-03T21:18:49.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:49.915+0000 7f172f7fe700 1 -- 172.21.15.67:0/3065580613 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 16) ==== 149652+0+0 (secure 0 0 0) 0x7f1730018020 con 0x7f1740103750 2024-05-03T21:18:49.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:49.916+0000 7f172f7fe700 1 --2- 172.21.15.67:0/3065580613 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f17280b2880 0x7f17280b4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:18:49.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:49.916+0000 7f173dd9b700 1 --2- 172.21.15.67:0/3065580613 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f17280b2880 0x7f17280b4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:18:49.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:49.916+0000 7f172f7fe700 1 -- 172.21.15.67:0/3065580613 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7979+0+0 (secure 0 0 0) 0x7f17300e1b60 con 0x7f1740103750 2024-05-03T21:18:49.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:49.917+0000 7f173dd9b700 1 --2- 172.21.15.67:0/3065580613 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f17280b2880 0x7f17280b4d40 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f174019b300 tx=0x7f1734005fb0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T21:18:49.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:49.920+0000 7f172f7fe700 1 -- 172.21.15.67:0/3065580613 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191839 (secure 0 0 0) 0x7f173001f020 con 0x7f1740103750 2024-05-03T21:18:50.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:50.138+0000 7f17449b7700 1 -- 172.21.15.67:0/3065580613 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f174019afe0 con 0x7f1740103750 2024-05-03T21:18:50.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:50.139+0000 7f172f7fe700 1 -- 172.21.15.67:0/3065580613 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v82) ==== 74+0+22205 (secure 0 0 0) 0x7f17300a99d0 con 0x7f1740103750 2024-05-03T21:18:50.142 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-05-03T21:18:50.142 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":82,"fsid":"713bc57a-0991-11ef-bc97-c7b262605968","created":"2024-05-03T21:10:34.070178+0000","modified":"2024-05-03T21:18:44.665863+0000","last_up_change":"2024-05-03T21:17:39.993919+0000","last_in_change":"2024-05-03T21:17:19.978727+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-03T21:13:59.810446+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-05-03T21:18:37.953919+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"82","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-05-03T21:18:41.571707+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"82","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7939f31c-6fb7-42bd-af96-88a448d11bd2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":3692483807},{"type":"v1","addr":"172.21.15.67:6803","nonce":3692483807}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":3692483807},{"type":"v1","addr":"172.21.15.67:6805","nonce":3692483807}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":3692483807},{"type":"v1","addr":"172.21.15.67:6809","nonce":3692483807}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":3692483807},{"type":"v1","addr":"172.21.15.67:6807","nonce":3692483807}]},"public_addr":"172.21.15.67:6803/3692483807","cluster_addr":"172.21.15.67:6805/3692483807","heartbeat_back_addr":"172.21.15.67:6809/3692483807","heartbeat_front_addr":"172.21.15.67:6807/3692483807","state":["exists","up"]},{"osd":1,"uuid":"71858193-05a6-4c2c-9bb7-bc3d811d634e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6800","nonce":1139376397},{"type":"v1","addr":"172.21.15.82:6801","nonce":1139376397}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6802","nonce":1139376397},{"type":"v1","addr":"172.21.15.82:6803","nonce":1139376397}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6806","nonce":1139376397},{"type":"v1","addr":"172.21.15.82:6807","nonce":1139376397}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6804","nonce":1139376397},{"type":"v1","addr":"172.21.15.82:6805","nonce":1139376397}]},"public_addr":"172.21.15.82:6801/1139376397","cluster_addr":"172.21.15.82:6803/1139376397","heartbeat_back_addr":"172.21.15.82:6807/1139376397","heartbeat_front_addr":"172.21.15.82:6805/1139376397","state":["exists","up"]},{"osd":2,"uuid":"f9ece78d-6a19-4d46-9022-6ed184074745","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6800","nonce":2774330123},{"type":"v1","addr":"172.21.15.138:6801","nonce":2774330123}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6802","nonce":2774330123},{"type":"v1","addr":"172.21.15.138:6803","nonce":2774330123}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6806","nonce":2774330123},{"type":"v1","addr":"172.21.15.138:6807","nonce":2774330123}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6804","nonce":2774330123},{"type":"v1","addr":"172.21.15.138:6805","nonce":2774330123}]},"public_addr":"172.21.15.138:6801/2774330123","cluster_addr":"172.21.15.138:6803/2774330123","heartbeat_back_addr":"172.21.15.138:6807/2774330123","heartbeat_front_addr":"172.21.15.138:6805/2774330123","state":["exists","up"]},{"osd":3,"uuid":"3616df71-e7b3-4687-aaba-37b0b7766a3c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":27,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":3915934652},{"type":"v1","addr":"172.21.15.67:6811","nonce":3915934652}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":3915934652},{"type":"v1","addr":"172.21.15.67:6813","nonce":3915934652}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":3915934652},{"type":"v1","addr":"172.21.15.67:6817","nonce":3915934652}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":3915934652},{"type":"v1","addr":"172.21.15.67:6815","nonce":3915934652}]},"public_addr":"172.21.15.67:6811/3915934652","cluster_addr":"172.21.15.67:6813/3915934652","heartbeat_back_addr":"172.21.15.67:6817/3915934652","heartbeat_front_addr":"172.21.15.67:6815/3915934652","state":["exists","up"]},{"osd":4,"uuid":"c3fd0ffd-e23f-4279-b679-9d95a144cf32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6808","nonce":4031818951},{"type":"v1","addr":"172.21.15.82:6809","nonce":4031818951}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6810","nonce":4031818951},{"type":"v1","addr":"172.21.15.82:6811","nonce":4031818951}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6814","nonce":4031818951},{"type":"v1","addr":"172.21.15.82:6815","nonce":4031818951}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6812","nonce":4031818951},{"type":"v1","addr":"172.21.15.82:6813","nonce":4031818951}]},"public_addr":"172.21.15.82:6809/4031818951","cluster_addr":"172.21.15.82:6811/4031818951","heartbeat_back_addr":"172.21.15.82:6815/4031818951","heartbeat_front_addr":"172.21.15.82:6813/4031818951","state":["exists","up"]},{"osd":5,"uuid":"540ef0d9-a6dc-43c6-b09e-e87340ba24cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6808","nonce":2521889499},{"type":"v1","addr":"172.21.15.138:6809","nonce":2521889499}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6810","nonce":2521889499},{"type":"v1","addr":"172.21.15.138:6811","nonce":2521889499}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6814","nonce":2521889499},{"type":"v1","addr":"172.21.15.138:6815","nonce":2521889499}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6812","nonce":2521889499},{"type":"v1","addr":"172.21.15.138:6813","nonce":2521889499}]},"public_addr":"172.21.15.138:6809/2521889499","cluster_addr":"172.21.15.138:6811/2521889499","heartbeat_back_addr":"172.21.15.138:6815/2521889499","heartbeat_front_addr":"172.21.15.138:6813/2521889499","state":["exists","up"]},{"osd":6,"uuid":"49a89fd0-958a-4be9-a8db-6022f946153e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6818","nonce":3269682731},{"type":"v1","addr":"172.21.15.67:6819","nonce":3269682731}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":3269682731},{"type":"v1","addr":"172.21.15.67:6821","nonce":3269682731}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":3269682731},{"type":"v1","addr":"172.21.15.67:6825","nonce":3269682731}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":3269682731},{"type":"v1","addr":"172.21.15.67:6823","nonce":3269682731}]},"public_addr":"172.21.15.67:6819/3269682731","cluster_addr":"172.21.15.67:6821/3269682731","heartbeat_back_addr":"172.21.15.67:6825/3269682731","heartbeat_front_addr":"172.21.15.67:6823/3269682731","state":["exists","up"]},{"osd":7,"uuid":"8fa4cd64-ca6b-48ae-8db0-d369ce6102ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":50,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6816","nonce":3008589976},{"type":"v1","addr":"172.21.15.82:6817","nonce":3008589976}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6818","nonce":3008589976},{"type":"v1","addr":"172.21.15.82:6819","nonce":3008589976}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6822","nonce":3008589976},{"type":"v1","addr":"172.21.15.82:6823","nonce":3008589976}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6820","nonce":3008589976},{"type":"v1","addr":"172.21.15.82:6821","nonce":3008589976}]},"public_addr":"172.21.15.82:6817/3008589976","cluster_addr":"172.21.15.82:6819/3008589976","heartbeat_back_addr":"172.21.15.82:6823/3008589976","heartbeat_front_addr":"172.21.15.82:6821/3008589976","state":["exists","up"]},{"osd":8,"uuid":"fae39087-6c38-4771-a2b9-3f4fecb08edd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":57,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6816","nonce":2223920070},{"type":"v1","addr":"172.21.15.138:6817","nonce":2223920070}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6818","nonce":2223920070},{"type":"v1","addr":"172.21.15.138:6819","nonce":2223920070}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6822","nonce":2223920070},{"type":"v1","addr":"172.21.15.138:6823","nonce":2223920070}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6820","nonce":2223920070},{"type":"v1","addr":"172.21.15.138:6821","nonce":2223920070}]},"public_addr":"172.21.15.138:6817/2223920070","cluster_addr":"172.21.15.138:6819/2223920070","heartbeat_back_addr":"172.21.15.138:6823/2223920070","heartbeat_front_addr":"172.21.15.138:6821/2223920070","state":["exists","up"]},{"osd":9,"uuid":"3d387dbf-0163-41b7-98af-4fc45ed73ad4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":63,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":3939417129},{"type":"v1","addr":"172.21.15.67:6827","nonce":3939417129}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":3939417129},{"type":"v1","addr":"172.21.15.67:6829","nonce":3939417129}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":3939417129},{"type":"v1","addr":"172.21.15.67:6833","nonce":3939417129}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":3939417129},{"type":"v1","addr":"172.21.15.67:6831","nonce":3939417129}]},"public_addr":"172.21.15.67:6827/3939417129","cluster_addr":"172.21.15.67:6829/3939417129","heartbeat_back_addr":"172.21.15.67:6833/3939417129","heartbeat_front_addr":"172.21.15.67:6831/3939417129","state":["exists","up"]},{"osd":10,"uuid":"db0305a0-a83b-474b-837a-fbc239ac1738","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6824","nonce":2505507029},{"type":"v1","addr":"172.21.15.82:6825","nonce":2505507029}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6826","nonce":2505507029},{"type":"v1","addr":"172.21.15.82:6827","nonce":2505507029}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6830","nonce":2505507029},{"type":"v1","addr":"172.21.15.82:6831","nonce":2505507029}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6828","nonce":2505507029},{"type":"v1","addr":"172.21.15.82:6829","nonce":2505507029}]},"public_addr":"172.21.15.82:6825/2505507029","cluster_addr":"172.21.15.82:6827/2505507029","heartbeat_back_addr":"172.21.15.82:6831/2505507029","heartbeat_front_addr":"172.21.15.82:6829/2505507029","state":["exists","up"]},{"osd":11,"uuid":"479c2747-f28e-4704-87e4-d0c31782e2de","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":74,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6824","nonce":2528685018},{"type":"v1","addr":"172.21.15.138:6825","nonce":2528685018}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6826","nonce":2528685018},{"type":"v1","addr":"172.21.15.138:6827","nonce":2528685018}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6830","nonce":2528685018},{"type":"v1","addr":"172.21.15.138:6831","nonce":2528685018}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6828","nonce":2528685018},{"type":"v1","addr":"172.21.15.138:6829","nonce":2528685018}]},"public_addr":"172.21.15.138:6825/2528685018","cluster_addr":"172.21.15.138:6827/2528685018","heartbeat_back_addr":"172.21.15.138:6831/2528685018","heartbeat_front_addr":"172.21.15.138:6829/2528685018","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-03T21:13:07.768773+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-03T21:13:31.870232+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-03T21:13:55.394206+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-03T21:14:20.540281+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-03T21:14:44.318523+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-03T21:15:08.490829+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-03T21:15:34.144734+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-03T21:15:58.717240+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-03T21:16:22.743013+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-03T21:16:49.099588+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-03T21:17:13.321126+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-03T21:17:38.001978+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.67:0/1239643865":"2024-05-04T21:11:33.721416+0000","172.21.15.67:6801/2576113038":"2024-05-04T21:11:33.721416+0000","172.21.15.67:6800/2576113038":"2024-05-04T21:11:33.721416+0000","172.21.15.67:6800/1698171509":"2024-05-04T21:11:07.333651+0000","172.21.15.67:0/2217893923":"2024-05-04T21:11:33.721416+0000","172.21.15.67:0/20864811":"2024-05-04T21:11:07.333651+0000","172.21.15.67:6801/1698171509":"2024-05-04T21:11:07.333651+0000","172.21.15.67:0/773376553":"2024-05-04T21:11:33.721416+0000","172.21.15.67:0/847549114":"2024-05-04T21:11:07.333651+0000","172.21.15.67:0/1291778035":"2024-05-04T21:11:07.333651+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-03T21:18:50.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:50.141+0000 7f17449b7700 1 -- 172.21.15.67:0/3065580613 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f17280b2880 msgr2=0x7f17280b4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:18:50.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:50.141+0000 7f17449b7700 1 --2- 172.21.15.67:0/3065580613 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f17280b2880 0x7f17280b4d40 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f174019b300 tx=0x7f1734005fb0 comp rx=0 tx=0).stop 2024-05-03T21:18:50.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:50.141+0000 7f17449b7700 1 -- 172.21.15.67:0/3065580613 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1740103750 msgr2=0x7f174019fa50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:18:50.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:50.141+0000 7f17449b7700 1 --2- 172.21.15.67:0/3065580613 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1740103750 0x7f174019fa50 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f173000b530 tx=0x7f173000b840 comp rx=0 tx=0).stop 2024-05-03T21:18:50.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:50.141+0000 7f17449b7700 1 -- 172.21.15.67:0/3065580613 shutdown_connections 2024-05-03T21:18:50.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:50.141+0000 7f17449b7700 1 --2- 172.21.15.67:0/3065580613 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f17280b2880 0x7f17280b4d40 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:18:50.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:50.141+0000 7f17449b7700 1 --2- 172.21.15.67:0/3065580613 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1740103750 0x7f174019fa50 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:18:50.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:50.141+0000 7f17449b7700 1 --2- 172.21.15.67:0/3065580613 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f1740108850 0x7f1740199b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:18:50.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:50.141+0000 7f17449b7700 1 --2- 172.21.15.67:0/3065580613 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f1740104100 0x7f174019ff90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:18:50.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:50.141+0000 7f17449b7700 1 -- 172.21.15.67:0/3065580613 >> 172.21.15.67:0/3065580613 conn(0x7f174007a600 msgr2=0x7f17401014a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T21:18:50.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:50.142+0000 7f17449b7700 1 -- 172.21.15.67:0/3065580613 shutdown_connections 2024-05-03T21:18:50.145 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:50.142+0000 7f17449b7700 1 -- 172.21.15.67:0/3065580613 wait complete. 2024-05-03T21:18:50.807 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9a047fe7599a549042d06a3fb5b6813a203eabea shell --fsid 713bc57a-0991-11ef-bc97-c7b262605968 -- ceph fs set cephfs standby_count_wanted 0 2024-05-03T21:18:51.059 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/713bc57a-0991-11ef-bc97-c7b262605968/mon.a/config 2024-05-03T21:18:51.179 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:18:50 smithi067 ceph-mon[29730]: pgmap v290: 97 pgs: 97 active+clean; 579 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s wr, 6 op/s 2024-05-03T21:18:51.179 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:18:50 smithi067 ceph-mon[29730]: from='client.? 172.21.15.67:0/3065580613' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-03T21:18:51.245 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:18:50 smithi082 ceph-mon[32933]: pgmap v290: 97 pgs: 97 active+clean; 579 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s wr, 6 op/s 2024-05-03T21:18:51.245 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:18:50 smithi082 ceph-mon[32933]: from='client.? 172.21.15.67:0/3065580613' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-03T21:18:51.324 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:18:50 smithi138 ceph-mon[33110]: pgmap v290: 97 pgs: 97 active+clean; 579 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s wr, 6 op/s 2024-05-03T21:18:51.324 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:18:50 smithi138 ceph-mon[33110]: from='client.? 172.21.15.67:0/3065580613' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-03T21:18:52.670 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:52.667+0000 7f137d922700 1 -- 172.21.15.67:0/4151162232 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f13781023a0 msgr2=0x7f1378102820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:18:52.670 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:52.667+0000 7f137d922700 1 --2- 172.21.15.67:0/4151162232 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f13781023a0 0x7f1378102820 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f1360009990 tx=0x7f1360009ca0 comp rx=0 tx=0).stop 2024-05-03T21:18:52.670 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:52.667+0000 7f137d922700 1 -- 172.21.15.67:0/4151162232 shutdown_connections 2024-05-03T21:18:52.670 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:52.667+0000 7f137d922700 1 --2- 172.21.15.67:0/4151162232 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f1378102d60 0x7f137810f250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:18:52.670 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:52.667+0000 7f137d922700 1 --2- 172.21.15.67:0/4151162232 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f1378108410 0x7f13781087f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:18:52.670 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:52.667+0000 7f137d922700 1 --2- 172.21.15.67:0/4151162232 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f13781023a0 0x7f1378102820 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:18:52.670 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:52.667+0000 7f137d922700 1 -- 172.21.15.67:0/4151162232 >> 172.21.15.67:0/4151162232 conn(0x7f1378079350 msgr2=0x7f1378079760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T21:18:52.670 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:52.667+0000 7f137d922700 1 -- 172.21.15.67:0/4151162232 shutdown_connections 2024-05-03T21:18:52.670 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:52.667+0000 7f137d922700 1 -- 172.21.15.67:0/4151162232 wait complete. 2024-05-03T21:18:52.670 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:52.668+0000 7f137d922700 1 Processor -- start 2024-05-03T21:18:52.672 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:52.668+0000 7f137d922700 1 -- start start 2024-05-03T21:18:52.672 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:52.668+0000 7f137d922700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f13781023a0 0x7f1378104080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:18:52.672 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:52.668+0000 7f137d922700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f1378102d60 0x7f13781045c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:18:52.672 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:52.668+0000 7f137d922700 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f1378108410 0x7f1378104b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:18:52.672 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:52.668+0000 7f137d922700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f13781127e0 con 0x7f13781023a0 2024-05-03T21:18:52.672 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:52.668+0000 7f137d922700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f1378112660 con 0x7f1378102d60 2024-05-03T21:18:52.672 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:52.668+0000 7f137d922700 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f1378112960 con 0x7f1378108410 2024-05-03T21:18:52.672 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:52.668+0000 7f1376ffd700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f13781023a0 0x7f1378104080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:18:52.672 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:52.668+0000 7f1376ffd700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f13781023a0 0x7f1378104080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:40420/0 (socket says 172.21.15.67:40420) 2024-05-03T21:18:52.673 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:52.668+0000 7f1376ffd700 1 -- 172.21.15.67:0/2707077818 learned_addr learned my addr 172.21.15.67:0/2707077818 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-03T21:18:52.673 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:52.668+0000 7f13767fc700 1 --2- 172.21.15.67:0/2707077818 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f1378102d60 0x7f13781045c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:18:52.673 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:52.668+0000 7f13777fe700 1 --2- 172.21.15.67:0/2707077818 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f1378108410 0x7f1378104b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:18:52.673 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:52.669+0000 7f1376ffd700 1 -- 172.21.15.67:0/2707077818 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f1378108410 msgr2=0x7f1378104b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:18:52.673 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:52.669+0000 7f1376ffd700 1 --2- 172.21.15.67:0/2707077818 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f1378108410 0x7f1378104b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:18:52.673 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:52.669+0000 7f1376ffd700 1 -- 172.21.15.67:0/2707077818 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f1378102d60 msgr2=0x7f13781045c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:18:52.673 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:52.669+0000 7f1376ffd700 1 --2- 172.21.15.67:0/2707077818 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f1378102d60 0x7f13781045c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:18:52.673 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:52.669+0000 7f1376ffd700 1 -- 172.21.15.67:0/2707077818 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f13781a8230 con 0x7f13781023a0 2024-05-03T21:18:52.673 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:52.669+0000 7f1376ffd700 1 --2- 172.21.15.67:0/2707077818 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f13781023a0 0x7f1378104080 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f1370007f10 tx=0x7f137000d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T21:18:52.673 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:52.669+0000 7f137c920700 1 -- 172.21.15.67:0/2707077818 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1370004bb0 con 0x7f13781023a0 2024-05-03T21:18:52.673 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:52.669+0000 7f137c920700 1 -- 172.21.15.67:0/2707077818 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(28 keys) ==== 1170+0+0 (secure 0 0 0) 0x7f1370004d50 con 0x7f13781023a0 2024-05-03T21:18:52.673 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:52.669+0000 7f137d922700 1 -- 172.21.15.67:0/2707077818 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f13781a8520 con 0x7f13781023a0 2024-05-03T21:18:52.673 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:52.669+0000 7f137c920700 1 -- 172.21.15.67:0/2707077818 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13700133f0 con 0x7f13781023a0 2024-05-03T21:18:52.673 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:52.669+0000 7f137d922700 1 -- 172.21.15.67:0/2707077818 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f13781a89e0 con 0x7f13781023a0 2024-05-03T21:18:52.675 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:52.670+0000 7f137d922700 1 -- 172.21.15.67:0/2707077818 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f137806a190 con 0x7f13781023a0 2024-05-03T21:18:52.675 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:52.671+0000 7f137c920700 1 -- 172.21.15.67:0/2707077818 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 16) ==== 149652+0+0 (secure 0 0 0) 0x7f137001a020 con 0x7f13781023a0 2024-05-03T21:18:52.675 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:52.672+0000 7f137c920700 1 --2- 172.21.15.67:0/2707077818 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f13640b2880 0x7f13640b4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:18:52.675 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:52.672+0000 7f13767fc700 1 --2- 172.21.15.67:0/2707077818 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f13640b2880 0x7f13640b4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:18:52.675 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:52.672+0000 7f137c920700 1 -- 172.21.15.67:0/2707077818 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7979+0+0 (secure 0 0 0) 0x7f13700e1760 con 0x7f13781023a0 2024-05-03T21:18:52.675 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:52.673+0000 7f13767fc700 1 --2- 172.21.15.67:0/2707077818 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f13640b2880 0x7f13640b4d40 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f136000b5c0 tx=0x7f1360005960 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T21:18:52.679 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:52.677+0000 7f137c920700 1 -- 172.21.15.67:0/2707077818 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191839 (secure 0 0 0) 0x7f1370010070 con 0x7f13781023a0 2024-05-03T21:18:52.941 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:52.938+0000 7f137d922700 1 -- 172.21.15.67:0/2707077818 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) -- 0x7f13781a8bb0 con 0x7f13781023a0 2024-05-03T21:18:52.941 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:18:52 smithi067 ceph-mon[29730]: pgmap v291: 97 pgs: 97 active+clean; 579 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s wr, 5 op/s 2024-05-03T21:18:53.245 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:18:52 smithi082 ceph-mon[32933]: pgmap v291: 97 pgs: 97 active+clean; 579 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s wr, 5 op/s 2024-05-03T21:18:53.324 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:18:52 smithi138 ceph-mon[33110]: pgmap v291: 97 pgs: 97 active+clean; 579 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s wr, 5 op/s 2024-05-03T21:18:53.846 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:53.844+0000 7f137c920700 1 -- 172.21.15.67:0/2707077818 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v17) ==== 118+0+0 (secure 0 0 0) 0x7f13700a95d0 con 0x7f13781023a0 2024-05-03T21:18:53.849 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:53.846+0000 7f137d922700 1 -- 172.21.15.67:0/2707077818 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f13640b2880 msgr2=0x7f13640b4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:18:53.849 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:53.846+0000 7f137d922700 1 --2- 172.21.15.67:0/2707077818 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f13640b2880 0x7f13640b4d40 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f136000b5c0 tx=0x7f1360005960 comp rx=0 tx=0).stop 2024-05-03T21:18:53.849 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:53.846+0000 7f137d922700 1 -- 172.21.15.67:0/2707077818 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f13781023a0 msgr2=0x7f1378104080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:18:53.849 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:53.846+0000 7f137d922700 1 --2- 172.21.15.67:0/2707077818 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f13781023a0 0x7f1378104080 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f1370007f10 tx=0x7f137000d3b0 comp rx=0 tx=0).stop 2024-05-03T21:18:53.849 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:53.846+0000 7f137d922700 1 -- 172.21.15.67:0/2707077818 shutdown_connections 2024-05-03T21:18:53.849 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:53.846+0000 7f137d922700 1 --2- 172.21.15.67:0/2707077818 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f13640b2880 0x7f13640b4d40 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:18:53.849 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:53.846+0000 7f137d922700 1 --2- 172.21.15.67:0/2707077818 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f13781023a0 0x7f1378104080 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:18:53.849 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:53.846+0000 7f137d922700 1 --2- 172.21.15.67:0/2707077818 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f1378108410 0x7f1378104b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:18:53.849 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:53.846+0000 7f137d922700 1 --2- 172.21.15.67:0/2707077818 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f1378102d60 0x7f13781045c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:18:53.849 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:53.846+0000 7f137d922700 1 -- 172.21.15.67:0/2707077818 >> 172.21.15.67:0/2707077818 conn(0x7f1378079350 msgr2=0x7f13781015e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T21:18:53.850 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:53.846+0000 7f137d922700 1 -- 172.21.15.67:0/2707077818 shutdown_connections 2024-05-03T21:18:53.850 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:53.846+0000 7f137d922700 1 -- 172.21.15.67:0/2707077818 wait complete. 2024-05-03T21:18:54.221 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:18:53 smithi067 ceph-mon[29730]: from='client.? 172.21.15.67:0/2707077818' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-05-03T21:18:54.245 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:18:53 smithi082 ceph-mon[32933]: from='client.? 172.21.15.67:0/2707077818' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-05-03T21:18:54.324 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:18:53 smithi138 ceph-mon[33110]: from='client.? 172.21.15.67:0/2707077818' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-05-03T21:18:54.506 DEBUG:tasks.cephfs.filesystem:fs_config: {'max_mds': 3, 'session_timeout': 300, 'standby_replay': True, 'subvols': {'create': 2}} 2024-05-03T21:18:54.506 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9a047fe7599a549042d06a3fb5b6813a203eabea shell --fsid 713bc57a-0991-11ef-bc97-c7b262605968 -- ceph fs set cephfs max_mds 3 2024-05-03T21:18:54.758 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/713bc57a-0991-11ef-bc97-c7b262605968/mon.a/config 2024-05-03T21:18:55.179 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:18:54 smithi067 ceph-mon[29730]: pgmap v292: 97 pgs: 97 active+clean; 579 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s wr, 4 op/s 2024-05-03T21:18:55.179 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:18:54 smithi067 ceph-mon[29730]: from='client.? 172.21.15.67:0/2707077818' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2024-05-03T21:18:55.179 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:18:54 smithi067 ceph-mon[29730]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-05-03T21:18:55.245 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:18:54 smithi082 ceph-mon[32933]: pgmap v292: 97 pgs: 97 active+clean; 579 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s wr, 4 op/s 2024-05-03T21:18:55.246 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:18:54 smithi082 ceph-mon[32933]: from='client.? 172.21.15.67:0/2707077818' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2024-05-03T21:18:55.246 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:18:54 smithi082 ceph-mon[32933]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-05-03T21:18:55.324 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:18:54 smithi138 ceph-mon[33110]: pgmap v292: 97 pgs: 97 active+clean; 579 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s wr, 4 op/s 2024-05-03T21:18:55.324 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:18:54 smithi138 ceph-mon[33110]: from='client.? 172.21.15.67:0/2707077818' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2024-05-03T21:18:55.324 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:18:54 smithi138 ceph-mon[33110]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-05-03T21:18:56.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:56.365+0000 7fa71edda700 1 -- 172.21.15.67:0/1148942672 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa7181026c0 msgr2=0x7fa718102b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:18:56.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:56.365+0000 7fa71edda700 1 --2- 172.21.15.67:0/1148942672 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa7181026c0 0x7fa718102b40 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fa70c009990 tx=0x7fa70c009ca0 comp rx=0 tx=0).stop 2024-05-03T21:18:56.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:56.365+0000 7fa71edda700 1 -- 172.21.15.67:0/1148942672 shutdown_connections 2024-05-03T21:18:56.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:56.365+0000 7fa71edda700 1 --2- 172.21.15.67:0/1148942672 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fa718103080 0x7fa71810f1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:18:56.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:56.365+0000 7fa71edda700 1 --2- 172.21.15.67:0/1148942672 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fa718108510 0x7fa7181088f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:18:56.370 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:56.365+0000 7fa71edda700 1 --2- 172.21.15.67:0/1148942672 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa7181026c0 0x7fa718102b40 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:18:56.370 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:56.365+0000 7fa71edda700 1 -- 172.21.15.67:0/1148942672 >> 172.21.15.67:0/1148942672 conn(0x7fa718079100 msgr2=0x7fa718079510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T21:18:56.370 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:56.365+0000 7fa71edda700 1 -- 172.21.15.67:0/1148942672 shutdown_connections 2024-05-03T21:18:56.370 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:56.365+0000 7fa71edda700 1 -- 172.21.15.67:0/1148942672 wait complete. 2024-05-03T21:18:56.370 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:56.365+0000 7fa71edda700 1 Processor -- start 2024-05-03T21:18:56.370 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:56.366+0000 7fa71edda700 1 -- start start 2024-05-03T21:18:56.370 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:56.366+0000 7fa71edda700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fa7181026c0 0x7fa71819f880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:18:56.370 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:56.366+0000 7fa71edda700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa718103080 0x7fa71819fdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:18:56.370 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:56.366+0000 7fa71edda700 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fa718108510 0x7fa718199a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:18:56.370 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:56.366+0000 7fa71edda700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fa718112810 con 0x7fa718103080 2024-05-03T21:18:56.370 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:56.366+0000 7fa71edda700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7fa718112690 con 0x7fa7181026c0 2024-05-03T21:18:56.370 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:56.366+0000 7fa71edda700 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7fa718112990 con 0x7fa718108510 2024-05-03T21:18:56.371 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:56.366+0000 7fa717fff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa718103080 0x7fa71819fdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:18:56.371 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:56.366+0000 7fa717fff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa718103080 0x7fa71819fdc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:42296/0 (socket says 172.21.15.67:42296) 2024-05-03T21:18:56.371 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:56.366+0000 7fa717fff700 1 -- 172.21.15.67:0/2896894751 learned_addr learned my addr 172.21.15.67:0/2896894751 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-03T21:18:56.371 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:56.366+0000 7fa71cb76700 1 --2- 172.21.15.67:0/2896894751 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fa7181026c0 0x7fa71819f880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:18:56.371 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:56.366+0000 7fa71d377700 1 --2- 172.21.15.67:0/2896894751 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fa718108510 0x7fa718199a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:18:56.371 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:56.366+0000 7fa717fff700 1 -- 172.21.15.67:0/2896894751 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fa718108510 msgr2=0x7fa718199a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:18:56.371 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:56.366+0000 7fa717fff700 1 --2- 172.21.15.67:0/2896894751 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fa718108510 0x7fa718199a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:18:56.371 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:56.366+0000 7fa717fff700 1 -- 172.21.15.67:0/2896894751 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fa7181026c0 msgr2=0x7fa71819f880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:18:56.371 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:56.366+0000 7fa717fff700 1 --2- 172.21.15.67:0/2896894751 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fa7181026c0 0x7fa71819f880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:18:56.371 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:56.366+0000 7fa717fff700 1 -- 172.21.15.67:0/2896894751 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa71819a1a0 con 0x7fa718103080 2024-05-03T21:18:56.371 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:56.366+0000 7fa717fff700 1 --2- 172.21.15.67:0/2896894751 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa718103080 0x7fa71819fdc0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fa70c000c00 tx=0x7fa70c0047b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T21:18:56.371 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:56.367+0000 7fa715ffb700 1 -- 172.21.15.67:0/2896894751 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa70c027070 con 0x7fa718103080 2024-05-03T21:18:56.371 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:56.367+0000 7fa71edda700 1 -- 172.21.15.67:0/2896894751 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa71819a430 con 0x7fa718103080 2024-05-03T21:18:56.372 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:56.367+0000 7fa715ffb700 1 -- 172.21.15.67:0/2896894751 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(28 keys) ==== 1170+0+0 (secure 0 0 0) 0x7fa70c022470 con 0x7fa718103080 2024-05-03T21:18:56.372 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:56.367+0000 7fa715ffb700 1 -- 172.21.15.67:0/2896894751 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa70c017650 con 0x7fa718103080 2024-05-03T21:18:56.372 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:56.367+0000 7fa71edda700 1 -- 172.21.15.67:0/2896894751 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa7181a6750 con 0x7fa718103080 2024-05-03T21:18:56.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:56.368+0000 7fa71edda700 1 -- 172.21.15.67:0/2896894751 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa718103b30 con 0x7fa718103080 2024-05-03T21:18:56.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:56.369+0000 7fa715ffb700 1 -- 172.21.15.67:0/2896894751 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 16) ==== 149652+0+0 (secure 0 0 0) 0x7fa70c02a070 con 0x7fa718103080 2024-05-03T21:18:56.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:56.370+0000 7fa715ffb700 1 --2- 172.21.15.67:0/2896894751 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7fa7000b2880 0x7fa7000b4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:18:56.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:56.370+0000 7fa71cb76700 1 --2- 172.21.15.67:0/2896894751 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7fa7000b2880 0x7fa7000b4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:18:56.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:56.370+0000 7fa715ffb700 1 -- 172.21.15.67:0/2896894751 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7979+0+0 (secure 0 0 0) 0x7fa70c0e3090 con 0x7fa718103080 2024-05-03T21:18:56.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:56.370+0000 7fa71cb76700 1 --2- 172.21.15.67:0/2896894751 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7fa7000b2880 0x7fa7000b4d40 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fa71819ab80 tx=0x7fa708009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T21:18:56.375 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:56.373+0000 7fa715ffb700 1 -- 172.21.15.67:0/2896894751 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191839 (secure 0 0 0) 0x7fa70c013070 con 0x7fa718103080 2024-05-03T21:18:56.638 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:56.634+0000 7fa71edda700 1 -- 172.21.15.67:0/2896894751 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"} v 0) -- 0x7fa7181a6920 con 0x7fa718103080 2024-05-03T21:18:56.850 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:56.848+0000 7fa715ffb700 1 -- 172.21.15.67:0/2896894751 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]=0 v18) ==== 105+0+0 (secure 0 0 0) 0x7fa70c0aaf90 con 0x7fa718103080 2024-05-03T21:18:56.853 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:56.850+0000 7fa71edda700 1 -- 172.21.15.67:0/2896894751 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7fa7000b2880 msgr2=0x7fa7000b4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:18:56.853 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:56.850+0000 7fa71edda700 1 --2- 172.21.15.67:0/2896894751 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7fa7000b2880 0x7fa7000b4d40 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fa71819ab80 tx=0x7fa708009260 comp rx=0 tx=0).stop 2024-05-03T21:18:56.853 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:56.850+0000 7fa71edda700 1 -- 172.21.15.67:0/2896894751 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa718103080 msgr2=0x7fa71819fdc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:18:56.853 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:56.850+0000 7fa71edda700 1 --2- 172.21.15.67:0/2896894751 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa718103080 0x7fa71819fdc0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fa70c000c00 tx=0x7fa70c0047b0 comp rx=0 tx=0).stop 2024-05-03T21:18:56.853 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:56.850+0000 7fa71edda700 1 -- 172.21.15.67:0/2896894751 shutdown_connections 2024-05-03T21:18:56.853 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:56.850+0000 7fa71edda700 1 --2- 172.21.15.67:0/2896894751 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7fa7000b2880 0x7fa7000b4d40 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:18:56.853 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:56.850+0000 7fa71edda700 1 --2- 172.21.15.67:0/2896894751 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fa718108510 0x7fa718199a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:18:56.853 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:56.850+0000 7fa71edda700 1 --2- 172.21.15.67:0/2896894751 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fa7181026c0 0x7fa71819f880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:18:56.854 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:56.850+0000 7fa71edda700 1 --2- 172.21.15.67:0/2896894751 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa718103080 0x7fa71819fdc0 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:18:56.854 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:56.850+0000 7fa71edda700 1 -- 172.21.15.67:0/2896894751 >> 172.21.15.67:0/2896894751 conn(0x7fa718079100 msgr2=0x7fa718101860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T21:18:56.854 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:56.851+0000 7fa71edda700 1 -- 172.21.15.67:0/2896894751 shutdown_connections 2024-05-03T21:18:56.854 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:56.851+0000 7fa71edda700 1 -- 172.21.15.67:0/2896894751 wait complete. 2024-05-03T21:18:56.971 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:18:56 smithi067 ceph-mon[29730]: pgmap v293: 97 pgs: 97 active+clean; 579 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 KiB/s wr, 4 op/s 2024-05-03T21:18:56.971 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:18:56 smithi067 ceph-mon[29730]: from='client.? 172.21.15.67:0/2896894751' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]: dispatch 2024-05-03T21:18:57.246 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:18:56 smithi082 ceph-mon[32933]: pgmap v293: 97 pgs: 97 active+clean; 579 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 KiB/s wr, 4 op/s 2024-05-03T21:18:57.246 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:18:56 smithi082 ceph-mon[32933]: from='client.? 172.21.15.67:0/2896894751' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]: dispatch 2024-05-03T21:18:57.325 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:18:56 smithi138 ceph-mon[33110]: pgmap v293: 97 pgs: 97 active+clean; 579 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 KiB/s wr, 4 op/s 2024-05-03T21:18:57.325 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:18:56 smithi138 ceph-mon[33110]: from='client.? 172.21.15.67:0/2896894751' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]: dispatch 2024-05-03T21:18:57.559 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9a047fe7599a549042d06a3fb5b6813a203eabea shell --fsid 713bc57a-0991-11ef-bc97-c7b262605968 -- ceph fs set cephfs allow_standby_replay true 2024-05-03T21:18:57.818 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/713bc57a-0991-11ef-bc97-c7b262605968/mon.a/config 2024-05-03T21:18:58.115 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:18:57 smithi067 ceph-mon[29730]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-05-03T21:18:58.115 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:18:57 smithi067 ceph-mon[29730]: from='client.? 172.21.15.67:0/2896894751' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]': finished 2024-05-03T21:18:58.116 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:18:57 smithi067 ceph-mon[29730]: daemon mds.h assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-05-03T21:18:58.116 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:18:57 smithi067 ceph-mon[29730]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-05-03T21:18:58.116 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:18:57 smithi067 ceph-mon[29730]: fsmap cephfs:2 {0=b=up:active,1=h=up:creating} 10 up:standby 2024-05-03T21:18:58.116 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:18:57 smithi067 ceph-mon[29730]: daemon mds.h is now active in filesystem cephfs as rank 1 2024-05-03T21:18:58.245 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:18:57 smithi082 ceph-mon[32933]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-05-03T21:18:58.245 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:18:57 smithi082 ceph-mon[32933]: from='client.? 172.21.15.67:0/2896894751' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]': finished 2024-05-03T21:18:58.245 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:18:57 smithi082 ceph-mon[32933]: daemon mds.h assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-05-03T21:18:58.246 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:18:57 smithi082 ceph-mon[32933]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-05-03T21:18:58.246 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:18:57 smithi082 ceph-mon[32933]: fsmap cephfs:2 {0=b=up:active,1=h=up:creating} 10 up:standby 2024-05-03T21:18:58.246 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:18:57 smithi082 ceph-mon[32933]: daemon mds.h is now active in filesystem cephfs as rank 1 2024-05-03T21:18:58.324 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:18:57 smithi138 ceph-mon[33110]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-05-03T21:18:58.324 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:18:57 smithi138 ceph-mon[33110]: from='client.? 172.21.15.67:0/2896894751' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]': finished 2024-05-03T21:18:58.325 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:18:57 smithi138 ceph-mon[33110]: daemon mds.h assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-05-03T21:18:58.325 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:18:57 smithi138 ceph-mon[33110]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-05-03T21:18:58.325 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:18:57 smithi138 ceph-mon[33110]: fsmap cephfs:2 {0=b=up:active,1=h=up:creating} 10 up:standby 2024-05-03T21:18:58.325 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:18:57 smithi138 ceph-mon[33110]: daemon mds.h is now active in filesystem cephfs as rank 1 2024-05-03T21:18:58.971 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:18:58 smithi067 ceph-mon[29730]: pgmap v294: 97 pgs: 97 active+clean; 580 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s wr, 4 op/s 2024-05-03T21:18:58.971 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:18:58 smithi067 ceph-mon[29730]: mds.? [v2:172.21.15.82:6836/1275987852,v1:172.21.15.82:6837/1275987852] up:active 2024-05-03T21:18:58.971 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:18:58 smithi067 ceph-mon[29730]: daemon mds.e assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2024-05-03T21:18:58.971 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:18:58 smithi067 ceph-mon[29730]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-05-03T21:18:58.971 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:18:58 smithi067 ceph-mon[29730]: Cluster is now healthy 2024-05-03T21:18:58.971 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:18:58 smithi067 ceph-mon[29730]: fsmap cephfs:2 {0=b=up:active,1=h=up:active} 10 up:standby 2024-05-03T21:18:58.972 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:18:58 smithi067 ceph-mon[29730]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:creating} 9 up:standby 2024-05-03T21:18:58.972 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:18:58 smithi067 ceph-mon[29730]: daemon mds.e is now active in filesystem cephfs as rank 2 2024-05-03T21:18:59.245 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:18:58 smithi082 ceph-mon[32933]: pgmap v294: 97 pgs: 97 active+clean; 580 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s wr, 4 op/s 2024-05-03T21:18:59.245 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:18:58 smithi082 ceph-mon[32933]: mds.? [v2:172.21.15.82:6836/1275987852,v1:172.21.15.82:6837/1275987852] up:active 2024-05-03T21:18:59.245 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:18:58 smithi082 ceph-mon[32933]: daemon mds.e assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2024-05-03T21:18:59.246 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:18:58 smithi082 ceph-mon[32933]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-05-03T21:18:59.246 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:18:58 smithi082 ceph-mon[32933]: Cluster is now healthy 2024-05-03T21:18:59.246 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:18:58 smithi082 ceph-mon[32933]: fsmap cephfs:2 {0=b=up:active,1=h=up:active} 10 up:standby 2024-05-03T21:18:59.246 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:18:58 smithi082 ceph-mon[32933]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:creating} 9 up:standby 2024-05-03T21:18:59.246 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:18:58 smithi082 ceph-mon[32933]: daemon mds.e is now active in filesystem cephfs as rank 2 2024-05-03T21:18:59.325 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:18:58 smithi138 ceph-mon[33110]: pgmap v294: 97 pgs: 97 active+clean; 580 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s wr, 4 op/s 2024-05-03T21:18:59.325 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:18:58 smithi138 ceph-mon[33110]: mds.? [v2:172.21.15.82:6836/1275987852,v1:172.21.15.82:6837/1275987852] up:active 2024-05-03T21:18:59.325 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:18:58 smithi138 ceph-mon[33110]: daemon mds.e assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2024-05-03T21:18:59.325 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:18:58 smithi138 ceph-mon[33110]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-05-03T21:18:59.325 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:18:58 smithi138 ceph-mon[33110]: Cluster is now healthy 2024-05-03T21:18:59.325 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:18:58 smithi138 ceph-mon[33110]: fsmap cephfs:2 {0=b=up:active,1=h=up:active} 10 up:standby 2024-05-03T21:18:59.325 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:18:58 smithi138 ceph-mon[33110]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:creating} 9 up:standby 2024-05-03T21:18:59.325 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:18:58 smithi138 ceph-mon[33110]: daemon mds.e is now active in filesystem cephfs as rank 2 2024-05-03T21:18:59.600 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:59.598+0000 7f0980d65700 1 -- 172.21.15.67:0/2700628427 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f097c102820 msgr2=0x7f097c110290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:18:59.600 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:59.598+0000 7f0980d65700 1 --2- 172.21.15.67:0/2700628427 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f097c102820 0x7f097c110290 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f097400b0a0 tx=0x7f097400b3b0 comp rx=0 tx=0).stop 2024-05-03T21:18:59.600 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:59.598+0000 7f0980d65700 1 -- 172.21.15.67:0/2700628427 shutdown_connections 2024-05-03T21:18:59.600 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:59.598+0000 7f0980d65700 1 --2- 172.21.15.67:0/2700628427 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f097c102820 0x7f097c110290 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:18:59.601 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:59.598+0000 7f0980d65700 1 --2- 172.21.15.67:0/2700628427 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f097c108580 0x7f097c108960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:18:59.601 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:59.598+0000 7f0980d65700 1 --2- 172.21.15.67:0/2700628427 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f097c101e60 0x7f097c1022e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:18:59.601 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:59.598+0000 7f0980d65700 1 -- 172.21.15.67:0/2700628427 >> 172.21.15.67:0/2700628427 conn(0x7f097c0776c0 msgr2=0x7f097c077ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T21:18:59.601 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:59.598+0000 7f0980d65700 1 -- 172.21.15.67:0/2700628427 shutdown_connections 2024-05-03T21:18:59.601 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:59.598+0000 7f0980d65700 1 -- 172.21.15.67:0/2700628427 wait complete. 2024-05-03T21:18:59.603 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:59.598+0000 7f0980d65700 1 Processor -- start 2024-05-03T21:18:59.603 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:59.599+0000 7f0980d65700 1 -- start start 2024-05-03T21:18:59.603 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:59.599+0000 7f0980d65700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f097c101e60 0x7f097c19f920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:18:59.603 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:59.599+0000 7f0980d65700 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f097c102820 0x7f097c19fe60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:18:59.603 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:59.599+0000 7f0980d65700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f097c108580 0x7f097c199ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:18:59.603 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:59.599+0000 7f0980d65700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f097c113920 con 0x7f097c108580 2024-05-03T21:18:59.603 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:59.599+0000 7f0980d65700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f097c1137a0 con 0x7f097c101e60 2024-05-03T21:18:59.603 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:59.599+0000 7f0980d65700 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f097c113aa0 con 0x7f097c102820 2024-05-03T21:18:59.603 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:59.599+0000 7f097ad9d700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f097c108580 0x7f097c199ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:18:59.603 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:59.599+0000 7f097ad9d700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f097c108580 0x7f097c199ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:42326/0 (socket says 172.21.15.67:42326) 2024-05-03T21:18:59.603 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:59.599+0000 7f097ad9d700 1 -- 172.21.15.67:0/2483237456 learned_addr learned my addr 172.21.15.67:0/2483237456 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-03T21:18:59.604 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:59.599+0000 7f097a59c700 1 --2- 172.21.15.67:0/2483237456 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f097c101e60 0x7f097c19f920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:18:59.604 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:59.599+0000 7f0973fff700 1 --2- 172.21.15.67:0/2483237456 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f097c102820 0x7f097c19fe60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:18:59.604 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:59.599+0000 7f097ad9d700 1 -- 172.21.15.67:0/2483237456 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f097c102820 msgr2=0x7f097c19fe60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:18:59.604 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:59.599+0000 7f097ad9d700 1 --2- 172.21.15.67:0/2483237456 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f097c102820 0x7f097c19fe60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:18:59.604 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:59.599+0000 7f097ad9d700 1 -- 172.21.15.67:0/2483237456 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f097c101e60 msgr2=0x7f097c19f920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:18:59.604 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:59.599+0000 7f097ad9d700 1 --2- 172.21.15.67:0/2483237456 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f097c101e60 0x7f097c19f920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:18:59.604 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:59.599+0000 7f097ad9d700 1 -- 172.21.15.67:0/2483237456 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f097c19a240 con 0x7f097c108580 2024-05-03T21:18:59.604 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:59.599+0000 7f0973fff700 1 --2- 172.21.15.67:0/2483237456 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f097c102820 0x7f097c19fe60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-03T21:18:59.604 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:59.599+0000 7f097a59c700 1 --2- 172.21.15.67:0/2483237456 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f097c101e60 0x7f097c19f920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-03T21:18:59.604 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:59.599+0000 7f097ad9d700 1 --2- 172.21.15.67:0/2483237456 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f097c108580 0x7f097c199ae0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f097400b070 tx=0x7f0974004d00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T21:18:59.604 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:59.600+0000 7f09737fe700 1 -- 172.21.15.67:0/2483237456 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0974021070 con 0x7f097c108580 2024-05-03T21:18:59.604 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:59.600+0000 7f09737fe700 1 -- 172.21.15.67:0/2483237456 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(28 keys) ==== 1170+0+0 (secure 0 0 0) 0x7f0974007b80 con 0x7f097c108580 2024-05-03T21:18:59.604 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:59.600+0000 7f09737fe700 1 -- 172.21.15.67:0/2483237456 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09740053c0 con 0x7f097c108580 2024-05-03T21:18:59.604 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:59.600+0000 7f0980d65700 1 -- 172.21.15.67:0/2483237456 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f097c19a4d0 con 0x7f097c108580 2024-05-03T21:18:59.605 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:59.600+0000 7f0980d65700 1 -- 172.21.15.67:0/2483237456 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f097c1a67f0 con 0x7f097c108580 2024-05-03T21:18:59.605 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:59.601+0000 7f0980d65700 1 -- 172.21.15.67:0/2483237456 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f097c06b670 con 0x7f097c108580 2024-05-03T21:18:59.605 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:59.602+0000 7f09737fe700 1 -- 172.21.15.67:0/2483237456 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 16) ==== 149652+0+0 (secure 0 0 0) 0x7f0974024070 con 0x7f097c108580 2024-05-03T21:18:59.605 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:59.603+0000 7f09737fe700 1 --2- 172.21.15.67:0/2483237456 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f09680b2880 0x7f09680b4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:18:59.606 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:59.603+0000 7f097a59c700 1 --2- 172.21.15.67:0/2483237456 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f09680b2880 0x7f09680b4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:18:59.606 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:59.603+0000 7f09737fe700 1 -- 172.21.15.67:0/2483237456 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7979+0+0 (secure 0 0 0) 0x7f09740e3150 con 0x7f097c108580 2024-05-03T21:18:59.606 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:59.603+0000 7f097a59c700 1 --2- 172.21.15.67:0/2483237456 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f09680b2880 0x7f09680b4d40 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f097c078c40 tx=0x7f096c00b410 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T21:18:59.608 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:59.607+0000 7f09737fe700 1 -- 172.21.15.67:0/2483237456 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191839 (secure 0 0 0) 0x7f09740ab050 con 0x7f097c108580 2024-05-03T21:18:59.876 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:18:59.873+0000 7f0980d65700 1 -- 172.21.15.67:0/2483237456 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"} v 0) -- 0x7f097c1a69c0 con 0x7f097c108580 2024-05-03T21:19:00.221 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:18:59 smithi067 ceph-mon[29730]: mds.? [v2:172.21.15.82:6834/280518761,v1:172.21.15.82:6835/280518761] up:active 2024-05-03T21:19:00.221 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:18:59 smithi067 ceph-mon[29730]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 9 up:standby 2024-05-03T21:19:00.245 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:18:59 smithi082 ceph-mon[32933]: mds.? [v2:172.21.15.82:6834/280518761,v1:172.21.15.82:6835/280518761] up:active 2024-05-03T21:19:00.245 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:18:59 smithi082 ceph-mon[32933]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 9 up:standby 2024-05-03T21:19:00.324 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:18:59 smithi138 ceph-mon[33110]: mds.? [v2:172.21.15.82:6834/280518761,v1:172.21.15.82:6835/280518761] up:active 2024-05-03T21:19:00.324 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:18:59 smithi138 ceph-mon[33110]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 9 up:standby 2024-05-03T21:19:00.880 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:00.878+0000 7f09737fe700 1 -- 172.21.15.67:0/2483237456 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]=0 v23) ==== 121+0+0 (secure 0 0 0) 0x7f09740b3040 con 0x7f097c108580 2024-05-03T21:19:00.884 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:00.881+0000 7f0980d65700 1 -- 172.21.15.67:0/2483237456 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f09680b2880 msgr2=0x7f09680b4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:19:00.884 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:00.881+0000 7f0980d65700 1 --2- 172.21.15.67:0/2483237456 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f09680b2880 0x7f09680b4d40 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f097c078c40 tx=0x7f096c00b410 comp rx=0 tx=0).stop 2024-05-03T21:19:00.884 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:00.881+0000 7f0980d65700 1 -- 172.21.15.67:0/2483237456 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f097c108580 msgr2=0x7f097c199ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:19:00.884 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:00.881+0000 7f0980d65700 1 --2- 172.21.15.67:0/2483237456 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f097c108580 0x7f097c199ae0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f097400b070 tx=0x7f0974004d00 comp rx=0 tx=0).stop 2024-05-03T21:19:00.884 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:00.881+0000 7f0980d65700 1 -- 172.21.15.67:0/2483237456 shutdown_connections 2024-05-03T21:19:00.884 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:00.881+0000 7f0980d65700 1 --2- 172.21.15.67:0/2483237456 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f09680b2880 0x7f09680b4d40 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:19:00.884 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:00.881+0000 7f0980d65700 1 --2- 172.21.15.67:0/2483237456 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f097c108580 0x7f097c199ae0 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:19:00.884 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:00.881+0000 7f0980d65700 1 --2- 172.21.15.67:0/2483237456 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f097c101e60 0x7f097c19f920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:19:00.884 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:00.881+0000 7f0980d65700 1 --2- 172.21.15.67:0/2483237456 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f097c102820 0x7f097c19fe60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:19:00.884 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:00.881+0000 7f0980d65700 1 -- 172.21.15.67:0/2483237456 >> 172.21.15.67:0/2483237456 conn(0x7f097c0776c0 msgr2=0x7f097c100860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T21:19:00.884 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:00.881+0000 7f0980d65700 1 -- 172.21.15.67:0/2483237456 shutdown_connections 2024-05-03T21:19:00.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:00.882+0000 7f0980d65700 1 -- 172.21.15.67:0/2483237456 wait complete. 2024-05-03T21:19:01.221 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:19:00 smithi067 ceph-mon[29730]: pgmap v295: 97 pgs: 97 active+clean; 581 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s wr, 2 op/s 2024-05-03T21:19:01.221 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:19:00 smithi067 ceph-mon[29730]: from='client.? 172.21.15.67:0/2483237456' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-05-03T21:19:01.245 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:19:00 smithi082 ceph-mon[32933]: pgmap v295: 97 pgs: 97 active+clean; 581 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s wr, 2 op/s 2024-05-03T21:19:01.245 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:19:00 smithi082 ceph-mon[32933]: from='client.? 172.21.15.67:0/2483237456' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-05-03T21:19:01.324 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:19:00 smithi138 ceph-mon[33110]: pgmap v295: 97 pgs: 97 active+clean; 581 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s wr, 2 op/s 2024-05-03T21:19:01.324 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:19:00 smithi138 ceph-mon[33110]: from='client.? 172.21.15.67:0/2483237456' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-05-03T21:19:01.622 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9a047fe7599a549042d06a3fb5b6813a203eabea shell --fsid 713bc57a-0991-11ef-bc97-c7b262605968 -- ceph fs set cephfs session_timeout 300 2024-05-03T21:19:01.879 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/713bc57a-0991-11ef-bc97-c7b262605968/mon.a/config 2024-05-03T21:19:02.200 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:19:01 smithi067 ceph-mon[29730]: from='client.? 172.21.15.67:0/2483237456' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-05-03T21:19:02.200 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:19:01 smithi067 ceph-mon[29730]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 9 up:standby 2024-05-03T21:19:02.200 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:19:01 smithi067 ceph-mon[29730]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 3 up:standby-replay 6 up:standby 2024-05-03T21:19:02.245 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:19:01 smithi082 ceph-mon[32933]: from='client.? 172.21.15.67:0/2483237456' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-05-03T21:19:02.245 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:19:01 smithi082 ceph-mon[32933]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 9 up:standby 2024-05-03T21:19:02.245 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:19:01 smithi082 ceph-mon[32933]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 3 up:standby-replay 6 up:standby 2024-05-03T21:19:02.324 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:19:01 smithi138 ceph-mon[33110]: from='client.? 172.21.15.67:0/2483237456' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-05-03T21:19:02.324 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:19:01 smithi138 ceph-mon[33110]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 9 up:standby 2024-05-03T21:19:02.324 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:19:01 smithi138 ceph-mon[33110]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 3 up:standby-replay 6 up:standby 2024-05-03T21:19:02.971 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:19:02 smithi067 ceph-mon[29730]: pgmap v296: 97 pgs: 97 active+clean; 581 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 511 B/s rd, 1.7 KiB/s wr, 6 op/s 2024-05-03T21:19:03.245 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:19:02 smithi082 ceph-mon[32933]: pgmap v296: 97 pgs: 97 active+clean; 581 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 511 B/s rd, 1.7 KiB/s wr, 6 op/s 2024-05-03T21:19:03.324 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:19:02 smithi138 ceph-mon[33110]: pgmap v296: 97 pgs: 97 active+clean; 581 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 511 B/s rd, 1.7 KiB/s wr, 6 op/s 2024-05-03T21:19:03.501 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:03.497+0000 7f8f46e70700 1 -- 172.21.15.67:0/4104845082 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8f4006bb40 msgr2=0x7f8f40110330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:19:03.501 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:03.497+0000 7f8f46e70700 1 --2- 172.21.15.67:0/4104845082 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8f4006bb40 0x7f8f40110330 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f8f3800b0a0 tx=0x7f8f3800b3b0 comp rx=0 tx=0).stop 2024-05-03T21:19:03.501 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:03.497+0000 7f8f46e70700 1 -- 172.21.15.67:0/4104845082 shutdown_connections 2024-05-03T21:19:03.501 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:03.497+0000 7f8f46e70700 1 --2- 172.21.15.67:0/4104845082 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8f4006bb40 0x7f8f40110330 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:19:03.501 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:03.497+0000 7f8f46e70700 1 --2- 172.21.15.67:0/4104845082 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f8f40109510 0x7f8f401098f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:19:03.502 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:03.497+0000 7f8f46e70700 1 --2- 172.21.15.67:0/4104845082 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f8f4006b180 0x7f8f4006b600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:19:03.502 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:03.497+0000 7f8f46e70700 1 -- 172.21.15.67:0/4104845082 >> 172.21.15.67:0/4104845082 conn(0x7f8f40079350 msgr2=0x7f8f40079760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T21:19:03.502 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:03.497+0000 7f8f46e70700 1 -- 172.21.15.67:0/4104845082 shutdown_connections 2024-05-03T21:19:03.502 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:03.497+0000 7f8f46e70700 1 -- 172.21.15.67:0/4104845082 wait complete. 2024-05-03T21:19:03.502 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:03.497+0000 7f8f46e70700 1 Processor -- start 2024-05-03T21:19:03.502 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:03.497+0000 7f8f46e70700 1 -- start start 2024-05-03T21:19:03.502 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:03.498+0000 7f8f46e70700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f8f4006b180 0x7f8f40102470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:19:03.502 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:03.498+0000 7f8f46e70700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8f4006bb40 0x7f8f401029b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:19:03.502 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:03.498+0000 7f8f46e70700 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f8f40109510 0x7f8f40102ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:19:03.502 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:03.498+0000 7f8f46e70700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f8f401138b0 con 0x7f8f4006bb40 2024-05-03T21:19:03.502 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:03.498+0000 7f8f46e70700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f8f40113730 con 0x7f8f4006b180 2024-05-03T21:19:03.502 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:03.498+0000 7f8f46e70700 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f8f40113a30 con 0x7f8f40109510 2024-05-03T21:19:03.502 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:03.498+0000 7f8f3ffff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8f4006bb40 0x7f8f401029b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:19:03.503 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:03.498+0000 7f8f3ffff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8f4006bb40 0x7f8f401029b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:42358/0 (socket says 172.21.15.67:42358) 2024-05-03T21:19:03.503 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:03.498+0000 7f8f3ffff700 1 -- 172.21.15.67:0/1456087817 learned_addr learned my addr 172.21.15.67:0/1456087817 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-03T21:19:03.503 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:03.498+0000 7f8f4540d700 1 --2- 172.21.15.67:0/1456087817 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f8f40109510 0x7f8f40102ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:19:03.503 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:03.498+0000 7f8f44c0c700 1 --2- 172.21.15.67:0/1456087817 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f8f4006b180 0x7f8f40102470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:19:03.503 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:03.498+0000 7f8f3ffff700 1 -- 172.21.15.67:0/1456087817 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f8f40109510 msgr2=0x7f8f40102ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:19:03.503 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:03.498+0000 7f8f3ffff700 1 --2- 172.21.15.67:0/1456087817 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f8f40109510 0x7f8f40102ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:19:03.503 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:03.498+0000 7f8f3ffff700 1 -- 172.21.15.67:0/1456087817 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f8f4006b180 msgr2=0x7f8f40102470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:19:03.503 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:03.498+0000 7f8f3ffff700 1 --2- 172.21.15.67:0/1456087817 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f8f4006b180 0x7f8f40102470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:19:03.503 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:03.498+0000 7f8f3ffff700 1 -- 172.21.15.67:0/1456087817 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8f401a81b0 con 0x7f8f4006bb40 2024-05-03T21:19:03.503 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:03.498+0000 7f8f44c0c700 1 --2- 172.21.15.67:0/1456087817 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f8f4006b180 0x7f8f40102470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-03T21:19:03.503 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:03.498+0000 7f8f3ffff700 1 --2- 172.21.15.67:0/1456087817 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8f4006bb40 0x7f8f401029b0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f8f3400c920 tx=0x7f8f3400cc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T21:19:03.503 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:03.499+0000 7f8f3dffb700 1 -- 172.21.15.67:0/1456087817 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f34007b00 con 0x7f8f4006bb40 2024-05-03T21:19:03.503 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:03.499+0000 7f8f46e70700 1 -- 172.21.15.67:0/1456087817 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8f401a84a0 con 0x7f8f4006bb40 2024-05-03T21:19:03.503 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:03.499+0000 7f8f3dffb700 1 -- 172.21.15.67:0/1456087817 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(28 keys) ==== 1170+0+0 (secure 0 0 0) 0x7f8f34004d30 con 0x7f8f4006bb40 2024-05-03T21:19:03.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:03.499+0000 7f8f3dffb700 1 -- 172.21.15.67:0/1456087817 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f34005630 con 0x7f8f4006bb40 2024-05-03T21:19:03.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:03.499+0000 7f8f46e70700 1 -- 172.21.15.67:0/1456087817 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8f401a8a30 con 0x7f8f4006bb40 2024-05-03T21:19:03.505 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:03.500+0000 7f8f46e70700 1 -- 172.21.15.67:0/1456087817 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8f4006a190 con 0x7f8f4006bb40 2024-05-03T21:19:03.505 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:03.502+0000 7f8f3dffb700 1 -- 172.21.15.67:0/1456087817 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 16) ==== 149652+0+0 (secure 0 0 0) 0x7f8f34004750 con 0x7f8f4006bb40 2024-05-03T21:19:03.505 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:03.503+0000 7f8f3dffb700 1 --2- 172.21.15.67:0/1456087817 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f8f280b2880 0x7f8f280b4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:19:03.505 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:03.503+0000 7f8f44c0c700 1 --2- 172.21.15.67:0/1456087817 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f8f280b2880 0x7f8f280b4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:19:03.505 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:03.503+0000 7f8f3dffb700 1 -- 172.21.15.67:0/1456087817 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7979+0+0 (secure 0 0 0) 0x7f8f340e2070 con 0x7f8f4006bb40 2024-05-03T21:19:03.505 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:03.503+0000 7f8f44c0c700 1 --2- 172.21.15.67:0/1456087817 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f8f280b2880 0x7f8f280b4d40 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f8f30005e10 tx=0x7f8f30005d60 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T21:19:03.508 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:03.506+0000 7f8f3dffb700 1 -- 172.21.15.67:0/1456087817 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191839 (secure 0 0 0) 0x7f8f34014030 con 0x7f8f4006bb40 2024-05-03T21:19:03.776 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:03.774+0000 7f8f46e70700 1 -- 172.21.15.67:0/1456087817 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"} v 0) -- 0x7f8f401039a0 con 0x7f8f4006bb40 2024-05-03T21:19:03.897 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:03.895+0000 7f8f3dffb700 1 -- 172.21.15.67:0/1456087817 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]=0 v25) ==== 115+0+0 (secure 0 0 0) 0x7f8f340a9ee0 con 0x7f8f4006bb40 2024-05-03T21:19:03.900 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:03.897+0000 7f8f46e70700 1 -- 172.21.15.67:0/1456087817 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f8f280b2880 msgr2=0x7f8f280b4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:19:03.900 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:03.897+0000 7f8f46e70700 1 --2- 172.21.15.67:0/1456087817 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f8f280b2880 0x7f8f280b4d40 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f8f30005e10 tx=0x7f8f30005d60 comp rx=0 tx=0).stop 2024-05-03T21:19:03.900 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:03.897+0000 7f8f46e70700 1 -- 172.21.15.67:0/1456087817 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8f4006bb40 msgr2=0x7f8f401029b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:19:03.900 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:03.897+0000 7f8f46e70700 1 --2- 172.21.15.67:0/1456087817 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8f4006bb40 0x7f8f401029b0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f8f3400c920 tx=0x7f8f3400cc30 comp rx=0 tx=0).stop 2024-05-03T21:19:03.900 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:03.897+0000 7f8f46e70700 1 -- 172.21.15.67:0/1456087817 shutdown_connections 2024-05-03T21:19:03.900 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:03.897+0000 7f8f46e70700 1 --2- 172.21.15.67:0/1456087817 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f8f280b2880 0x7f8f280b4d40 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:19:03.900 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:03.897+0000 7f8f46e70700 1 --2- 172.21.15.67:0/1456087817 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f8f40109510 0x7f8f40102ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:19:03.900 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:03.897+0000 7f8f46e70700 1 --2- 172.21.15.67:0/1456087817 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f8f4006b180 0x7f8f40102470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:19:03.900 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:03.897+0000 7f8f46e70700 1 --2- 172.21.15.67:0/1456087817 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8f4006bb40 0x7f8f401029b0 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:19:03.900 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:03.897+0000 7f8f46e70700 1 -- 172.21.15.67:0/1456087817 >> 172.21.15.67:0/1456087817 conn(0x7f8f40079350 msgr2=0x7f8f40100e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T21:19:03.900 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:03.898+0000 7f8f46e70700 1 -- 172.21.15.67:0/1456087817 shutdown_connections 2024-05-03T21:19:03.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:03.898+0000 7f8f46e70700 1 -- 172.21.15.67:0/1456087817 wait complete. 2024-05-03T21:19:04.209 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:19:03 smithi067 ceph-mon[29730]: from='client.? 172.21.15.67:0/1456087817' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-05-03T21:19:04.245 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:19:03 smithi082 ceph-mon[32933]: from='client.? 172.21.15.67:0/1456087817' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-05-03T21:19:04.324 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:19:03 smithi138 ceph-mon[33110]: from='client.? 172.21.15.67:0/1456087817' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-05-03T21:19:04.515 DEBUG:tasks.cephfs.filesystem:Creating {'create': 2} subvols for filesystem 'cephfs' 2024-05-03T21:19:04.515 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9a047fe7599a549042d06a3fb5b6813a203eabea shell --fsid 713bc57a-0991-11ef-bc97-c7b262605968 -- ceph fs subvolumegroup create cephfs qa 2024-05-03T21:19:04.764 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/713bc57a-0991-11ef-bc97-c7b262605968/mon.a/config 2024-05-03T21:19:05.221 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:19:04 smithi067 ceph-mon[29730]: pgmap v297: 97 pgs: 97 active+clean; 582 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 2.0 KiB/s wr, 8 op/s 2024-05-03T21:19:05.221 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:19:04 smithi067 ceph-mon[29730]: from='client.? 172.21.15.67:0/1456087817' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2024-05-03T21:19:05.221 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:19:04 smithi067 ceph-mon[29730]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 3 up:standby-replay 6 up:standby 2024-05-03T21:19:05.245 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:19:04 smithi082 ceph-mon[32933]: pgmap v297: 97 pgs: 97 active+clean; 582 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 2.0 KiB/s wr, 8 op/s 2024-05-03T21:19:05.245 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:19:04 smithi082 ceph-mon[32933]: from='client.? 172.21.15.67:0/1456087817' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2024-05-03T21:19:05.246 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:19:04 smithi082 ceph-mon[32933]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 3 up:standby-replay 6 up:standby 2024-05-03T21:19:05.324 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:19:04 smithi138 ceph-mon[33110]: pgmap v297: 97 pgs: 97 active+clean; 582 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 2.0 KiB/s wr, 8 op/s 2024-05-03T21:19:05.324 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:19:04 smithi138 ceph-mon[33110]: from='client.? 172.21.15.67:0/1456087817' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2024-05-03T21:19:05.324 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:19:04 smithi138 ceph-mon[33110]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 3 up:standby-replay 6 up:standby 2024-05-03T21:19:06.352 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:06.349+0000 7fdf65f53700 1 -- 172.21.15.67:0/3680647492 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdf60102630 msgr2=0x7fdf60102a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:19:06.352 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:06.349+0000 7fdf65f53700 1 --2- 172.21.15.67:0/3680647492 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdf60102630 0x7fdf60102a10 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fdf500099c0 tx=0x7fdf50009cd0 comp rx=0 tx=0).stop 2024-05-03T21:19:06.352 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:06.349+0000 7fdf65f53700 1 -- 172.21.15.67:0/3680647492 shutdown_connections 2024-05-03T21:19:06.352 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:06.349+0000 7fdf65f53700 1 --2- 172.21.15.67:0/3680647492 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdf60102630 0x7fdf60102a10 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:19:06.352 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:06.349+0000 7fdf65f53700 1 --2- 172.21.15.67:0/3680647492 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fdf601076c0 0x7fdf6010f290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:19:06.353 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:06.349+0000 7fdf65f53700 1 --2- 172.21.15.67:0/3680647492 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fdf60102fe0 0x7fdf60106fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:19:06.353 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:06.349+0000 7fdf65f53700 1 -- 172.21.15.67:0/3680647492 >> 172.21.15.67:0/3680647492 conn(0x7fdf60079350 msgr2=0x7fdf60079760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T21:19:06.353 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:06.349+0000 7fdf65f53700 1 -- 172.21.15.67:0/3680647492 shutdown_connections 2024-05-03T21:19:06.353 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:06.350+0000 7fdf65f53700 1 -- 172.21.15.67:0/3680647492 wait complete. 2024-05-03T21:19:06.353 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:06.350+0000 7fdf65f53700 1 Processor -- start 2024-05-03T21:19:06.354 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:06.350+0000 7fdf65f53700 1 -- start start 2024-05-03T21:19:06.354 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:06.351+0000 7fdf65f53700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fdf60102630 0x7fdf6019fa60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:19:06.354 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:06.351+0000 7fdf65f53700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdf60102fe0 0x7fdf6019ffa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:19:06.355 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:06.351+0000 7fdf65f53700 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fdf601076c0 0x7fdf60199b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:19:06.355 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:06.351+0000 7fdf65f53700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fdf60112860 con 0x7fdf60102fe0 2024-05-03T21:19:06.355 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:06.351+0000 7fdf65f53700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7fdf601126e0 con 0x7fdf60102630 2024-05-03T21:19:06.355 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:06.351+0000 7fdf65f53700 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7fdf601129e0 con 0x7fdf601076c0 2024-05-03T21:19:06.355 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:06.351+0000 7fdf5effd700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdf60102fe0 0x7fdf6019ffa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:19:06.355 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:06.351+0000 7fdf5effd700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdf60102fe0 0x7fdf6019ffa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:50434/0 (socket says 172.21.15.67:50434) 2024-05-03T21:19:06.355 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:06.351+0000 7fdf5effd700 1 -- 172.21.15.67:0/3891637691 learned_addr learned my addr 172.21.15.67:0/3891637691 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-03T21:19:06.355 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:06.351+0000 7fdf5f7fe700 1 --2- 172.21.15.67:0/3891637691 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fdf60102630 0x7fdf6019fa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:19:06.355 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:06.351+0000 7fdf5ffff700 1 --2- 172.21.15.67:0/3891637691 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fdf601076c0 0x7fdf60199b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:19:06.355 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:06.351+0000 7fdf5effd700 1 -- 172.21.15.67:0/3891637691 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fdf601076c0 msgr2=0x7fdf60199b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:19:06.355 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:06.351+0000 7fdf5effd700 1 --2- 172.21.15.67:0/3891637691 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fdf601076c0 0x7fdf60199b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:19:06.355 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:06.351+0000 7fdf5effd700 1 -- 172.21.15.67:0/3891637691 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fdf60102630 msgr2=0x7fdf6019fa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:19:06.355 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:06.351+0000 7fdf5effd700 1 --2- 172.21.15.67:0/3891637691 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fdf60102630 0x7fdf6019fa60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:19:06.355 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:06.351+0000 7fdf5effd700 1 -- 172.21.15.67:0/3891637691 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdf6019a410 con 0x7fdf60102fe0 2024-05-03T21:19:06.356 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:06.351+0000 7fdf5effd700 1 --2- 172.21.15.67:0/3891637691 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdf60102fe0 0x7fdf6019ffa0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fdf5400b500 tx=0x7fdf5400b810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T21:19:06.356 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:06.352+0000 7fdf5cff9700 1 -- 172.21.15.67:0/3891637691 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf54015020 con 0x7fdf60102fe0 2024-05-03T21:19:06.356 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:06.352+0000 7fdf65f53700 1 -- 172.21.15.67:0/3891637691 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdf6019a700 con 0x7fdf60102fe0 2024-05-03T21:19:06.356 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:06.352+0000 7fdf5cff9700 1 -- 172.21.15.67:0/3891637691 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(28 keys) ==== 1170+0+0 (secure 0 0 0) 0x7fdf54011d30 con 0x7fdf60102fe0 2024-05-03T21:19:06.356 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:06.352+0000 7fdf5cff9700 1 -- 172.21.15.67:0/3891637691 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf54010de0 con 0x7fdf60102fe0 2024-05-03T21:19:06.356 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:06.352+0000 7fdf65f53700 1 -- 172.21.15.67:0/3891637691 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdf601a69e0 con 0x7fdf60102fe0 2024-05-03T21:19:06.356 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:06.353+0000 7fdf65f53700 1 -- 172.21.15.67:0/3891637691 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdf60103ac0 con 0x7fdf60102fe0 2024-05-03T21:19:06.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:06.355+0000 7fdf5cff9700 1 -- 172.21.15.67:0/3891637691 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 16) ==== 149652+0+0 (secure 0 0 0) 0x7fdf54018020 con 0x7fdf60102fe0 2024-05-03T21:19:06.358 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:06.355+0000 7fdf5cff9700 1 --2- 172.21.15.67:0/3891637691 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7fdf480b2880 0x7fdf480b4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:19:06.358 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:06.356+0000 7fdf5f7fe700 1 --2- 172.21.15.67:0/3891637691 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7fdf480b2880 0x7fdf480b4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:19:06.358 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:06.356+0000 7fdf5cff9700 1 -- 172.21.15.67:0/3891637691 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7979+0+0 (secure 0 0 0) 0x7fdf540e1c80 con 0x7fdf60102fe0 2024-05-03T21:19:06.358 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:06.356+0000 7fdf5f7fe700 1 --2- 172.21.15.67:0/3891637691 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7fdf480b2880 0x7fdf480b4d40 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fdf50000c00 tx=0x7fdf50005fb0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T21:19:06.360 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:06.358+0000 7fdf5cff9700 1 -- 172.21.15.67:0/3891637691 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191839 (secure 0 0 0) 0x7fdf5401f020 con 0x7fdf60102fe0 2024-05-03T21:19:06.611 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:06.609+0000 7fdf65f53700 1 -- 172.21.15.67:0/3891637691 --> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] -- mgr_command(tid 0: {"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}) -- 0x7fdf6006c380 con 0x7fdf480b2880 2024-05-03T21:19:06.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:06.638+0000 7fdf5cff9700 1 -- 172.21.15.67:0/3891637691 <== mgr.14150 v2:172.21.15.67:6800/604451592 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7fdf6006c380 con 0x7fdf480b2880 2024-05-03T21:19:06.643 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:06.640+0000 7fdf65f53700 1 -- 172.21.15.67:0/3891637691 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7fdf480b2880 msgr2=0x7fdf480b4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:19:06.643 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:06.640+0000 7fdf65f53700 1 --2- 172.21.15.67:0/3891637691 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7fdf480b2880 0x7fdf480b4d40 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fdf50000c00 tx=0x7fdf50005fb0 comp rx=0 tx=0).stop 2024-05-03T21:19:06.643 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:06.640+0000 7fdf65f53700 1 -- 172.21.15.67:0/3891637691 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdf60102fe0 msgr2=0x7fdf6019ffa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:19:06.643 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:06.640+0000 7fdf65f53700 1 --2- 172.21.15.67:0/3891637691 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdf60102fe0 0x7fdf6019ffa0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fdf5400b500 tx=0x7fdf5400b810 comp rx=0 tx=0).stop 2024-05-03T21:19:06.643 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:06.640+0000 7fdf65f53700 1 -- 172.21.15.67:0/3891637691 shutdown_connections 2024-05-03T21:19:06.643 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:06.641+0000 7fdf65f53700 1 --2- 172.21.15.67:0/3891637691 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7fdf480b2880 0x7fdf480b4d40 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:19:06.643 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:06.641+0000 7fdf65f53700 1 --2- 172.21.15.67:0/3891637691 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fdf601076c0 0x7fdf60199b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:19:06.643 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:06.641+0000 7fdf65f53700 1 --2- 172.21.15.67:0/3891637691 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fdf60102630 0x7fdf6019fa60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:19:06.643 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:06.641+0000 7fdf65f53700 1 --2- 172.21.15.67:0/3891637691 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdf60102fe0 0x7fdf6019ffa0 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:19:06.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:06.641+0000 7fdf65f53700 1 -- 172.21.15.67:0/3891637691 >> 172.21.15.67:0/3891637691 conn(0x7fdf60079350 msgr2=0x7fdf6006bc70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T21:19:06.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:06.641+0000 7fdf65f53700 1 -- 172.21.15.67:0/3891637691 shutdown_connections 2024-05-03T21:19:06.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:06.641+0000 7fdf65f53700 1 -- 172.21.15.67:0/3891637691 wait complete. 2024-05-03T21:19:06.901 INFO:journalctl@ceph.mgr.x.smithi067.stdout:May 03 21:19:06 smithi067 ceph-713bc57a-0991-11ef-bc97-c7b262605968-mgr-x[29964]: 2024-05-03T21:19:06.614+0000 7f138a950700 -1 client.0 error registering admin socket command: (17) File exists 2024-05-03T21:19:06.901 INFO:journalctl@ceph.mgr.x.smithi067.stdout:May 03 21:19:06 smithi067 ceph-713bc57a-0991-11ef-bc97-c7b262605968-mgr-x[29964]: 2024-05-03T21:19:06.614+0000 7f138a950700 -1 client.0 error registering admin socket command: (17) File exists 2024-05-03T21:19:06.901 INFO:journalctl@ceph.mgr.x.smithi067.stdout:May 03 21:19:06 smithi067 ceph-713bc57a-0991-11ef-bc97-c7b262605968-mgr-x[29964]: 2024-05-03T21:19:06.614+0000 7f138a950700 -1 client.0 error registering admin socket command: (17) File exists 2024-05-03T21:19:06.901 INFO:journalctl@ceph.mgr.x.smithi067.stdout:May 03 21:19:06 smithi067 ceph-713bc57a-0991-11ef-bc97-c7b262605968-mgr-x[29964]: 2024-05-03T21:19:06.614+0000 7f138a950700 -1 client.0 error registering admin socket command: (17) File exists 2024-05-03T21:19:06.901 INFO:journalctl@ceph.mgr.x.smithi067.stdout:May 03 21:19:06 smithi067 ceph-713bc57a-0991-11ef-bc97-c7b262605968-mgr-x[29964]: 2024-05-03T21:19:06.614+0000 7f138a950700 -1 client.0 error registering admin socket command: (17) File exists 2024-05-03T21:19:07.221 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:19:06 smithi067 ceph-mon[29730]: pgmap v298: 97 pgs: 97 active+clean; 582 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 2.0 KiB/s wr, 10 op/s 2024-05-03T21:19:07.245 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:19:06 smithi082 ceph-mon[32933]: pgmap v298: 97 pgs: 97 active+clean; 582 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 2.0 KiB/s wr, 10 op/s 2024-05-03T21:19:07.324 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:19:06 smithi138 ceph-mon[33110]: pgmap v298: 97 pgs: 97 active+clean; 582 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 2.0 KiB/s wr, 10 op/s 2024-05-03T21:19:07.390 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9a047fe7599a549042d06a3fb5b6813a203eabea shell --fsid 713bc57a-0991-11ef-bc97-c7b262605968 -- ceph fs subvolume create cephfs sv_0 --group_name qa 2024-05-03T21:19:07.642 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/713bc57a-0991-11ef-bc97-c7b262605968/mon.a/config 2024-05-03T21:19:08.120 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:19:07 smithi067 ceph-mon[29730]: from='client.14688 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-05-03T21:19:08.245 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:19:07 smithi082 ceph-mon[32933]: from='client.14688 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-05-03T21:19:08.324 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:19:07 smithi138 ceph-mon[33110]: from='client.14688 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-05-03T21:19:09.202 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:19:08 smithi067 ceph-mon[29730]: pgmap v299: 97 pgs: 97 active+clean; 582 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 2.2 KiB/s wr, 11 op/s 2024-05-03T21:19:09.203 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:19:08 smithi067 ceph-mon[29730]: mgrmap e17: x(active, since 7m), standbys: z, y 2024-05-03T21:19:09.245 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:19:08 smithi082 ceph-mon[32933]: pgmap v299: 97 pgs: 97 active+clean; 582 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 2.2 KiB/s wr, 11 op/s 2024-05-03T21:19:09.245 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:19:08 smithi082 ceph-mon[32933]: mgrmap e17: x(active, since 7m), standbys: z, y 2024-05-03T21:19:09.311 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:09.309+0000 7fa89c864700 1 -- 172.21.15.67:0/3881137723 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa898104130 msgr2=0x7fa898108140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:19:09.311 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:09.309+0000 7fa89c864700 1 --2- 172.21.15.67:0/3881137723 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa898104130 0x7fa898108140 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fa88c0099c0 tx=0x7fa88c009cd0 comp rx=0 tx=0).stop 2024-05-03T21:19:09.312 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:09.309+0000 7fa89c864700 1 -- 172.21.15.67:0/3881137723 shutdown_connections 2024-05-03T21:19:09.312 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:09.309+0000 7fa89c864700 1 --2- 172.21.15.67:0/3881137723 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fa898108870 0x7fa8981103f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:19:09.312 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:09.309+0000 7fa89c864700 1 --2- 172.21.15.67:0/3881137723 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fa898103780 0x7fa898103b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:19:09.312 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:09.309+0000 7fa89c864700 1 --2- 172.21.15.67:0/3881137723 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa898104130 0x7fa898108140 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:19:09.312 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:09.309+0000 7fa89c864700 1 -- 172.21.15.67:0/3881137723 >> 172.21.15.67:0/3881137723 conn(0x7fa89807a610 msgr2=0x7fa89807aa20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T21:19:09.312 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:09.309+0000 7fa89c864700 1 -- 172.21.15.67:0/3881137723 shutdown_connections 2024-05-03T21:19:09.312 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:09.309+0000 7fa89c864700 1 -- 172.21.15.67:0/3881137723 wait complete. 2024-05-03T21:19:09.314 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:09.310+0000 7fa89c864700 1 Processor -- start 2024-05-03T21:19:09.314 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:09.310+0000 7fa89c864700 1 -- start start 2024-05-03T21:19:09.314 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:09.310+0000 7fa89c864700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fa898103780 0x7fa89819f920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:19:09.314 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:09.310+0000 7fa89c864700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa898104130 0x7fa89819fe60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:19:09.314 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:09.310+0000 7fa89c864700 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fa898108870 0x7fa898199ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:19:09.314 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:09.310+0000 7fa89c864700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fa8981138e0 con 0x7fa898104130 2024-05-03T21:19:09.315 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:09.310+0000 7fa89c864700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7fa898113760 con 0x7fa898103780 2024-05-03T21:19:09.315 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:09.310+0000 7fa89c864700 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7fa898113a60 con 0x7fa898108870 2024-05-03T21:19:09.315 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:09.311+0000 7fa895d9b700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa898104130 0x7fa89819fe60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:19:09.315 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:09.311+0000 7fa895d9b700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa898104130 0x7fa89819fe60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:50464/0 (socket says 172.21.15.67:50464) 2024-05-03T21:19:09.315 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:09.311+0000 7fa895d9b700 1 -- 172.21.15.67:0/819212115 learned_addr learned my addr 172.21.15.67:0/819212115 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-03T21:19:09.315 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:09.311+0000 7fa89659c700 1 --2- 172.21.15.67:0/819212115 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fa898103780 0x7fa89819f920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:19:09.315 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:09.311+0000 7fa896d9d700 1 --2- 172.21.15.67:0/819212115 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fa898108870 0x7fa898199ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:19:09.315 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:09.311+0000 7fa895d9b700 1 -- 172.21.15.67:0/819212115 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fa898108870 msgr2=0x7fa898199ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:19:09.315 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:09.311+0000 7fa895d9b700 1 --2- 172.21.15.67:0/819212115 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fa898108870 0x7fa898199ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:19:09.315 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:09.311+0000 7fa895d9b700 1 -- 172.21.15.67:0/819212115 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fa898103780 msgr2=0x7fa89819f920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:19:09.315 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:09.311+0000 7fa895d9b700 1 --2- 172.21.15.67:0/819212115 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fa898103780 0x7fa89819f920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:19:09.315 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:09.311+0000 7fa895d9b700 1 -- 172.21.15.67:0/819212115 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa89819a240 con 0x7fa898104130 2024-05-03T21:19:09.315 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:09.311+0000 7fa89659c700 1 --2- 172.21.15.67:0/819212115 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fa898103780 0x7fa89819f920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-03T21:19:09.316 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:09.311+0000 7fa895d9b700 1 --2- 172.21.15.67:0/819212115 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa898104130 0x7fa89819fe60 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fa88c006010 tx=0x7fa88c003b90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T21:19:09.316 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:09.311+0000 7fa8877fe700 1 -- 172.21.15.67:0/819212115 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa88c0050c0 con 0x7fa898104130 2024-05-03T21:19:09.316 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:09.311+0000 7fa89c864700 1 -- 172.21.15.67:0/819212115 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa89819a4d0 con 0x7fa898104130 2024-05-03T21:19:09.316 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:09.311+0000 7fa8877fe700 1 -- 172.21.15.67:0/819212115 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(28 keys) ==== 1170+0+0 (secure 0 0 0) 0x7fa88c00f460 con 0x7fa898104130 2024-05-03T21:19:09.316 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:09.311+0000 7fa8877fe700 1 -- 172.21.15.67:0/819212115 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa88c017630 con 0x7fa898104130 2024-05-03T21:19:09.316 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:09.311+0000 7fa89c864700 1 -- 172.21.15.67:0/819212115 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa8981a67f0 con 0x7fa898104130 2024-05-03T21:19:09.317 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:09.313+0000 7fa89c864700 1 -- 172.21.15.67:0/819212115 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa89810ba40 con 0x7fa898104130 2024-05-03T21:19:09.317 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:09.314+0000 7fa8877fe700 1 -- 172.21.15.67:0/819212115 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 17) ==== 149703+0+0 (secure 0 0 0) 0x7fa88c00f610 con 0x7fa898104130 2024-05-03T21:19:09.317 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:09.314+0000 7fa8877fe700 1 --2- 172.21.15.67:0/819212115 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7fa8800b2910 0x7fa8800b4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:19:09.317 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:09.315+0000 7fa89659c700 1 --2- 172.21.15.67:0/819212115 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7fa8800b2910 0x7fa8800b4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:19:09.317 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:09.315+0000 7fa8877fe700 1 -- 172.21.15.67:0/819212115 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7979+0+0 (secure 0 0 0) 0x7fa88c0e2fd0 con 0x7fa898104130 2024-05-03T21:19:09.317 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:09.315+0000 7fa89659c700 1 --2- 172.21.15.67:0/819212115 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7fa8800b2910 0x7fa8800b4dd0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fa8880104e0 tx=0x7fa88800b3c0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T21:19:09.320 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:09.318+0000 7fa8877fe700 1 -- 172.21.15.67:0/819212115 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191839 (secure 0 0 0) 0x7fa88c013070 con 0x7fa898104130 2024-05-03T21:19:09.324 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:19:08 smithi138 ceph-mon[33110]: pgmap v299: 97 pgs: 97 active+clean; 582 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 2.2 KiB/s wr, 11 op/s 2024-05-03T21:19:09.324 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:19:08 smithi138 ceph-mon[33110]: mgrmap e17: x(active, since 7m), standbys: z, y 2024-05-03T21:19:09.570 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:09.568+0000 7fa89c864700 1 -- 172.21.15.67:0/819212115 --> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] -- mgr_command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}) -- 0x7fa89819afd0 con 0x7fa8800b2910 2024-05-03T21:19:09.597 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:09.595+0000 7fa8877fe700 1 -- 172.21.15.67:0/819212115 <== mgr.14150 v2:172.21.15.67:6800/604451592 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7fa89819afd0 con 0x7fa8800b2910 2024-05-03T21:19:09.600 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:09.597+0000 7fa89c864700 1 -- 172.21.15.67:0/819212115 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7fa8800b2910 msgr2=0x7fa8800b4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:19:09.600 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:09.597+0000 7fa89c864700 1 --2- 172.21.15.67:0/819212115 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7fa8800b2910 0x7fa8800b4dd0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fa8880104e0 tx=0x7fa88800b3c0 comp rx=0 tx=0).stop 2024-05-03T21:19:09.600 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:09.597+0000 7fa89c864700 1 -- 172.21.15.67:0/819212115 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa898104130 msgr2=0x7fa89819fe60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:19:09.600 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:09.597+0000 7fa89c864700 1 --2- 172.21.15.67:0/819212115 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa898104130 0x7fa89819fe60 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fa88c006010 tx=0x7fa88c003b90 comp rx=0 tx=0).stop 2024-05-03T21:19:09.600 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:09.597+0000 7fa89c864700 1 -- 172.21.15.67:0/819212115 shutdown_connections 2024-05-03T21:19:09.600 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:09.597+0000 7fa89c864700 1 --2- 172.21.15.67:0/819212115 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7fa8800b2910 0x7fa8800b4dd0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:19:09.600 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:09.597+0000 7fa89c864700 1 --2- 172.21.15.67:0/819212115 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fa898108870 0x7fa898199ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:19:09.600 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:09.597+0000 7fa89c864700 1 --2- 172.21.15.67:0/819212115 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fa898103780 0x7fa89819f920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:19:09.600 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:09.597+0000 7fa89c864700 1 --2- 172.21.15.67:0/819212115 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa898104130 0x7fa89819fe60 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:19:09.600 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:09.597+0000 7fa89c864700 1 -- 172.21.15.67:0/819212115 >> 172.21.15.67:0/819212115 conn(0x7fa89807a610 msgr2=0x7fa898101790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T21:19:09.600 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:09.598+0000 7fa89c864700 1 -- 172.21.15.67:0/819212115 shutdown_connections 2024-05-03T21:19:09.600 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:09.598+0000 7fa89c864700 1 -- 172.21.15.67:0/819212115 wait complete. 2024-05-03T21:19:10.378 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9a047fe7599a549042d06a3fb5b6813a203eabea shell --fsid 713bc57a-0991-11ef-bc97-c7b262605968 -- ceph fs subvolume getpath cephfs --group_name qa sv_0 2024-05-03T21:19:10.628 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/713bc57a-0991-11ef-bc97-c7b262605968/mon.a/config 2024-05-03T21:19:11.009 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:19:10 smithi067 ceph-mon[29730]: from='client.14700 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-05-03T21:19:11.010 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:19:10 smithi067 ceph-mon[29730]: pgmap v300: 97 pgs: 97 active+clean; 582 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 1.4 KiB/s wr, 10 op/s 2024-05-03T21:19:11.245 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:19:10 smithi082 ceph-mon[32933]: from='client.14700 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-05-03T21:19:11.245 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:19:10 smithi082 ceph-mon[32933]: pgmap v300: 97 pgs: 97 active+clean; 582 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 1.4 KiB/s wr, 10 op/s 2024-05-03T21:19:11.324 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:19:10 smithi138 ceph-mon[33110]: from='client.14700 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-05-03T21:19:11.324 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:19:10 smithi138 ceph-mon[33110]: pgmap v300: 97 pgs: 97 active+clean; 582 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 1.4 KiB/s wr, 10 op/s 2024-05-03T21:19:12.268 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:12.265+0000 7efcee9a9700 1 -- 172.21.15.67:0/1841383240 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efce806c630 msgr2=0x7efce8110360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:19:12.268 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:12.265+0000 7efcee9a9700 1 --2- 172.21.15.67:0/1841383240 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efce806c630 0x7efce8110360 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7efce000b0d0 tx=0x7efce000b3e0 comp rx=0 tx=0).stop 2024-05-03T21:19:12.268 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:12.265+0000 7efcee9a9700 1 -- 172.21.15.67:0/1841383240 shutdown_connections 2024-05-03T21:19:12.268 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:12.265+0000 7efcee9a9700 1 --2- 172.21.15.67:0/1841383240 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efce806c630 0x7efce8110360 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:19:12.268 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:12.265+0000 7efcee9a9700 1 --2- 172.21.15.67:0/1841383240 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7efce8109590 0x7efce8109970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:19:12.268 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:12.265+0000 7efcee9a9700 1 --2- 172.21.15.67:0/1841383240 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7efce806bc70 0x7efce806c0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:19:12.269 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:12.265+0000 7efcee9a9700 1 -- 172.21.15.67:0/1841383240 >> 172.21.15.67:0/1841383240 conn(0x7efce8079440 msgr2=0x7efce8079850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T21:19:12.269 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:12.265+0000 7efcee9a9700 1 -- 172.21.15.67:0/1841383240 shutdown_connections 2024-05-03T21:19:12.269 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:12.266+0000 7efcee9a9700 1 -- 172.21.15.67:0/1841383240 wait complete. 2024-05-03T21:19:12.269 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:12.266+0000 7efcee9a9700 1 Processor -- start 2024-05-03T21:19:12.269 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:12.266+0000 7efcee9a9700 1 -- start start 2024-05-03T21:19:12.271 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:12.267+0000 7efcee9a9700 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7efce806bc70 0x7efce819f890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:19:12.271 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:12.267+0000 7efcee9a9700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7efce806c630 0x7efce819fdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:19:12.271 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:12.267+0000 7efcee9a9700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efce8109590 0x7efce8199a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:19:12.271 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:12.267+0000 7efcee9a9700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7efce8113910 con 0x7efce8109590 2024-05-03T21:19:12.271 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:12.267+0000 7efcee9a9700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7efce8113790 con 0x7efce806c630 2024-05-03T21:19:12.271 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:12.267+0000 7efcee9a9700 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7efce8113a90 con 0x7efce806bc70 2024-05-03T21:19:12.271 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:12.267+0000 7efcecf46700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efce8109590 0x7efce8199a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:19:12.271 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:12.267+0000 7efcecf46700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efce8109590 0x7efce8199a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:50486/0 (socket says 172.21.15.67:50486) 2024-05-03T21:19:12.271 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:12.267+0000 7efcecf46700 1 -- 172.21.15.67:0/134214966 learned_addr learned my addr 172.21.15.67:0/134214966 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-03T21:19:12.271 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:12.267+0000 7efce77fe700 1 --2- 172.21.15.67:0/134214966 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7efce806c630 0x7efce819fdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:19:12.271 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:12.267+0000 7efce7fff700 1 --2- 172.21.15.67:0/134214966 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7efce806bc70 0x7efce819f890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:19:12.271 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:12.267+0000 7efcecf46700 1 -- 172.21.15.67:0/134214966 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7efce806bc70 msgr2=0x7efce819f890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:19:12.272 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:12.267+0000 7efcecf46700 1 --2- 172.21.15.67:0/134214966 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7efce806bc70 0x7efce819f890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:19:12.272 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:12.267+0000 7efcecf46700 1 -- 172.21.15.67:0/134214966 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7efce806c630 msgr2=0x7efce819fdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:19:12.272 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:12.267+0000 7efcecf46700 1 --2- 172.21.15.67:0/134214966 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7efce806c630 0x7efce819fdd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:19:12.272 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:12.267+0000 7efcecf46700 1 -- 172.21.15.67:0/134214966 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efce819a1b0 con 0x7efce8109590 2024-05-03T21:19:12.272 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:12.267+0000 7efce77fe700 1 --2- 172.21.15.67:0/134214966 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7efce806c630 0x7efce819fdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-03T21:19:12.272 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:12.267+0000 7efce7fff700 1 --2- 172.21.15.67:0/134214966 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7efce806bc70 0x7efce819f890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-03T21:19:12.272 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:12.267+0000 7efcecf46700 1 --2- 172.21.15.67:0/134214966 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efce8109590 0x7efce8199a50 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7efce000d010 tx=0x7efce0007ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T21:19:12.272 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:12.267+0000 7efce57fa700 1 -- 172.21.15.67:0/134214966 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efce001f070 con 0x7efce8109590 2024-05-03T21:19:12.272 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:12.267+0000 7efce57fa700 1 -- 172.21.15.67:0/134214966 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(28 keys) ==== 1170+0+0 (secure 0 0 0) 0x7efce0003c00 con 0x7efce8109590 2024-05-03T21:19:12.272 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:12.267+0000 7efce57fa700 1 -- 172.21.15.67:0/134214966 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efce001b840 con 0x7efce8109590 2024-05-03T21:19:12.272 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:12.268+0000 7efcee9a9700 1 -- 172.21.15.67:0/134214966 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efce819a440 con 0x7efce8109590 2024-05-03T21:19:12.272 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:12.268+0000 7efcee9a9700 1 -- 172.21.15.67:0/134214966 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efce819a840 con 0x7efce8109590 2024-05-03T21:19:12.272 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:12.268+0000 7efcee9a9700 1 -- 172.21.15.67:0/134214966 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efce806a190 con 0x7efce8109590 2024-05-03T21:19:12.273 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:12.270+0000 7efce57fa700 1 -- 172.21.15.67:0/134214966 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 17) ==== 149703+0+0 (secure 0 0 0) 0x7efce0022070 con 0x7efce8109590 2024-05-03T21:19:12.273 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:12.270+0000 7efce57fa700 1 --2- 172.21.15.67:0/134214966 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7efcd00b2910 0x7efcd00b4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:19:12.273 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:12.271+0000 7efce7fff700 1 --2- 172.21.15.67:0/134214966 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7efcd00b2910 0x7efcd00b4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:19:12.273 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:12.271+0000 7efce57fa700 1 -- 172.21.15.67:0/134214966 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7979+0+0 (secure 0 0 0) 0x7efce000f020 con 0x7efce8109590 2024-05-03T21:19:12.273 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:12.271+0000 7efce7fff700 1 --2- 172.21.15.67:0/134214966 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7efcd00b2910 0x7efcd00b4dd0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7efcd8005950 tx=0x7efcd8005f50 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T21:19:12.276 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:12.274+0000 7efce57fa700 1 -- 172.21.15.67:0/134214966 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191839 (secure 0 0 0) 0x7efce0029020 con 0x7efce8109590 2024-05-03T21:19:12.523 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:12.521+0000 7efcee9a9700 1 -- 172.21.15.67:0/134214966 --> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}) -- 0x7efce8101460 con 0x7efcd00b2910 2024-05-03T21:19:12.528 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:12.526+0000 7efce57fa700 1 -- 172.21.15.67:0/134214966 <== mgr.14150 v2:172.21.15.67:6800/604451592 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+54 (secure 0 0 0) 0x7efce8101460 con 0x7efcd00b2910 2024-05-03T21:19:12.528 INFO:teuthology.orchestra.run.smithi067.stdout:/volumes/qa/sv_0/e5202a87-05ea-4978-8241-93ff95ddb867 2024-05-03T21:19:12.531 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:12.528+0000 7efcee9a9700 1 -- 172.21.15.67:0/134214966 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7efcd00b2910 msgr2=0x7efcd00b4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:19:12.531 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:12.528+0000 7efcee9a9700 1 --2- 172.21.15.67:0/134214966 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7efcd00b2910 0x7efcd00b4dd0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7efcd8005950 tx=0x7efcd8005f50 comp rx=0 tx=0).stop 2024-05-03T21:19:12.531 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:12.528+0000 7efcee9a9700 1 -- 172.21.15.67:0/134214966 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efce8109590 msgr2=0x7efce8199a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:19:12.531 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:12.528+0000 7efcee9a9700 1 --2- 172.21.15.67:0/134214966 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efce8109590 0x7efce8199a50 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7efce000d010 tx=0x7efce0007ad0 comp rx=0 tx=0).stop 2024-05-03T21:19:12.531 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:12.528+0000 7efcee9a9700 1 -- 172.21.15.67:0/134214966 shutdown_connections 2024-05-03T21:19:12.531 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:12.528+0000 7efcee9a9700 1 --2- 172.21.15.67:0/134214966 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7efcd00b2910 0x7efcd00b4dd0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:19:12.531 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:12.528+0000 7efcee9a9700 1 --2- 172.21.15.67:0/134214966 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efce8109590 0x7efce8199a50 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:19:12.531 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:12.528+0000 7efcee9a9700 1 --2- 172.21.15.67:0/134214966 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7efce806bc70 0x7efce819f890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:19:12.531 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:12.528+0000 7efcee9a9700 1 --2- 172.21.15.67:0/134214966 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7efce806c630 0x7efce819fdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:19:12.531 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:12.528+0000 7efcee9a9700 1 -- 172.21.15.67:0/134214966 >> 172.21.15.67:0/134214966 conn(0x7efce8079440 msgr2=0x7efce8100d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T21:19:12.531 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:12.528+0000 7efcee9a9700 1 -- 172.21.15.67:0/134214966 shutdown_connections 2024-05-03T21:19:12.531 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:12.528+0000 7efcee9a9700 1 -- 172.21.15.67:0/134214966 wait complete. 2024-05-03T21:19:13.179 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:19:12 smithi067 ceph-mon[29730]: pgmap v301: 97 pgs: 97 active+clean; 603 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 3.5 KiB/s wr, 12 op/s 2024-05-03T21:19:13.209 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9a047fe7599a549042d06a3fb5b6813a203eabea shell --fsid 713bc57a-0991-11ef-bc97-c7b262605968 -- ceph fs subvolume create cephfs sv_1 --group_name qa 2024-05-03T21:19:13.245 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:19:12 smithi082 ceph-mon[32933]: pgmap v301: 97 pgs: 97 active+clean; 603 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 3.5 KiB/s wr, 12 op/s 2024-05-03T21:19:13.324 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:19:12 smithi138 ceph-mon[33110]: pgmap v301: 97 pgs: 97 active+clean; 603 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 3.5 KiB/s wr, 12 op/s 2024-05-03T21:19:13.462 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/713bc57a-0991-11ef-bc97-c7b262605968/mon.a/config 2024-05-03T21:19:14.221 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:19:13 smithi067 ceph-mon[29730]: from='client.14706 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2024-05-03T21:19:14.245 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:19:13 smithi082 ceph-mon[32933]: from='client.14706 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2024-05-03T21:19:14.322 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:19:13 smithi138 ceph-mon[33110]: from='client.14706 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2024-05-03T21:19:15.142 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:19:14 smithi067 ceph-mon[29730]: pgmap v302: 97 pgs: 97 active+clean; 603 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 2.7 KiB/s wr, 9 op/s 2024-05-03T21:19:15.167 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:15.165+0000 7fa615b9c700 1 -- 172.21.15.67:0/618338857 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa61006c3d0 msgr2=0x7fa610110320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:19:15.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:15.165+0000 7fa615b9c700 1 --2- 172.21.15.67:0/618338857 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa61006c3d0 0x7fa610110320 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fa60800b0a0 tx=0x7fa60800b3b0 comp rx=0 tx=0).stop 2024-05-03T21:19:15.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:15.165+0000 7fa615b9c700 1 -- 172.21.15.67:0/618338857 shutdown_connections 2024-05-03T21:19:15.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:15.165+0000 7fa615b9c700 1 --2- 172.21.15.67:0/618338857 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa61006c3d0 0x7fa610110320 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:19:15.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:15.165+0000 7fa615b9c700 1 --2- 172.21.15.67:0/618338857 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fa610109500 0x7fa6101098e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:19:15.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:15.165+0000 7fa615b9c700 1 --2- 172.21.15.67:0/618338857 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fa61006ba10 0x7fa61006be90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:19:15.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:15.165+0000 7fa615b9c700 1 -- 172.21.15.67:0/618338857 >> 172.21.15.67:0/618338857 conn(0x7fa610079310 msgr2=0x7fa610079720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T21:19:15.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:15.165+0000 7fa615b9c700 1 -- 172.21.15.67:0/618338857 shutdown_connections 2024-05-03T21:19:15.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:15.165+0000 7fa615b9c700 1 -- 172.21.15.67:0/618338857 wait complete. 2024-05-03T21:19:15.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:15.166+0000 7fa615b9c700 1 Processor -- start 2024-05-03T21:19:15.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:15.166+0000 7fa615b9c700 1 -- start start 2024-05-03T21:19:15.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:15.166+0000 7fa615b9c700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fa61006ba10 0x7fa610102460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:19:15.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:15.166+0000 7fa615b9c700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa61006c3d0 0x7fa6101029a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:19:15.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:15.166+0000 7fa615b9c700 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fa610109500 0x7fa610102ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:19:15.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:15.166+0000 7fa615b9c700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fa6101138c0 con 0x7fa61006c3d0 2024-05-03T21:19:15.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:15.166+0000 7fa615b9c700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7fa610113740 con 0x7fa61006ba10 2024-05-03T21:19:15.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:15.166+0000 7fa615b9c700 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7fa610113a40 con 0x7fa610109500 2024-05-03T21:19:15.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:15.166+0000 7fa60effd700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa61006c3d0 0x7fa6101029a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:19:15.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:15.166+0000 7fa60effd700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa61006c3d0 0x7fa6101029a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:50502/0 (socket says 172.21.15.67:50502) 2024-05-03T21:19:15.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:15.166+0000 7fa60effd700 1 -- 172.21.15.67:0/2104440186 learned_addr learned my addr 172.21.15.67:0/2104440186 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-03T21:19:15.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:15.166+0000 7fa60f7fe700 1 --2- 172.21.15.67:0/2104440186 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fa61006ba10 0x7fa610102460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:19:15.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:15.166+0000 7fa60ffff700 1 --2- 172.21.15.67:0/2104440186 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fa610109500 0x7fa610102ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:19:15.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:15.167+0000 7fa60effd700 1 -- 172.21.15.67:0/2104440186 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fa610109500 msgr2=0x7fa610102ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:19:15.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:15.167+0000 7fa60effd700 1 --2- 172.21.15.67:0/2104440186 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fa610109500 0x7fa610102ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:19:15.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:15.167+0000 7fa60effd700 1 -- 172.21.15.67:0/2104440186 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fa61006ba10 msgr2=0x7fa610102460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:19:15.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:15.167+0000 7fa60effd700 1 --2- 172.21.15.67:0/2104440186 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fa61006ba10 0x7fa610102460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:19:15.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:15.167+0000 7fa60effd700 1 -- 172.21.15.67:0/2104440186 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa6101a81c0 con 0x7fa61006c3d0 2024-05-03T21:19:15.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:15.167+0000 7fa60effd700 1 --2- 172.21.15.67:0/2104440186 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa61006c3d0 0x7fa6101029a0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fa60400c920 tx=0x7fa60400cc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T21:19:15.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:15.167+0000 7fa60cff9700 1 -- 172.21.15.67:0/2104440186 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa604007b00 con 0x7fa61006c3d0 2024-05-03T21:19:15.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:15.167+0000 7fa615b9c700 1 -- 172.21.15.67:0/2104440186 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa6101a84b0 con 0x7fa61006c3d0 2024-05-03T21:19:15.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:15.167+0000 7fa60cff9700 1 -- 172.21.15.67:0/2104440186 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(28 keys) ==== 1170+0+0 (secure 0 0 0) 0x7fa604004d30 con 0x7fa61006c3d0 2024-05-03T21:19:15.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:15.167+0000 7fa60cff9700 1 -- 172.21.15.67:0/2104440186 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa604005680 con 0x7fa61006c3d0 2024-05-03T21:19:15.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:15.167+0000 7fa615b9c700 1 -- 172.21.15.67:0/2104440186 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa6101a8a40 con 0x7fa61006c3d0 2024-05-03T21:19:15.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:15.168+0000 7fa615b9c700 1 -- 172.21.15.67:0/2104440186 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa61010b9e0 con 0x7fa61006c3d0 2024-05-03T21:19:15.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:15.169+0000 7fa60cff9700 1 -- 172.21.15.67:0/2104440186 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 17) ==== 149703+0+0 (secure 0 0 0) 0x7fa604004750 con 0x7fa61006c3d0 2024-05-03T21:19:15.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:15.170+0000 7fa60cff9700 1 --2- 172.21.15.67:0/2104440186 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7fa5f80b2990 0x7fa5f80b4e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:19:15.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:15.170+0000 7fa60f7fe700 1 --2- 172.21.15.67:0/2104440186 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7fa5f80b2990 0x7fa5f80b4e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:19:15.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:15.170+0000 7fa60cff9700 1 -- 172.21.15.67:0/2104440186 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7979+0+0 (secure 0 0 0) 0x7fa6040e2100 con 0x7fa61006c3d0 2024-05-03T21:19:15.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:15.170+0000 7fa60f7fe700 1 --2- 172.21.15.67:0/2104440186 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7fa5f80b2990 0x7fa5f80b4e50 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fa6000045b0 tx=0x7fa60000b3c0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T21:19:15.175 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:15.173+0000 7fa60cff9700 1 -- 172.21.15.67:0/2104440186 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191839 (secure 0 0 0) 0x7fa604014030 con 0x7fa61006c3d0 2024-05-03T21:19:15.245 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:19:14 smithi082 ceph-mon[32933]: pgmap v302: 97 pgs: 97 active+clean; 603 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 2.7 KiB/s wr, 9 op/s 2024-05-03T21:19:15.324 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:19:14 smithi138 ceph-mon[33110]: pgmap v302: 97 pgs: 97 active+clean; 603 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 2.7 KiB/s wr, 9 op/s 2024-05-03T21:19:15.420 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:15.418+0000 7fa615b9c700 1 -- 172.21.15.67:0/2104440186 --> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] -- mgr_command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}) -- 0x7fa6100638e0 con 0x7fa5f80b2990 2024-05-03T21:19:15.447 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:15.445+0000 7fa60cff9700 1 -- 172.21.15.67:0/2104440186 <== mgr.14150 v2:172.21.15.67:6800/604451592 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7fa6100638e0 con 0x7fa5f80b2990 2024-05-03T21:19:15.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:15.447+0000 7fa615b9c700 1 -- 172.21.15.67:0/2104440186 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7fa5f80b2990 msgr2=0x7fa5f80b4e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:19:15.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:15.447+0000 7fa615b9c700 1 --2- 172.21.15.67:0/2104440186 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7fa5f80b2990 0x7fa5f80b4e50 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fa6000045b0 tx=0x7fa60000b3c0 comp rx=0 tx=0).stop 2024-05-03T21:19:15.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:15.447+0000 7fa615b9c700 1 -- 172.21.15.67:0/2104440186 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa61006c3d0 msgr2=0x7fa6101029a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:19:15.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:15.447+0000 7fa615b9c700 1 --2- 172.21.15.67:0/2104440186 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa61006c3d0 0x7fa6101029a0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fa60400c920 tx=0x7fa60400cc30 comp rx=0 tx=0).stop 2024-05-03T21:19:15.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:15.447+0000 7fa615b9c700 1 -- 172.21.15.67:0/2104440186 shutdown_connections 2024-05-03T21:19:15.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:15.447+0000 7fa615b9c700 1 --2- 172.21.15.67:0/2104440186 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7fa5f80b2990 0x7fa5f80b4e50 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:19:15.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:15.447+0000 7fa615b9c700 1 --2- 172.21.15.67:0/2104440186 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fa610109500 0x7fa610102ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:19:15.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:15.447+0000 7fa615b9c700 1 --2- 172.21.15.67:0/2104440186 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fa61006ba10 0x7fa610102460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:19:15.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:15.447+0000 7fa615b9c700 1 --2- 172.21.15.67:0/2104440186 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa61006c3d0 0x7fa6101029a0 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:19:15.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:15.447+0000 7fa615b9c700 1 -- 172.21.15.67:0/2104440186 >> 172.21.15.67:0/2104440186 conn(0x7fa610079310 msgr2=0x7fa610100e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T21:19:15.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:15.447+0000 7fa615b9c700 1 -- 172.21.15.67:0/2104440186 shutdown_connections 2024-05-03T21:19:15.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:15.447+0000 7fa615b9c700 1 -- 172.21.15.67:0/2104440186 wait complete. 2024-05-03T21:19:16.051 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9a047fe7599a549042d06a3fb5b6813a203eabea shell --fsid 713bc57a-0991-11ef-bc97-c7b262605968 -- ceph fs subvolume getpath cephfs --group_name qa sv_1 2024-05-03T21:19:16.303 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/713bc57a-0991-11ef-bc97-c7b262605968/mon.a/config 2024-05-03T21:19:17.078 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:19:16 smithi067 ceph-mon[29730]: from='client.14712 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-05-03T21:19:17.078 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:19:16 smithi067 ceph-mon[29730]: pgmap v303: 97 pgs: 97 active+clean; 607 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 2.8 KiB/s wr, 8 op/s 2024-05-03T21:19:17.245 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:19:16 smithi082 ceph-mon[32933]: from='client.14712 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-05-03T21:19:17.245 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:19:16 smithi082 ceph-mon[32933]: pgmap v303: 97 pgs: 97 active+clean; 607 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 2.8 KiB/s wr, 8 op/s 2024-05-03T21:19:17.324 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:19:16 smithi138 ceph-mon[33110]: from='client.14712 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-05-03T21:19:17.325 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:19:16 smithi138 ceph-mon[33110]: pgmap v303: 97 pgs: 97 active+clean; 607 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 2.8 KiB/s wr, 8 op/s 2024-05-03T21:19:17.968 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:17.966+0000 7fdb4f6e1700 1 -- 172.21.15.67:0/2034248853 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdb481087c0 msgr2=0x7fdb481103d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:19:17.969 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:17.966+0000 7fdb4f6e1700 1 --2- 172.21.15.67:0/2034248853 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdb481087c0 0x7fdb481103d0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fdb4000b0d0 tx=0x7fdb4000b3e0 comp rx=0 tx=0).stop 2024-05-03T21:19:17.969 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:17.966+0000 7fdb4f6e1700 1 -- 172.21.15.67:0/2034248853 shutdown_connections 2024-05-03T21:19:17.969 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:17.966+0000 7fdb4f6e1700 1 --2- 172.21.15.67:0/2034248853 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdb481087c0 0x7fdb481103d0 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:19:17.969 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:17.966+0000 7fdb4f6e1700 1 --2- 172.21.15.67:0/2034248853 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fdb4806be70 0x7fdb4806c250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:19:17.969 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:17.966+0000 7fdb4f6e1700 1 --2- 172.21.15.67:0/2034248853 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fdb4806c820 0x7fdb481080e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:19:17.969 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:17.966+0000 7fdb4f6e1700 1 -- 172.21.15.67:0/2034248853 >> 172.21.15.67:0/2034248853 conn(0x7fdb480783f0 msgr2=0x7fdb48078800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T21:19:17.969 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:17.966+0000 7fdb4f6e1700 1 -- 172.21.15.67:0/2034248853 shutdown_connections 2024-05-03T21:19:17.969 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:17.966+0000 7fdb4f6e1700 1 -- 172.21.15.67:0/2034248853 wait complete. 2024-05-03T21:19:17.969 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:17.966+0000 7fdb4f6e1700 1 Processor -- start 2024-05-03T21:19:17.969 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:17.967+0000 7fdb4f6e1700 1 -- start start 2024-05-03T21:19:17.970 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:17.967+0000 7fdb4f6e1700 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fdb4806be70 0x7fdb4819fa40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:19:17.970 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:17.967+0000 7fdb4f6e1700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fdb4806c820 0x7fdb4819ff80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:19:17.970 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:17.967+0000 7fdb4f6e1700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdb481087c0 0x7fdb48199b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:19:17.970 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:17.967+0000 7fdb4f6e1700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fdb48113950 con 0x7fdb481087c0 2024-05-03T21:19:17.970 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:17.967+0000 7fdb4f6e1700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7fdb481137d0 con 0x7fdb4806c820 2024-05-03T21:19:17.970 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:17.967+0000 7fdb4f6e1700 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7fdb48113ad0 con 0x7fdb4806be70 2024-05-03T21:19:17.970 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:17.967+0000 7fdb4dc7e700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdb481087c0 0x7fdb48199b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:19:17.970 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:17.967+0000 7fdb4dc7e700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdb481087c0 0x7fdb48199b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:39894/0 (socket says 172.21.15.67:39894) 2024-05-03T21:19:17.970 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:17.967+0000 7fdb4dc7e700 1 -- 172.21.15.67:0/3341805509 learned_addr learned my addr 172.21.15.67:0/3341805509 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-03T21:19:17.970 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:17.967+0000 7fdb4d47d700 1 --2- 172.21.15.67:0/3341805509 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fdb4806be70 0x7fdb4819fa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:19:17.970 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:17.967+0000 7fdb4cc7c700 1 --2- 172.21.15.67:0/3341805509 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fdb4806c820 0x7fdb4819ff80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:19:17.970 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:17.967+0000 7fdb4dc7e700 1 -- 172.21.15.67:0/3341805509 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fdb4806be70 msgr2=0x7fdb4819fa40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:19:17.971 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:17.967+0000 7fdb4dc7e700 1 --2- 172.21.15.67:0/3341805509 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fdb4806be70 0x7fdb4819fa40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:19:17.971 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:17.967+0000 7fdb4dc7e700 1 -- 172.21.15.67:0/3341805509 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fdb4806c820 msgr2=0x7fdb4819ff80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:19:17.971 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:17.967+0000 7fdb4dc7e700 1 --2- 172.21.15.67:0/3341805509 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fdb4806c820 0x7fdb4819ff80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:19:17.971 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:17.967+0000 7fdb4dc7e700 1 -- 172.21.15.67:0/3341805509 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdb4819a3f0 con 0x7fdb481087c0 2024-05-03T21:19:17.971 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:17.968+0000 7fdb4dc7e700 1 --2- 172.21.15.67:0/3341805509 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdb481087c0 0x7fdb48199b10 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fdb4000d010 tx=0x7fdb40007a20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T21:19:17.972 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:17.968+0000 7fdb3a7fc700 1 -- 172.21.15.67:0/3341805509 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb4001f070 con 0x7fdb481087c0 2024-05-03T21:19:17.972 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:17.968+0000 7fdb3a7fc700 1 -- 172.21.15.67:0/3341805509 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(28 keys) ==== 1170+0+0 (secure 0 0 0) 0x7fdb40003c00 con 0x7fdb481087c0 2024-05-03T21:19:17.972 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:17.968+0000 7fdb4f6e1700 1 -- 172.21.15.67:0/3341805509 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdb4819a680 con 0x7fdb481087c0 2024-05-03T21:19:17.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:17.968+0000 7fdb4f6e1700 1 -- 172.21.15.67:0/3341805509 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdb481a67f0 con 0x7fdb481087c0 2024-05-03T21:19:17.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:17.968+0000 7fdb3a7fc700 1 -- 172.21.15.67:0/3341805509 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb4001b770 con 0x7fdb481087c0 2024-05-03T21:19:17.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:17.969+0000 7fdb4f6e1700 1 -- 172.21.15.67:0/3341805509 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdb48104c30 con 0x7fdb481087c0 2024-05-03T21:19:17.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:17.970+0000 7fdb3a7fc700 1 -- 172.21.15.67:0/3341805509 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 17) ==== 149703+0+0 (secure 0 0 0) 0x7fdb40022070 con 0x7fdb481087c0 2024-05-03T21:19:17.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:17.971+0000 7fdb3a7fc700 1 --2- 172.21.15.67:0/3341805509 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7fdb340b2910 0x7fdb340b4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:19:17.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:17.971+0000 7fdb4d47d700 1 --2- 172.21.15.67:0/3341805509 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7fdb340b2910 0x7fdb340b4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:19:17.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:17.971+0000 7fdb3a7fc700 1 -- 172.21.15.67:0/3341805509 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7979+0+0 (secure 0 0 0) 0x7fdb4000f020 con 0x7fdb481087c0 2024-05-03T21:19:17.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:17.971+0000 7fdb4d47d700 1 --2- 172.21.15.67:0/3341805509 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7fdb340b2910 0x7fdb340b4dd0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fdb3c004630 tx=0x7fdb3c00b3c0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T21:19:17.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:17.974+0000 7fdb3a7fc700 1 -- 172.21.15.67:0/3341805509 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191839 (secure 0 0 0) 0x7fdb40029020 con 0x7fdb481087c0 2024-05-03T21:19:18.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:18.223+0000 7fdb4f6e1700 1 -- 172.21.15.67:0/3341805509 --> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}) -- 0x7fdb48100ab0 con 0x7fdb340b2910 2024-05-03T21:19:18.229 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:18.228+0000 7fdb3a7fc700 1 -- 172.21.15.67:0/3341805509 <== mgr.14150 v2:172.21.15.67:6800/604451592 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+54 (secure 0 0 0) 0x7fdb48100ab0 con 0x7fdb340b2910 2024-05-03T21:19:18.229 INFO:teuthology.orchestra.run.smithi067.stdout:/volumes/qa/sv_1/bfa342a3-081d-4b09-8d8d-b10dd6f0067e 2024-05-03T21:19:18.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:18.230+0000 7fdb4f6e1700 1 -- 172.21.15.67:0/3341805509 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7fdb340b2910 msgr2=0x7fdb340b4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:19:18.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:18.230+0000 7fdb4f6e1700 1 --2- 172.21.15.67:0/3341805509 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7fdb340b2910 0x7fdb340b4dd0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fdb3c004630 tx=0x7fdb3c00b3c0 comp rx=0 tx=0).stop 2024-05-03T21:19:18.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:18.230+0000 7fdb4f6e1700 1 -- 172.21.15.67:0/3341805509 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdb481087c0 msgr2=0x7fdb48199b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:19:18.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:18.230+0000 7fdb4f6e1700 1 --2- 172.21.15.67:0/3341805509 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdb481087c0 0x7fdb48199b10 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fdb4000d010 tx=0x7fdb40007a20 comp rx=0 tx=0).stop 2024-05-03T21:19:18.232 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:18.230+0000 7fdb4f6e1700 1 -- 172.21.15.67:0/3341805509 shutdown_connections 2024-05-03T21:19:18.232 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:18.230+0000 7fdb4f6e1700 1 --2- 172.21.15.67:0/3341805509 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7fdb340b2910 0x7fdb340b4dd0 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:19:18.232 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:18.230+0000 7fdb4f6e1700 1 --2- 172.21.15.67:0/3341805509 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdb481087c0 0x7fdb48199b10 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:19:18.232 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:18.230+0000 7fdb4f6e1700 1 --2- 172.21.15.67:0/3341805509 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fdb4806be70 0x7fdb4819fa40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:19:18.232 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:18.230+0000 7fdb4f6e1700 1 --2- 172.21.15.67:0/3341805509 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fdb4806c820 0x7fdb4819ff80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:19:18.232 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:18.230+0000 7fdb4f6e1700 1 -- 172.21.15.67:0/3341805509 >> 172.21.15.67:0/3341805509 conn(0x7fdb480783f0 msgr2=0x7fdb481003a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T21:19:18.232 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:18.230+0000 7fdb4f6e1700 1 -- 172.21.15.67:0/3341805509 shutdown_connections 2024-05-03T21:19:18.232 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:18.230+0000 7fdb4f6e1700 1 -- 172.21.15.67:0/3341805509 wait complete. 2024-05-03T21:19:18.942 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9a047fe7599a549042d06a3fb5b6813a203eabea shell --fsid 713bc57a-0991-11ef-bc97-c7b262605968 -- ceph fs dump --format=json 2024-05-03T21:19:19.193 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/713bc57a-0991-11ef-bc97-c7b262605968/mon.a/config 2024-05-03T21:19:19.221 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:19:18 smithi067 ceph-mon[29730]: pgmap v304: 97 pgs: 97 active+clean; 607 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 2.9 KiB/s wr, 8 op/s 2024-05-03T21:19:19.221 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:19:18 smithi067 ceph-mon[29730]: from='client.14718 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-05-03T21:19:19.245 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:19:18 smithi082 ceph-mon[32933]: pgmap v304: 97 pgs: 97 active+clean; 607 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 2.9 KiB/s wr, 8 op/s 2024-05-03T21:19:19.245 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:19:18 smithi082 ceph-mon[32933]: from='client.14718 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-05-03T21:19:19.324 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:19:18 smithi138 ceph-mon[33110]: pgmap v304: 97 pgs: 97 active+clean; 607 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 2.9 KiB/s wr, 8 op/s 2024-05-03T21:19:19.324 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:19:18 smithi138 ceph-mon[33110]: from='client.14718 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-05-03T21:19:20.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:20.917+0000 7f3979be0700 1 -- 172.21.15.67:0/2658215706 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3974109600 msgr2=0x7f39741099e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:19:20.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:20.917+0000 7f3979be0700 1 --2- 172.21.15.67:0/2658215706 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3974109600 0x7f39741099e0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f3964009990 tx=0x7f3964009ca0 comp rx=0 tx=0).stop 2024-05-03T21:19:20.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:20.917+0000 7f3979be0700 1 -- 172.21.15.67:0/2658215706 shutdown_connections 2024-05-03T21:19:20.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:20.917+0000 7f3979be0700 1 --2- 172.21.15.67:0/2658215706 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3974109600 0x7f39741099e0 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:19:20.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:20.917+0000 7f3979be0700 1 --2- 172.21.15.67:0/2658215706 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f3974103eb0 0x7f3974110390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:19:20.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:20.917+0000 7f3979be0700 1 --2- 172.21.15.67:0/2658215706 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f39741034f0 0x7f3974103970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:19:20.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:20.917+0000 7f3979be0700 1 -- 172.21.15.67:0/2658215706 >> 172.21.15.67:0/2658215706 conn(0x7f397407a650 msgr2=0x7f397407aa60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T21:19:20.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:20.917+0000 7f3979be0700 1 -- 172.21.15.67:0/2658215706 shutdown_connections 2024-05-03T21:19:20.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:20.917+0000 7f3979be0700 1 -- 172.21.15.67:0/2658215706 wait complete. 2024-05-03T21:19:20.921 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:20.918+0000 7f3979be0700 1 Processor -- start 2024-05-03T21:19:20.921 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:20.918+0000 7f3979be0700 1 -- start start 2024-05-03T21:19:20.921 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:20.919+0000 7f3979be0700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f39741034f0 0x7f39740757a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:19:20.921 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:20.919+0000 7f3979be0700 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f3974103eb0 0x7f3974075ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:19:20.921 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:20.919+0000 7f3979be0700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f3974109600 0x7f3974076220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:19:20.921 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:20.919+0000 7f3979be0700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f3974113900 con 0x7f39741034f0 2024-05-03T21:19:20.921 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:20.919+0000 7f3979be0700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f3974113780 con 0x7f3974109600 2024-05-03T21:19:20.921 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:20.919+0000 7f3979be0700 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f3974113a80 con 0x7f3974103eb0 2024-05-03T21:19:20.922 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:20.919+0000 7f39737fe700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f39741034f0 0x7f39740757a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:19:20.922 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:20.919+0000 7f39737fe700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f39741034f0 0x7f39740757a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:39918/0 (socket says 172.21.15.67:39918) 2024-05-03T21:19:20.922 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:20.919+0000 7f39737fe700 1 -- 172.21.15.67:0/3044172349 learned_addr learned my addr 172.21.15.67:0/3044172349 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-03T21:19:20.922 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:20.919+0000 7f39737fe700 1 -- 172.21.15.67:0/3044172349 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f3974103eb0 msgr2=0x7f3974075ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:19:20.922 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:20.919+0000 7f3973fff700 1 --2- 172.21.15.67:0/3044172349 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f3974109600 0x7f3974076220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:19:20.922 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:20.919+0000 7f39737fe700 1 --2- 172.21.15.67:0/3044172349 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f3974103eb0 0x7f3974075ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:19:20.922 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:20.919+0000 7f39737fe700 1 -- 172.21.15.67:0/3044172349 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f3974109600 msgr2=0x7f3974076220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:19:20.922 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:20.919+0000 7f39737fe700 1 --2- 172.21.15.67:0/3044172349 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f3974109600 0x7f3974076220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:19:20.922 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:20.919+0000 7f39737fe700 1 -- 172.21.15.67:0/3044172349 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f39741a81d0 con 0x7f39741034f0 2024-05-03T21:19:20.922 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:20.919+0000 7f39737fe700 1 --2- 172.21.15.67:0/3044172349 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f39741034f0 0x7f39740757a0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f3964000c00 tx=0x7f396400f720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T21:19:20.922 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:20.919+0000 7f3970ff9700 1 -- 172.21.15.67:0/3044172349 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3964027070 con 0x7f39741034f0 2024-05-03T21:19:20.922 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:20.919+0000 7f3979be0700 1 -- 172.21.15.67:0/3044172349 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f39741a8460 con 0x7f39741034f0 2024-05-03T21:19:20.922 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:20.919+0000 7f3970ff9700 1 -- 172.21.15.67:0/3044172349 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(28 keys) ==== 1170+0+0 (secure 0 0 0) 0x7f396400fb50 con 0x7f39741034f0 2024-05-03T21:19:20.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:20.919+0000 7f3970ff9700 1 -- 172.21.15.67:0/3044172349 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3964017730 con 0x7f39741034f0 2024-05-03T21:19:20.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:20.920+0000 7f3979be0700 1 -- 172.21.15.67:0/3044172349 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f39741a8910 con 0x7f39741034f0 2024-05-03T21:19:20.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:20.921+0000 7f3979be0700 1 -- 172.21.15.67:0/3044172349 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f397410ba50 con 0x7f39741034f0 2024-05-03T21:19:20.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:20.922+0000 7f3970ff9700 1 -- 172.21.15.67:0/3044172349 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 17) ==== 149703+0+0 (secure 0 0 0) 0x7f396402a070 con 0x7f39741034f0 2024-05-03T21:19:20.924 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:20.923+0000 7f3970ff9700 1 --2- 172.21.15.67:0/3044172349 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f395c0b2910 0x7f395c0b4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:19:20.924 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:20.923+0000 7f3972ffd700 1 --2- 172.21.15.67:0/3044172349 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f395c0b2910 0x7f395c0b4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:19:20.924 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:20.923+0000 7f3970ff9700 1 -- 172.21.15.67:0/3044172349 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7979+0+0 (secure 0 0 0) 0x7f39640e3310 con 0x7f39741034f0 2024-05-03T21:19:20.924 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:20.923+0000 7f3972ffd700 1 --2- 172.21.15.67:0/3044172349 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f395c0b2910 0x7f395c0b4dd0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f397407ab70 tx=0x7f3968005c80 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T21:19:20.927 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:20.926+0000 7f3970ff9700 1 -- 172.21.15.67:0/3044172349 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191839 (secure 0 0 0) 0x7f3964013070 con 0x7f39741034f0 2024-05-03T21:19:21.196 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:21.195+0000 7f3979be0700 1 -- 172.21.15.67:0/3044172349 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f397410b430 con 0x7f39741034f0 2024-05-03T21:19:21.197 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:19:20 smithi067 ceph-mon[29730]: pgmap v305: 97 pgs: 97 active+clean; 607 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 2.7 KiB/s wr, 6 op/s 2024-05-03T21:19:21.197 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:21.196+0000 7f3970ff9700 1 -- 172.21.15.67:0/3044172349 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+11868 (secure 0 0 0) 0x7f39640ab110 con 0x7f39741034f0 2024-05-03T21:19:21.197 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-05-03T21:19:21.198 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14541,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6835/4224826546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":4224826546},{"type":"v1","addr":"172.21.15.67:6835","nonce":4224826546}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14553,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6837/2485231621","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6836","nonce":2485231621},{"type":"v1","addr":"172.21.15.67:6837","nonce":2485231621}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14598,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6839/2106078262","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6838","nonce":2106078262},{"type":"v1","addr":"172.21.15.67:6839","nonce":2106078262}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14610,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6841/1559196215","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6840","nonce":1559196215},{"type":"v1","addr":"172.21.15.67:6841","nonce":1559196215}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24376,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.138:6833/3028022602","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6832","nonce":3028022602},{"type":"v1","addr":"172.21.15.138:6833","nonce":3028022602}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24400,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.138:6835/3342864697","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6834","nonce":3342864697},{"type":"v1","addr":"172.21.15.138:6835","nonce":3342864697}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7}],"filesystems":[{"mdsmap":{"epoch":25,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-05-03T21:18:44.665825+0000","modified":"2024-05-03T21:19:03.775546+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24473,"mds_1":24449,"mds_2":24425},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24406":{"gid":24406,"name":"c","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.138:6837/2203273249","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6836","nonce":2203273249},{"type":"v1","addr":"172.21.15.138:6837","nonce":2203273249}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24407":{"gid":24407,"name":"k","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.82:6833/1127287628","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6832","nonce":1127287628},{"type":"v1","addr":"172.21.15.82:6833","nonce":1127287628}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24418":{"gid":24418,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.138:6839/334382443","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6838","nonce":334382443},{"type":"v1","addr":"172.21.15.138:6839","nonce":334382443}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24425":{"gid":24425,"name":"e","rank":2,"incarnation":21,"state":"up:active","state_seq":14,"addr":"172.21.15.82:6835/280518761","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6834","nonce":280518761},{"type":"v1","addr":"172.21.15.82:6835","nonce":280518761}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24449":{"gid":24449,"name":"h","rank":1,"incarnation":19,"state":"up:active","state_seq":11,"addr":"172.21.15.82:6837/1275987852","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6836","nonce":1275987852},{"type":"v1","addr":"172.21.15.82:6837","nonce":1275987852}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24473":{"gid":24473,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":5,"addr":"172.21.15.82:6839/3644499578","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6838","nonce":3644499578},{"type":"v1","addr":"172.21.15.82:6839","nonce":3644499578}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24473,"qdb_cluster":[24449,24473,24425]},"id":1}]} 2024-05-03T21:19:21.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:21.198+0000 7f3979be0700 1 -- 172.21.15.67:0/3044172349 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f395c0b2910 msgr2=0x7f395c0b4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:19:21.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:21.198+0000 7f3979be0700 1 --2- 172.21.15.67:0/3044172349 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f395c0b2910 0x7f395c0b4dd0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f397407ab70 tx=0x7f3968005c80 comp rx=0 tx=0).stop 2024-05-03T21:19:21.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:21.198+0000 7f3979be0700 1 -- 172.21.15.67:0/3044172349 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f39741034f0 msgr2=0x7f39740757a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:19:21.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:21.198+0000 7f3979be0700 1 --2- 172.21.15.67:0/3044172349 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f39741034f0 0x7f39740757a0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f3964000c00 tx=0x7f396400f720 comp rx=0 tx=0).stop 2024-05-03T21:19:21.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:21.198+0000 7f3979be0700 1 -- 172.21.15.67:0/3044172349 shutdown_connections 2024-05-03T21:19:21.200 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:21.198+0000 7f3979be0700 1 --2- 172.21.15.67:0/3044172349 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f395c0b2910 0x7f395c0b4dd0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:19:21.200 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:21.198+0000 7f3979be0700 1 --2- 172.21.15.67:0/3044172349 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f39741034f0 0x7f39740757a0 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:19:21.200 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:21.198+0000 7f3979be0700 1 --2- 172.21.15.67:0/3044172349 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f3974109600 0x7f3974076220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:19:21.200 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:21.198+0000 7f3979be0700 1 --2- 172.21.15.67:0/3044172349 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f3974103eb0 0x7f3974075ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:19:21.200 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:21.198+0000 7f3979be0700 1 -- 172.21.15.67:0/3044172349 >> 172.21.15.67:0/3044172349 conn(0x7f397407a650 msgr2=0x7f3974101f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T21:19:21.200 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:21.198+0000 7f3979be0700 1 -- 172.21.15.67:0/3044172349 shutdown_connections 2024-05-03T21:19:21.200 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:21.198+0000 7f3979be0700 1 -- 172.21.15.67:0/3044172349 wait complete. 2024-05-03T21:19:21.200 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 25 2024-05-03T21:19:21.245 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:19:20 smithi082 ceph-mon[32933]: pgmap v305: 97 pgs: 97 active+clean; 607 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 2.7 KiB/s wr, 6 op/s 2024-05-03T21:19:21.324 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:19:20 smithi138 ceph-mon[33110]: pgmap v305: 97 pgs: 97 active+clean; 607 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 2.7 KiB/s wr, 6 op/s 2024-05-03T21:19:21.783 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9a047fe7599a549042d06a3fb5b6813a203eabea shell --fsid 713bc57a-0991-11ef-bc97-c7b262605968 -- ceph osd dump --format=json 2024-05-03T21:19:22.035 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/713bc57a-0991-11ef-bc97-c7b262605968/mon.a/config 2024-05-03T21:19:22.169 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:19:21 smithi067 ceph-mon[29730]: from='client.? 172.21.15.67:0/3044172349' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-03T21:19:22.245 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:19:21 smithi082 ceph-mon[32933]: from='client.? 172.21.15.67:0/3044172349' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-03T21:19:22.324 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:19:21 smithi138 ceph-mon[33110]: from='client.? 172.21.15.67:0/3044172349' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-03T21:19:23.221 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:19:22 smithi067 ceph-mon[29730]: pgmap v306: 97 pgs: 97 active+clean; 630 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 7.4 KiB/s rd, 4.9 KiB/s wr, 9 op/s 2024-05-03T21:19:23.245 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:19:22 smithi082 ceph-mon[32933]: pgmap v306: 97 pgs: 97 active+clean; 630 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 7.4 KiB/s rd, 4.9 KiB/s wr, 9 op/s 2024-05-03T21:19:23.324 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:19:22 smithi138 ceph-mon[33110]: pgmap v306: 97 pgs: 97 active+clean; 630 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 7.4 KiB/s rd, 4.9 KiB/s wr, 9 op/s 2024-05-03T21:19:23.792 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:23.789+0000 7f35c5116700 1 -- 172.21.15.67:0/1000846598 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f35c0102fe0 msgr2=0x7f35c0106fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:19:23.792 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:23.789+0000 7f35c5116700 1 --2- 172.21.15.67:0/1000846598 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f35c0102fe0 0x7f35c0106fe0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f35b40099c0 tx=0x7f35b4009cd0 comp rx=0 tx=0).stop 2024-05-03T21:19:23.792 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:23.789+0000 7f35c5116700 1 -- 172.21.15.67:0/1000846598 shutdown_connections 2024-05-03T21:19:23.792 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:23.789+0000 7f35c5116700 1 --2- 172.21.15.67:0/1000846598 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f35c01076c0 0x7f35c010f290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:19:23.792 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:23.789+0000 7f35c5116700 1 --2- 172.21.15.67:0/1000846598 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f35c0102630 0x7f35c0102a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:19:23.792 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:23.789+0000 7f35c5116700 1 --2- 172.21.15.67:0/1000846598 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f35c0102fe0 0x7f35c0106fe0 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:19:23.792 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:23.789+0000 7f35c5116700 1 -- 172.21.15.67:0/1000846598 >> 172.21.15.67:0/1000846598 conn(0x7f35c0079350 msgr2=0x7f35c0079760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T21:19:23.793 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:23.789+0000 7f35c5116700 1 -- 172.21.15.67:0/1000846598 shutdown_connections 2024-05-03T21:19:23.793 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:23.789+0000 7f35c5116700 1 -- 172.21.15.67:0/1000846598 wait complete. 2024-05-03T21:19:23.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:23.790+0000 7f35c5116700 1 Processor -- start 2024-05-03T21:19:23.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:23.790+0000 7f35c5116700 1 -- start start 2024-05-03T21:19:23.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:23.791+0000 7f35c5116700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f35c0102630 0x7f35c019ab60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:19:23.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:23.791+0000 7f35c5116700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f35c0102fe0 0x7f35c019b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:19:23.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:23.791+0000 7f35c5116700 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f35c01076c0 0x7f35c01a5be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:19:23.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:23.791+0000 7f35c5116700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f35c01127c0 con 0x7f35c0102fe0 2024-05-03T21:19:23.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:23.791+0000 7f35c5116700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f35c0112640 con 0x7f35c0102630 2024-05-03T21:19:23.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:23.791+0000 7f35c5116700 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f35c0112940 con 0x7f35c01076c0 2024-05-03T21:19:23.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:23.791+0000 7f35be59c700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f35c0102fe0 0x7f35c019b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:19:23.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:23.791+0000 7f35be59c700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f35c0102fe0 0x7f35c019b0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:39924/0 (socket says 172.21.15.67:39924) 2024-05-03T21:19:23.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:23.791+0000 7f35be59c700 1 -- 172.21.15.67:0/3995083332 learned_addr learned my addr 172.21.15.67:0/3995083332 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-03T21:19:23.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:23.791+0000 7f35bf59e700 1 --2- 172.21.15.67:0/3995083332 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f35c01076c0 0x7f35c01a5be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:19:23.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:23.791+0000 7f35bed9d700 1 --2- 172.21.15.67:0/3995083332 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f35c0102630 0x7f35c019ab60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:19:23.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:23.791+0000 7f35be59c700 1 -- 172.21.15.67:0/3995083332 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f35c01076c0 msgr2=0x7f35c01a5be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:19:23.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:23.791+0000 7f35be59c700 1 --2- 172.21.15.67:0/3995083332 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f35c01076c0 0x7f35c01a5be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:19:23.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:23.791+0000 7f35be59c700 1 -- 172.21.15.67:0/3995083332 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f35c0102630 msgr2=0x7f35c019ab60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:19:23.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:23.791+0000 7f35be59c700 1 --2- 172.21.15.67:0/3995083332 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f35c0102630 0x7f35c019ab60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:19:23.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:23.791+0000 7f35be59c700 1 -- 172.21.15.67:0/3995083332 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f35c01a6120 con 0x7f35c0102fe0 2024-05-03T21:19:23.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:23.791+0000 7f35be59c700 1 --2- 172.21.15.67:0/3995083332 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f35c0102fe0 0x7f35c019b0a0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f35b400b580 tx=0x7f35b400bb20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T21:19:23.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:23.792+0000 7f35a7fff700 1 -- 172.21.15.67:0/3995083332 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35b4027070 con 0x7f35c0102fe0 2024-05-03T21:19:23.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:23.792+0000 7f35a7fff700 1 -- 172.21.15.67:0/3995083332 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(28 keys) ==== 1170+0+0 (secure 0 0 0) 0x7f35b4003f70 con 0x7f35c0102fe0 2024-05-03T21:19:23.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:23.792+0000 7f35a7fff700 1 -- 172.21.15.67:0/3995083332 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35b4022d80 con 0x7f35c0102fe0 2024-05-03T21:19:23.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:23.792+0000 7f35c5116700 1 -- 172.21.15.67:0/3995083332 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f35c01a62f0 con 0x7f35c0102fe0 2024-05-03T21:19:23.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:23.792+0000 7f35c5116700 1 -- 172.21.15.67:0/3995083332 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f35c01a66f0 con 0x7f35c0102fe0 2024-05-03T21:19:23.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:23.793+0000 7f35c5116700 1 -- 172.21.15.67:0/3995083332 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f35c006a190 con 0x7f35c0102fe0 2024-05-03T21:19:23.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:23.794+0000 7f35a7fff700 1 -- 172.21.15.67:0/3995083332 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 17) ==== 149703+0+0 (secure 0 0 0) 0x7f35b402a070 con 0x7f35c0102fe0 2024-05-03T21:19:23.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:23.795+0000 7f35a7fff700 1 --2- 172.21.15.67:0/3995083332 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f35a80b2990 0x7f35a80b4e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:19:23.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:23.795+0000 7f35bed9d700 1 --2- 172.21.15.67:0/3995083332 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f35a80b2990 0x7f35a80b4e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:19:23.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:23.795+0000 7f35a7fff700 1 -- 172.21.15.67:0/3995083332 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7979+0+0 (secure 0 0 0) 0x7f35b40e36b0 con 0x7f35c0102fe0 2024-05-03T21:19:23.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:23.795+0000 7f35bed9d700 1 --2- 172.21.15.67:0/3995083332 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f35a80b2990 0x7f35a80b4e50 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f35b0005ea0 tx=0x7f35b0005e30 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T21:19:23.800 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:23.798+0000 7f35a7fff700 1 -- 172.21.15.67:0/3995083332 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191839 (secure 0 0 0) 0x7f35b4031070 con 0x7f35c0102fe0 2024-05-03T21:19:24.041 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:24.038+0000 7f35c5116700 1 -- 172.21.15.67:0/3995083332 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f35c002e6d0 con 0x7f35c0102fe0 2024-05-03T21:19:24.042 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:24.040+0000 7f35a7fff700 1 -- 172.21.15.67:0/3995083332 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v82) ==== 74+0+22205 (secure 0 0 0) 0x7f35b4013070 con 0x7f35c0102fe0 2024-05-03T21:19:24.043 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-05-03T21:19:24.043 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":82,"fsid":"713bc57a-0991-11ef-bc97-c7b262605968","created":"2024-05-03T21:10:34.070178+0000","modified":"2024-05-03T21:18:44.665863+0000","last_up_change":"2024-05-03T21:17:39.993919+0000","last_in_change":"2024-05-03T21:17:19.978727+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-03T21:13:59.810446+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-05-03T21:18:37.953919+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"82","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-05-03T21:18:41.571707+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"82","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7939f31c-6fb7-42bd-af96-88a448d11bd2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":3692483807},{"type":"v1","addr":"172.21.15.67:6803","nonce":3692483807}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":3692483807},{"type":"v1","addr":"172.21.15.67:6805","nonce":3692483807}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":3692483807},{"type":"v1","addr":"172.21.15.67:6809","nonce":3692483807}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":3692483807},{"type":"v1","addr":"172.21.15.67:6807","nonce":3692483807}]},"public_addr":"172.21.15.67:6803/3692483807","cluster_addr":"172.21.15.67:6805/3692483807","heartbeat_back_addr":"172.21.15.67:6809/3692483807","heartbeat_front_addr":"172.21.15.67:6807/3692483807","state":["exists","up"]},{"osd":1,"uuid":"71858193-05a6-4c2c-9bb7-bc3d811d634e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6800","nonce":1139376397},{"type":"v1","addr":"172.21.15.82:6801","nonce":1139376397}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6802","nonce":1139376397},{"type":"v1","addr":"172.21.15.82:6803","nonce":1139376397}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6806","nonce":1139376397},{"type":"v1","addr":"172.21.15.82:6807","nonce":1139376397}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6804","nonce":1139376397},{"type":"v1","addr":"172.21.15.82:6805","nonce":1139376397}]},"public_addr":"172.21.15.82:6801/1139376397","cluster_addr":"172.21.15.82:6803/1139376397","heartbeat_back_addr":"172.21.15.82:6807/1139376397","heartbeat_front_addr":"172.21.15.82:6805/1139376397","state":["exists","up"]},{"osd":2,"uuid":"f9ece78d-6a19-4d46-9022-6ed184074745","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6800","nonce":2774330123},{"type":"v1","addr":"172.21.15.138:6801","nonce":2774330123}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6802","nonce":2774330123},{"type":"v1","addr":"172.21.15.138:6803","nonce":2774330123}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6806","nonce":2774330123},{"type":"v1","addr":"172.21.15.138:6807","nonce":2774330123}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6804","nonce":2774330123},{"type":"v1","addr":"172.21.15.138:6805","nonce":2774330123}]},"public_addr":"172.21.15.138:6801/2774330123","cluster_addr":"172.21.15.138:6803/2774330123","heartbeat_back_addr":"172.21.15.138:6807/2774330123","heartbeat_front_addr":"172.21.15.138:6805/2774330123","state":["exists","up"]},{"osd":3,"uuid":"3616df71-e7b3-4687-aaba-37b0b7766a3c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":27,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":3915934652},{"type":"v1","addr":"172.21.15.67:6811","nonce":3915934652}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":3915934652},{"type":"v1","addr":"172.21.15.67:6813","nonce":3915934652}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":3915934652},{"type":"v1","addr":"172.21.15.67:6817","nonce":3915934652}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":3915934652},{"type":"v1","addr":"172.21.15.67:6815","nonce":3915934652}]},"public_addr":"172.21.15.67:6811/3915934652","cluster_addr":"172.21.15.67:6813/3915934652","heartbeat_back_addr":"172.21.15.67:6817/3915934652","heartbeat_front_addr":"172.21.15.67:6815/3915934652","state":["exists","up"]},{"osd":4,"uuid":"c3fd0ffd-e23f-4279-b679-9d95a144cf32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6808","nonce":4031818951},{"type":"v1","addr":"172.21.15.82:6809","nonce":4031818951}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6810","nonce":4031818951},{"type":"v1","addr":"172.21.15.82:6811","nonce":4031818951}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6814","nonce":4031818951},{"type":"v1","addr":"172.21.15.82:6815","nonce":4031818951}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6812","nonce":4031818951},{"type":"v1","addr":"172.21.15.82:6813","nonce":4031818951}]},"public_addr":"172.21.15.82:6809/4031818951","cluster_addr":"172.21.15.82:6811/4031818951","heartbeat_back_addr":"172.21.15.82:6815/4031818951","heartbeat_front_addr":"172.21.15.82:6813/4031818951","state":["exists","up"]},{"osd":5,"uuid":"540ef0d9-a6dc-43c6-b09e-e87340ba24cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6808","nonce":2521889499},{"type":"v1","addr":"172.21.15.138:6809","nonce":2521889499}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6810","nonce":2521889499},{"type":"v1","addr":"172.21.15.138:6811","nonce":2521889499}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6814","nonce":2521889499},{"type":"v1","addr":"172.21.15.138:6815","nonce":2521889499}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6812","nonce":2521889499},{"type":"v1","addr":"172.21.15.138:6813","nonce":2521889499}]},"public_addr":"172.21.15.138:6809/2521889499","cluster_addr":"172.21.15.138:6811/2521889499","heartbeat_back_addr":"172.21.15.138:6815/2521889499","heartbeat_front_addr":"172.21.15.138:6813/2521889499","state":["exists","up"]},{"osd":6,"uuid":"49a89fd0-958a-4be9-a8db-6022f946153e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6818","nonce":3269682731},{"type":"v1","addr":"172.21.15.67:6819","nonce":3269682731}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":3269682731},{"type":"v1","addr":"172.21.15.67:6821","nonce":3269682731}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":3269682731},{"type":"v1","addr":"172.21.15.67:6825","nonce":3269682731}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":3269682731},{"type":"v1","addr":"172.21.15.67:6823","nonce":3269682731}]},"public_addr":"172.21.15.67:6819/3269682731","cluster_addr":"172.21.15.67:6821/3269682731","heartbeat_back_addr":"172.21.15.67:6825/3269682731","heartbeat_front_addr":"172.21.15.67:6823/3269682731","state":["exists","up"]},{"osd":7,"uuid":"8fa4cd64-ca6b-48ae-8db0-d369ce6102ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":50,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6816","nonce":3008589976},{"type":"v1","addr":"172.21.15.82:6817","nonce":3008589976}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6818","nonce":3008589976},{"type":"v1","addr":"172.21.15.82:6819","nonce":3008589976}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6822","nonce":3008589976},{"type":"v1","addr":"172.21.15.82:6823","nonce":3008589976}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6820","nonce":3008589976},{"type":"v1","addr":"172.21.15.82:6821","nonce":3008589976}]},"public_addr":"172.21.15.82:6817/3008589976","cluster_addr":"172.21.15.82:6819/3008589976","heartbeat_back_addr":"172.21.15.82:6823/3008589976","heartbeat_front_addr":"172.21.15.82:6821/3008589976","state":["exists","up"]},{"osd":8,"uuid":"fae39087-6c38-4771-a2b9-3f4fecb08edd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":57,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6816","nonce":2223920070},{"type":"v1","addr":"172.21.15.138:6817","nonce":2223920070}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6818","nonce":2223920070},{"type":"v1","addr":"172.21.15.138:6819","nonce":2223920070}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6822","nonce":2223920070},{"type":"v1","addr":"172.21.15.138:6823","nonce":2223920070}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6820","nonce":2223920070},{"type":"v1","addr":"172.21.15.138:6821","nonce":2223920070}]},"public_addr":"172.21.15.138:6817/2223920070","cluster_addr":"172.21.15.138:6819/2223920070","heartbeat_back_addr":"172.21.15.138:6823/2223920070","heartbeat_front_addr":"172.21.15.138:6821/2223920070","state":["exists","up"]},{"osd":9,"uuid":"3d387dbf-0163-41b7-98af-4fc45ed73ad4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":63,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":3939417129},{"type":"v1","addr":"172.21.15.67:6827","nonce":3939417129}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":3939417129},{"type":"v1","addr":"172.21.15.67:6829","nonce":3939417129}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":3939417129},{"type":"v1","addr":"172.21.15.67:6833","nonce":3939417129}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":3939417129},{"type":"v1","addr":"172.21.15.67:6831","nonce":3939417129}]},"public_addr":"172.21.15.67:6827/3939417129","cluster_addr":"172.21.15.67:6829/3939417129","heartbeat_back_addr":"172.21.15.67:6833/3939417129","heartbeat_front_addr":"172.21.15.67:6831/3939417129","state":["exists","up"]},{"osd":10,"uuid":"db0305a0-a83b-474b-837a-fbc239ac1738","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6824","nonce":2505507029},{"type":"v1","addr":"172.21.15.82:6825","nonce":2505507029}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6826","nonce":2505507029},{"type":"v1","addr":"172.21.15.82:6827","nonce":2505507029}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6830","nonce":2505507029},{"type":"v1","addr":"172.21.15.82:6831","nonce":2505507029}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6828","nonce":2505507029},{"type":"v1","addr":"172.21.15.82:6829","nonce":2505507029}]},"public_addr":"172.21.15.82:6825/2505507029","cluster_addr":"172.21.15.82:6827/2505507029","heartbeat_back_addr":"172.21.15.82:6831/2505507029","heartbeat_front_addr":"172.21.15.82:6829/2505507029","state":["exists","up"]},{"osd":11,"uuid":"479c2747-f28e-4704-87e4-d0c31782e2de","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":74,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6824","nonce":2528685018},{"type":"v1","addr":"172.21.15.138:6825","nonce":2528685018}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6826","nonce":2528685018},{"type":"v1","addr":"172.21.15.138:6827","nonce":2528685018}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6830","nonce":2528685018},{"type":"v1","addr":"172.21.15.138:6831","nonce":2528685018}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6828","nonce":2528685018},{"type":"v1","addr":"172.21.15.138:6829","nonce":2528685018}]},"public_addr":"172.21.15.138:6825/2528685018","cluster_addr":"172.21.15.138:6827/2528685018","heartbeat_back_addr":"172.21.15.138:6831/2528685018","heartbeat_front_addr":"172.21.15.138:6829/2528685018","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-03T21:13:07.768773+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-03T21:13:31.870232+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-03T21:13:55.394206+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-03T21:14:20.540281+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-03T21:14:44.318523+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-03T21:15:08.490829+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-03T21:15:34.144734+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-03T21:15:58.717240+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-03T21:16:22.743013+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-03T21:16:49.099588+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-03T21:17:13.321126+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-03T21:17:38.001978+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.67:0/1239643865":"2024-05-04T21:11:33.721416+0000","172.21.15.67:6801/2576113038":"2024-05-04T21:11:33.721416+0000","172.21.15.67:6800/2576113038":"2024-05-04T21:11:33.721416+0000","172.21.15.67:6800/1698171509":"2024-05-04T21:11:07.333651+0000","172.21.15.67:0/2217893923":"2024-05-04T21:11:33.721416+0000","172.21.15.67:0/20864811":"2024-05-04T21:11:07.333651+0000","172.21.15.67:6801/1698171509":"2024-05-04T21:11:07.333651+0000","172.21.15.67:0/773376553":"2024-05-04T21:11:33.721416+0000","172.21.15.67:0/847549114":"2024-05-04T21:11:07.333651+0000","172.21.15.67:0/1291778035":"2024-05-04T21:11:07.333651+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-03T21:19:24.045 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:24.042+0000 7f35c5116700 1 -- 172.21.15.67:0/3995083332 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f35a80b2990 msgr2=0x7f35a80b4e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:19:24.045 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:24.042+0000 7f35c5116700 1 --2- 172.21.15.67:0/3995083332 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f35a80b2990 0x7f35a80b4e50 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f35b0005ea0 tx=0x7f35b0005e30 comp rx=0 tx=0).stop 2024-05-03T21:19:24.045 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:24.042+0000 7f35c5116700 1 -- 172.21.15.67:0/3995083332 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f35c0102fe0 msgr2=0x7f35c019b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:19:24.045 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:24.042+0000 7f35c5116700 1 --2- 172.21.15.67:0/3995083332 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f35c0102fe0 0x7f35c019b0a0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f35b400b580 tx=0x7f35b400bb20 comp rx=0 tx=0).stop 2024-05-03T21:19:24.045 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:24.042+0000 7f35c5116700 1 -- 172.21.15.67:0/3995083332 shutdown_connections 2024-05-03T21:19:24.045 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:24.042+0000 7f35c5116700 1 --2- 172.21.15.67:0/3995083332 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f35a80b2990 0x7f35a80b4e50 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:19:24.045 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:24.042+0000 7f35c5116700 1 --2- 172.21.15.67:0/3995083332 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f35c01076c0 0x7f35c01a5be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:19:24.045 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:24.042+0000 7f35c5116700 1 --2- 172.21.15.67:0/3995083332 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f35c0102630 0x7f35c019ab60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:19:24.045 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:24.042+0000 7f35c5116700 1 --2- 172.21.15.67:0/3995083332 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f35c0102fe0 0x7f35c019b0a0 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:19:24.045 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:24.042+0000 7f35c5116700 1 -- 172.21.15.67:0/3995083332 >> 172.21.15.67:0/3995083332 conn(0x7f35c0079350 msgr2=0x7f35c01003a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T21:19:24.045 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:24.043+0000 7f35c5116700 1 -- 172.21.15.67:0/3995083332 shutdown_connections 2024-05-03T21:19:24.046 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:24.043+0000 7f35c5116700 1 -- 172.21.15.67:0/3995083332 wait complete. 2024-05-03T21:19:24.627 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-05-03T21:19:24.628 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9a047fe7599a549042d06a3fb5b6813a203eabea shell --fsid 713bc57a-0991-11ef-bc97-c7b262605968 -- ceph pg dump --format=json 2024-05-03T21:19:24.876 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/713bc57a-0991-11ef-bc97-c7b262605968/mon.a/config 2024-05-03T21:19:25.221 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:19:24 smithi067 ceph-mon[29730]: pgmap v307: 97 pgs: 97 active+clean; 630 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 2.7 KiB/s wr, 7 op/s 2024-05-03T21:19:25.221 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:19:24 smithi067 ceph-mon[29730]: from='client.? 172.21.15.67:0/3995083332' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-03T21:19:25.245 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:19:24 smithi082 ceph-mon[32933]: pgmap v307: 97 pgs: 97 active+clean; 630 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 2.7 KiB/s wr, 7 op/s 2024-05-03T21:19:25.245 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:19:24 smithi082 ceph-mon[32933]: from='client.? 172.21.15.67:0/3995083332' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-03T21:19:25.324 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:19:24 smithi138 ceph-mon[33110]: pgmap v307: 97 pgs: 97 active+clean; 630 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 2.7 KiB/s wr, 7 op/s 2024-05-03T21:19:25.324 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:19:24 smithi138 ceph-mon[33110]: from='client.? 172.21.15.67:0/3995083332' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-03T21:19:26.667 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:26.657+0000 7f98eccd2700 1 -- 172.21.15.67:0/1121618570 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f98e806bb40 msgr2=0x7f98e8110330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:19:26.667 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:26.657+0000 7f98eccd2700 1 --2- 172.21.15.67:0/1121618570 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f98e806bb40 0x7f98e8110330 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f98e000b0a0 tx=0x7f98e000b3b0 comp rx=0 tx=0).stop 2024-05-03T21:19:26.667 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:26.657+0000 7f98eccd2700 1 -- 172.21.15.67:0/1121618570 shutdown_connections 2024-05-03T21:19:26.667 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:26.657+0000 7f98eccd2700 1 --2- 172.21.15.67:0/1121618570 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f98e806bb40 0x7f98e8110330 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:19:26.667 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:26.657+0000 7f98eccd2700 1 --2- 172.21.15.67:0/1121618570 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f98e8109510 0x7f98e81098f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:19:26.667 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:26.657+0000 7f98eccd2700 1 --2- 172.21.15.67:0/1121618570 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f98e806b180 0x7f98e806b600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:19:26.667 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:26.657+0000 7f98eccd2700 1 -- 172.21.15.67:0/1121618570 >> 172.21.15.67:0/1121618570 conn(0x7f98e8079350 msgr2=0x7f98e8079760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T21:19:26.667 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:26.657+0000 7f98eccd2700 1 -- 172.21.15.67:0/1121618570 shutdown_connections 2024-05-03T21:19:26.667 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:26.657+0000 7f98eccd2700 1 -- 172.21.15.67:0/1121618570 wait complete. 2024-05-03T21:19:26.667 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:26.658+0000 7f98eccd2700 1 Processor -- start 2024-05-03T21:19:26.667 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:26.658+0000 7f98eccd2700 1 -- start start 2024-05-03T21:19:26.667 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:26.658+0000 7f98eccd2700 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f98e806b180 0x7f98e8102470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:19:26.667 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:26.658+0000 7f98eccd2700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f98e806bb40 0x7f98e81029b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:19:26.667 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:26.658+0000 7f98eccd2700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f98e8109510 0x7f98e8102ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:19:26.668 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:26.658+0000 7f98eccd2700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f98e81138d0 con 0x7f98e8109510 2024-05-03T21:19:26.668 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:26.658+0000 7f98eccd2700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f98e8113750 con 0x7f98e806bb40 2024-05-03T21:19:26.668 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:26.658+0000 7f98eccd2700 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f98e8113a50 con 0x7f98e806b180 2024-05-03T21:19:26.668 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:26.658+0000 7f98e6d9d700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f98e8109510 0x7f98e8102ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:19:26.668 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:26.658+0000 7f98e6d9d700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f98e8109510 0x7f98e8102ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:50232/0 (socket says 172.21.15.67:50232) 2024-05-03T21:19:26.668 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:26.658+0000 7f98e6d9d700 1 -- 172.21.15.67:0/2977073597 learned_addr learned my addr 172.21.15.67:0/2977073597 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-03T21:19:26.668 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:26.658+0000 7f98e659c700 1 --2- 172.21.15.67:0/2977073597 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f98e806b180 0x7f98e8102470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:19:26.668 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:26.659+0000 7f98e5d9b700 1 --2- 172.21.15.67:0/2977073597 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f98e806bb40 0x7f98e81029b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:19:26.668 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:26.659+0000 7f98e6d9d700 1 -- 172.21.15.67:0/2977073597 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f98e806b180 msgr2=0x7f98e8102470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:19:26.668 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:26.659+0000 7f98e6d9d700 1 --2- 172.21.15.67:0/2977073597 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f98e806b180 0x7f98e8102470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:19:26.668 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:26.659+0000 7f98e6d9d700 1 -- 172.21.15.67:0/2977073597 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f98e806bb40 msgr2=0x7f98e81029b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:19:26.668 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:26.659+0000 7f98e6d9d700 1 --2- 172.21.15.67:0/2977073597 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f98e806bb40 0x7f98e81029b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:19:26.669 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:26.659+0000 7f98e6d9d700 1 -- 172.21.15.67:0/2977073597 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f98e81a81c0 con 0x7f98e8109510 2024-05-03T21:19:26.669 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:26.659+0000 7f98e6d9d700 1 --2- 172.21.15.67:0/2977073597 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f98e8109510 0x7f98e8102ef0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f98e0006070 tx=0x7f98e0004ce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T21:19:26.669 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:26.659+0000 7f98d77fe700 1 -- 172.21.15.67:0/2977073597 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98e0021070 con 0x7f98e8109510 2024-05-03T21:19:26.669 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:26.659+0000 7f98eccd2700 1 -- 172.21.15.67:0/2977073597 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f98e81a8450 con 0x7f98e8109510 2024-05-03T21:19:26.669 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:26.659+0000 7f98d77fe700 1 -- 172.21.15.67:0/2977073597 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(28 keys) ==== 1170+0+0 (secure 0 0 0) 0x7f98e0003c00 con 0x7f98e8109510 2024-05-03T21:19:26.669 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:26.659+0000 7f98d77fe700 1 -- 172.21.15.67:0/2977073597 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98e001d450 con 0x7f98e8109510 2024-05-03T21:19:26.669 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:26.659+0000 7f98eccd2700 1 -- 172.21.15.67:0/2977073597 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f98e81a8900 con 0x7f98e8109510 2024-05-03T21:19:26.669 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:26.660+0000 7f98eccd2700 1 -- 172.21.15.67:0/2977073597 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f98e810b9f0 con 0x7f98e8109510 2024-05-03T21:19:26.669 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:26.662+0000 7f98d77fe700 1 -- 172.21.15.67:0/2977073597 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 17) ==== 149703+0+0 (secure 0 0 0) 0x7f98e0003db0 con 0x7f98e8109510 2024-05-03T21:19:26.669 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:26.662+0000 7f98d77fe700 1 --2- 172.21.15.67:0/2977073597 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f98d00b2990 0x7f98d00b4e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:19:26.669 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:26.662+0000 7f98d77fe700 1 -- 172.21.15.67:0/2977073597 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7979+0+0 (secure 0 0 0) 0x7f98e00e3180 con 0x7f98e8109510 2024-05-03T21:19:26.669 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:26.663+0000 7f98e659c700 1 --2- 172.21.15.67:0/2977073597 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f98d00b2990 0x7f98d00b4e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:19:26.669 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:26.663+0000 7f98e659c700 1 --2- 172.21.15.67:0/2977073597 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f98d00b2990 0x7f98d00b4e50 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f98d80045b0 tx=0x7f98d800b3c0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T21:19:26.670 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:26.667+0000 7f98d77fe700 1 -- 172.21.15.67:0/2977073597 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191839 (secure 0 0 0) 0x7f98e00aaf80 con 0x7f98e8109510 2024-05-03T21:19:26.879 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:26.876+0000 7f98eccd2700 1 -- 172.21.15.67:0/2977073597 --> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f98e80638e0 con 0x7f98d00b2990 2024-05-03T21:19:26.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:26.884+0000 7f98d77fe700 1 -- 172.21.15.67:0/2977073597 <== mgr.14150 v2:172.21.15.67:6800/604451592 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+258695 (secure 0 0 0) 0x7f98e80638e0 con 0x7f98d00b2990 2024-05-03T21:19:26.891 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-05-03T21:19:26.893 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:26.887+0000 7f98eccd2700 1 -- 172.21.15.67:0/2977073597 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f98d00b2990 msgr2=0x7f98d00b4e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:19:26.893 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:26.887+0000 7f98eccd2700 1 --2- 172.21.15.67:0/2977073597 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f98d00b2990 0x7f98d00b4e50 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f98d80045b0 tx=0x7f98d800b3c0 comp rx=0 tx=0).stop 2024-05-03T21:19:26.893 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:26.887+0000 7f98eccd2700 1 -- 172.21.15.67:0/2977073597 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f98e8109510 msgr2=0x7f98e8102ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:19:26.893 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:26.887+0000 7f98eccd2700 1 --2- 172.21.15.67:0/2977073597 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f98e8109510 0x7f98e8102ef0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f98e0006070 tx=0x7f98e0004ce0 comp rx=0 tx=0).stop 2024-05-03T21:19:26.893 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:26.887+0000 7f98eccd2700 1 -- 172.21.15.67:0/2977073597 shutdown_connections 2024-05-03T21:19:26.894 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:26.887+0000 7f98eccd2700 1 --2- 172.21.15.67:0/2977073597 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f98d00b2990 0x7f98d00b4e50 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:19:26.894 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:26.887+0000 7f98eccd2700 1 --2- 172.21.15.67:0/2977073597 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f98e8109510 0x7f98e8102ef0 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:19:26.894 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:26.887+0000 7f98eccd2700 1 --2- 172.21.15.67:0/2977073597 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f98e806b180 0x7f98e8102470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:19:26.894 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:26.887+0000 7f98eccd2700 1 --2- 172.21.15.67:0/2977073597 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f98e806bb40 0x7f98e81029b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:19:26.894 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:26.887+0000 7f98eccd2700 1 -- 172.21.15.67:0/2977073597 >> 172.21.15.67:0/2977073597 conn(0x7f98e8079350 msgr2=0x7f98e8100e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T21:19:26.894 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:26.887+0000 7f98eccd2700 1 -- 172.21.15.67:0/2977073597 shutdown_connections 2024-05-03T21:19:26.894 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:26.887+0000 7f98eccd2700 1 -- 172.21.15.67:0/2977073597 wait complete. 2024-05-03T21:19:26.894 INFO:teuthology.orchestra.run.smithi067.stderr:dumped all 2024-05-03T21:19:27.221 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:19:26 smithi067 ceph-mon[29730]: pgmap v308: 97 pgs: 97 active+clean; 630 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 2.7 KiB/s wr, 7 op/s 2024-05-03T21:19:27.245 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:19:26 smithi082 ceph-mon[32933]: pgmap v308: 97 pgs: 97 active+clean; 630 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 2.7 KiB/s wr, 7 op/s 2024-05-03T21:19:27.324 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:19:26 smithi138 ceph-mon[33110]: pgmap v308: 97 pgs: 97 active+clean; 630 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 2.7 KiB/s wr, 7 op/s 2024-05-03T21:19:27.535 INFO:teuthology.orchestra.run.smithi067.stdout:{"pg_ready":true,"pg_map":{"version":308,"stamp":"2024-05-03T21:19:25.815911+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":645104,"num_objects":64,"num_object_clones":0,"num_object_copies":192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":64,"num_whiteouts":0,"num_read":243,"num_read_kb":207,"num_write":281,"num_write_kb":2249,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":228,"ondisk_log_size":228,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":291,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":382372,"kb_used_data":12420,"kb_used_omap":18,"kb_used_meta":369837,"kb_avail":1124559452,"statfs":{"total":1151940427776,"available":1151548878848,"internally_reserved":0,"allocated":12718080,"data_stored":7028244,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19143,"internal_metadata":378713401},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":27897,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":77,"num_read_kb":68,"num_write":16,"num_write_kb":32,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003103"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":16,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-03T21:18:44.808490+0000","last_change":"2024-05-03T21:18:42.819790+0000","last_active":"2024-05-03T21:18:44.808490+0000","last_peered":"2024-05-03T21:18:44.808490+0000","last_clean":"2024-05-03T21:18:44.808490+0000","last_became_active":"2024-05-03T21:18:42.818989+0000","last_became_peered":"2024-05-03T21:18:42.818989+0000","last_unstale":"2024-05-03T21:18:44.808490+0000","last_undegraded":"2024-05-03T21:18:44.808490+0000","last_fullsized":"2024-05-03T21:18:44.808490+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_clean_scrub_stamp":"2024-05-03T21:18:41.788726+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T08:38:43.699198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":16,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-03T21:18:44.825312+0000","last_change":"2024-05-03T21:18:42.812454+0000","last_active":"2024-05-03T21:18:44.825312+0000","last_peered":"2024-05-03T21:18:44.825312+0000","last_clean":"2024-05-03T21:18:44.825312+0000","last_became_active":"2024-05-03T21:18:42.812140+0000","last_became_peered":"2024-05-03T21:18:42.812140+0000","last_unstale":"2024-05-03T21:18:44.825312+0000","last_undegraded":"2024-05-03T21:18:44.825312+0000","last_fullsized":"2024-05-03T21:18:44.825312+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_clean_scrub_stamp":"2024-05-03T21:18:41.788726+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-04T22:30:36.057079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":16,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-03T21:18:44.824785+0000","last_change":"2024-05-03T21:18:42.815547+0000","last_active":"2024-05-03T21:18:44.824785+0000","last_peered":"2024-05-03T21:18:44.824785+0000","last_clean":"2024-05-03T21:18:44.824785+0000","last_became_active":"2024-05-03T21:18:42.815192+0000","last_became_peered":"2024-05-03T21:18:42.815192+0000","last_unstale":"2024-05-03T21:18:44.824785+0000","last_undegraded":"2024-05-03T21:18:44.824785+0000","last_fullsized":"2024-05-03T21:18:44.824785+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_clean_scrub_stamp":"2024-05-03T21:18:41.788726+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T04:15:05.786275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":16,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-03T21:18:44.826855+0000","last_change":"2024-05-03T21:18:42.819376+0000","last_active":"2024-05-03T21:18:44.826855+0000","last_peered":"2024-05-03T21:18:44.826855+0000","last_clean":"2024-05-03T21:18:44.826855+0000","last_became_active":"2024-05-03T21:18:42.819027+0000","last_became_peered":"2024-05-03T21:18:42.819027+0000","last_unstale":"2024-05-03T21:18:44.826855+0000","last_undegraded":"2024-05-03T21:18:44.826855+0000","last_fullsized":"2024-05-03T21:18:44.826855+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_clean_scrub_stamp":"2024-05-03T21:18:41.788726+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T01:41:06.639885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":16,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-03T21:18:44.823042+0000","last_change":"2024-05-03T21:18:42.807225+0000","last_active":"2024-05-03T21:18:44.823042+0000","last_peered":"2024-05-03T21:18:44.823042+0000","last_clean":"2024-05-03T21:18:44.823042+0000","last_became_active":"2024-05-03T21:18:42.806856+0000","last_became_peered":"2024-05-03T21:18:42.806856+0000","last_unstale":"2024-05-03T21:18:44.823042+0000","last_undegraded":"2024-05-03T21:18:44.823042+0000","last_fullsized":"2024-05-03T21:18:44.823042+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_clean_scrub_stamp":"2024-05-03T21:18:41.788726+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-04T23:59:15.221592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":16,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-03T21:18:44.808536+0000","last_change":"2024-05-03T21:18:42.819099+0000","last_active":"2024-05-03T21:18:44.808536+0000","last_peered":"2024-05-03T21:18:44.808536+0000","last_clean":"2024-05-03T21:18:44.808536+0000","last_became_active":"2024-05-03T21:18:42.818775+0000","last_became_peered":"2024-05-03T21:18:42.818775+0000","last_unstale":"2024-05-03T21:18:44.808536+0000","last_undegraded":"2024-05-03T21:18:44.808536+0000","last_fullsized":"2024-05-03T21:18:44.808536+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_clean_scrub_stamp":"2024-05-03T21:18:41.788726+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T03:46:43.762816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":16,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-03T21:18:44.826168+0000","last_change":"2024-05-03T21:18:42.811779+0000","last_active":"2024-05-03T21:18:44.826168+0000","last_peered":"2024-05-03T21:18:44.826168+0000","last_clean":"2024-05-03T21:18:44.826168+0000","last_became_active":"2024-05-03T21:18:42.811376+0000","last_became_peered":"2024-05-03T21:18:42.811376+0000","last_unstale":"2024-05-03T21:18:44.826168+0000","last_undegraded":"2024-05-03T21:18:44.826168+0000","last_fullsized":"2024-05-03T21:18:44.826168+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_clean_scrub_stamp":"2024-05-03T21:18:41.788726+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-04T22:08:00.629106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":16,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-03T21:18:44.827151+0000","last_change":"2024-05-03T21:18:42.820643+0000","last_active":"2024-05-03T21:18:44.827151+0000","last_peered":"2024-05-03T21:18:44.827151+0000","last_clean":"2024-05-03T21:18:44.827151+0000","last_became_active":"2024-05-03T21:18:42.820317+0000","last_became_peered":"2024-05-03T21:18:42.820317+0000","last_unstale":"2024-05-03T21:18:44.827151+0000","last_undegraded":"2024-05-03T21:18:44.827151+0000","last_fullsized":"2024-05-03T21:18:44.827151+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_clean_scrub_stamp":"2024-05-03T21:18:41.788726+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-04T21:39:44.319396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":16,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-03T21:18:44.824601+0000","last_change":"2024-05-03T21:18:42.819895+0000","last_active":"2024-05-03T21:18:44.824601+0000","last_peered":"2024-05-03T21:18:44.824601+0000","last_clean":"2024-05-03T21:18:44.824601+0000","last_became_active":"2024-05-03T21:18:42.818360+0000","last_became_peered":"2024-05-03T21:18:42.818360+0000","last_unstale":"2024-05-03T21:18:44.824601+0000","last_undegraded":"2024-05-03T21:18:44.824601+0000","last_fullsized":"2024-05-03T21:18:44.824601+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_clean_scrub_stamp":"2024-05-03T21:18:41.788726+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T06:49:09.368610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":16,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-03T21:18:44.808891+0000","last_change":"2024-05-03T21:18:42.817064+0000","last_active":"2024-05-03T21:18:44.808891+0000","last_peered":"2024-05-03T21:18:44.808891+0000","last_clean":"2024-05-03T21:18:44.808891+0000","last_became_active":"2024-05-03T21:18:42.816727+0000","last_became_peered":"2024-05-03T21:18:42.816727+0000","last_unstale":"2024-05-03T21:18:44.808891+0000","last_undegraded":"2024-05-03T21:18:44.808891+0000","last_fullsized":"2024-05-03T21:18:44.808891+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_clean_scrub_stamp":"2024-05-03T21:18:41.788726+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T06:30:20.093765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":16,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-03T21:18:44.827690+0000","last_change":"2024-05-03T21:18:42.818728+0000","last_active":"2024-05-03T21:18:44.827690+0000","last_peered":"2024-05-03T21:18:44.827690+0000","last_clean":"2024-05-03T21:18:44.827690+0000","last_became_active":"2024-05-03T21:18:42.818250+0000","last_became_peered":"2024-05-03T21:18:42.818250+0000","last_unstale":"2024-05-03T21:18:44.827690+0000","last_undegraded":"2024-05-03T21:18:44.827690+0000","last_fullsized":"2024-05-03T21:18:44.827690+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_clean_scrub_stamp":"2024-05-03T21:18:41.788726+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T03:52:04.613842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":16,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-03T21:18:44.826558+0000","last_change":"2024-05-03T21:18:42.816897+0000","last_active":"2024-05-03T21:18:44.826558+0000","last_peered":"2024-05-03T21:18:44.826558+0000","last_clean":"2024-05-03T21:18:44.826558+0000","last_became_active":"2024-05-03T21:18:42.816274+0000","last_became_peered":"2024-05-03T21:18:42.816274+0000","last_unstale":"2024-05-03T21:18:44.826558+0000","last_undegraded":"2024-05-03T21:18:44.826558+0000","last_fullsized":"2024-05-03T21:18:44.826558+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_clean_scrub_stamp":"2024-05-03T21:18:41.788726+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-04T21:48:17.744413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":16,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-03T21:18:44.822961+0000","last_change":"2024-05-03T21:18:42.820523+0000","last_active":"2024-05-03T21:18:44.822961+0000","last_peered":"2024-05-03T21:18:44.822961+0000","last_clean":"2024-05-03T21:18:44.822961+0000","last_became_active":"2024-05-03T21:18:42.820133+0000","last_became_peered":"2024-05-03T21:18:42.820133+0000","last_unstale":"2024-05-03T21:18:44.822961+0000","last_undegraded":"2024-05-03T21:18:44.822961+0000","last_fullsized":"2024-05-03T21:18:44.822961+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_clean_scrub_stamp":"2024-05-03T21:18:41.788726+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-04T23:25:34.304369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":16,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-03T21:18:44.823838+0000","last_change":"2024-05-03T21:18:42.814467+0000","last_active":"2024-05-03T21:18:44.823838+0000","last_peered":"2024-05-03T21:18:44.823838+0000","last_clean":"2024-05-03T21:18:44.823838+0000","last_became_active":"2024-05-03T21:18:42.814124+0000","last_became_peered":"2024-05-03T21:18:42.814124+0000","last_unstale":"2024-05-03T21:18:44.823838+0000","last_undegraded":"2024-05-03T21:18:44.823838+0000","last_fullsized":"2024-05-03T21:18:44.823838+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_clean_scrub_stamp":"2024-05-03T21:18:41.788726+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T08:41:03.567620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":16,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-03T21:18:44.823508+0000","last_change":"2024-05-03T21:18:42.822158+0000","last_active":"2024-05-03T21:18:44.823508+0000","last_peered":"2024-05-03T21:18:44.823508+0000","last_clean":"2024-05-03T21:18:44.823508+0000","last_became_active":"2024-05-03T21:18:42.821793+0000","last_became_peered":"2024-05-03T21:18:42.821793+0000","last_unstale":"2024-05-03T21:18:44.823508+0000","last_undegraded":"2024-05-03T21:18:44.823508+0000","last_fullsized":"2024-05-03T21:18:44.823508+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_clean_scrub_stamp":"2024-05-03T21:18:41.788726+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T06:44:09.269513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":16,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-03T21:18:44.822931+0000","last_change":"2024-05-03T21:18:42.813904+0000","last_active":"2024-05-03T21:18:44.822931+0000","last_peered":"2024-05-03T21:18:44.822931+0000","last_clean":"2024-05-03T21:18:44.822931+0000","last_became_active":"2024-05-03T21:18:42.813488+0000","last_became_peered":"2024-05-03T21:18:42.813488+0000","last_unstale":"2024-05-03T21:18:44.822931+0000","last_undegraded":"2024-05-03T21:18:44.822931+0000","last_fullsized":"2024-05-03T21:18:44.822931+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_clean_scrub_stamp":"2024-05-03T21:18:41.788726+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T00:52:20.288003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":16,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-03T21:18:44.808670+0000","last_change":"2024-05-03T21:18:42.820157+0000","last_active":"2024-05-03T21:18:44.808670+0000","last_peered":"2024-05-03T21:18:44.808670+0000","last_clean":"2024-05-03T21:18:44.808670+0000","last_became_active":"2024-05-03T21:18:42.819450+0000","last_became_peered":"2024-05-03T21:18:42.819450+0000","last_unstale":"2024-05-03T21:18:44.808670+0000","last_undegraded":"2024-05-03T21:18:44.808670+0000","last_fullsized":"2024-05-03T21:18:44.808670+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_clean_scrub_stamp":"2024-05-03T21:18:41.788726+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T04:34:19.252499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":16,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-03T21:18:44.825529+0000","last_change":"2024-05-03T21:18:42.816582+0000","last_active":"2024-05-03T21:18:44.825529+0000","last_peered":"2024-05-03T21:18:44.825529+0000","last_clean":"2024-05-03T21:18:44.825529+0000","last_became_active":"2024-05-03T21:18:42.816247+0000","last_became_peered":"2024-05-03T21:18:42.816247+0000","last_unstale":"2024-05-03T21:18:44.825529+0000","last_undegraded":"2024-05-03T21:18:44.825529+0000","last_fullsized":"2024-05-03T21:18:44.825529+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_clean_scrub_stamp":"2024-05-03T21:18:41.788726+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T00:09:02.620176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":16,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-03T21:18:44.824827+0000","last_change":"2024-05-03T21:18:42.821587+0000","last_active":"2024-05-03T21:18:44.824827+0000","last_peered":"2024-05-03T21:18:44.824827+0000","last_clean":"2024-05-03T21:18:44.824827+0000","last_became_active":"2024-05-03T21:18:42.821236+0000","last_became_peered":"2024-05-03T21:18:42.821236+0000","last_unstale":"2024-05-03T21:18:44.824827+0000","last_undegraded":"2024-05-03T21:18:44.824827+0000","last_fullsized":"2024-05-03T21:18:44.824827+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_clean_scrub_stamp":"2024-05-03T21:18:41.788726+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-04T22:23:46.746787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"82'9","reported_seq":31,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-03T21:18:57.874729+0000","last_change":"2024-05-03T21:18:39.792001+0000","last_active":"2024-05-03T21:18:57.874729+0000","last_peered":"2024-05-03T21:18:57.874729+0000","last_clean":"2024-05-03T21:18:57.874729+0000","last_became_active":"2024-05-03T21:18:39.791644+0000","last_became_peered":"2024-05-03T21:18:39.791644+0000","last_unstale":"2024-05-03T21:18:57.874729+0000","last_undegraded":"2024-05-03T21:18:57.874729+0000","last_fullsized":"2024-05-03T21:18:57.874729+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T21:18:38.766727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T21:18:38.766727+0000","last_clean_scrub_stamp":"2024-05-03T21:18:38.766727+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-04T23:15:36.169243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":16,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-03T21:18:44.827474+0000","last_change":"2024-05-03T21:18:42.815115+0000","last_active":"2024-05-03T21:18:44.827474+0000","last_peered":"2024-05-03T21:18:44.827474+0000","last_clean":"2024-05-03T21:18:44.827474+0000","last_became_active":"2024-05-03T21:18:42.814734+0000","last_became_peered":"2024-05-03T21:18:42.814734+0000","last_unstale":"2024-05-03T21:18:44.827474+0000","last_undegraded":"2024-05-03T21:18:44.827474+0000","last_fullsized":"2024-05-03T21:18:44.827474+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_clean_scrub_stamp":"2024-05-03T21:18:41.788726+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-04T23:25:18.232982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"82'2","reported_seq":23,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-03T21:18:44.829754+0000","last_change":"2024-05-03T21:18:39.814107+0000","last_active":"2024-05-03T21:18:44.829754+0000","last_peered":"2024-05-03T21:18:44.829754+0000","last_clean":"2024-05-03T21:18:44.829754+0000","last_became_active":"2024-05-03T21:18:39.813615+0000","last_became_peered":"2024-05-03T21:18:39.813615+0000","last_unstale":"2024-05-03T21:18:44.829754+0000","last_undegraded":"2024-05-03T21:18:44.829754+0000","last_fullsized":"2024-05-03T21:18:44.829754+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T21:18:38.766727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T21:18:38.766727+0000","last_clean_scrub_stamp":"2024-05-03T21:18:38.766727+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-04T23:24:00.914805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":16,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-03T21:18:44.825794+0000","last_change":"2024-05-03T21:18:42.816925+0000","last_active":"2024-05-03T21:18:44.825794+0000","last_peered":"2024-05-03T21:18:44.825794+0000","last_clean":"2024-05-03T21:18:44.825794+0000","last_became_active":"2024-05-03T21:18:42.814731+0000","last_became_peered":"2024-05-03T21:18:42.814731+0000","last_unstale":"2024-05-03T21:18:44.825794+0000","last_undegraded":"2024-05-03T21:18:44.825794+0000","last_fullsized":"2024-05-03T21:18:44.825794+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_clean_scrub_stamp":"2024-05-03T21:18:41.788726+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T00:18:26.859784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"82'4","reported_seq":26,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-03T21:18:57.873396+0000","last_change":"2024-05-03T21:18:39.792101+0000","last_active":"2024-05-03T21:18:57.873396+0000","last_peered":"2024-05-03T21:18:57.873396+0000","last_clean":"2024-05-03T21:18:57.873396+0000","last_became_active":"2024-05-03T21:18:39.791725+0000","last_became_peered":"2024-05-03T21:18:39.791725+0000","last_unstale":"2024-05-03T21:18:57.873396+0000","last_undegraded":"2024-05-03T21:18:57.873396+0000","last_fullsized":"2024-05-03T21:18:57.873396+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T21:18:38.766727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T21:18:38.766727+0000","last_clean_scrub_stamp":"2024-05-03T21:18:38.766727+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T05:09:42.900318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":16,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-03T21:18:44.823496+0000","last_change":"2024-05-03T21:18:42.823318+0000","last_active":"2024-05-03T21:18:44.823496+0000","last_peered":"2024-05-03T21:18:44.823496+0000","last_clean":"2024-05-03T21:18:44.823496+0000","last_became_active":"2024-05-03T21:18:42.823011+0000","last_became_peered":"2024-05-03T21:18:42.823011+0000","last_unstale":"2024-05-03T21:18:44.823496+0000","last_undegraded":"2024-05-03T21:18:44.823496+0000","last_fullsized":"2024-05-03T21:18:44.823496+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_clean_scrub_stamp":"2024-05-03T21:18:41.788726+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T01:04:51.967269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"82'2","reported_seq":24,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-03T21:18:44.828425+0000","last_change":"2024-05-03T21:18:39.787730+0000","last_active":"2024-05-03T21:18:44.828425+0000","last_peered":"2024-05-03T21:18:44.828425+0000","last_clean":"2024-05-03T21:18:44.828425+0000","last_became_active":"2024-05-03T21:18:39.787403+0000","last_became_peered":"2024-05-03T21:18:39.787403+0000","last_unstale":"2024-05-03T21:18:44.828425+0000","last_undegraded":"2024-05-03T21:18:44.828425+0000","last_fullsized":"2024-05-03T21:18:44.828425+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T21:18:38.766727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T21:18:38.766727+0000","last_clean_scrub_stamp":"2024-05-03T21:18:38.766727+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T06:28:41.454461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":16,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-03T21:18:44.810654+0000","last_change":"2024-05-03T21:18:42.818987+0000","last_active":"2024-05-03T21:18:44.810654+0000","last_peered":"2024-05-03T21:18:44.810654+0000","last_clean":"2024-05-03T21:18:44.810654+0000","last_became_active":"2024-05-03T21:18:42.818645+0000","last_became_peered":"2024-05-03T21:18:42.818645+0000","last_unstale":"2024-05-03T21:18:44.810654+0000","last_undegraded":"2024-05-03T21:18:44.810654+0000","last_fullsized":"2024-05-03T21:18:44.810654+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_clean_scrub_stamp":"2024-05-03T21:18:41.788726+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T02:26:27.470390+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"82'6","reported_seq":53,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-03T21:19:20.922623+0000","last_change":"2024-05-03T21:18:39.793807+0000","last_active":"2024-05-03T21:19:20.922623+0000","last_peered":"2024-05-03T21:19:20.922623+0000","last_clean":"2024-05-03T21:19:20.922623+0000","last_became_active":"2024-05-03T21:18:39.793447+0000","last_became_peered":"2024-05-03T21:18:39.793447+0000","last_unstale":"2024-05-03T21:19:20.922623+0000","last_undegraded":"2024-05-03T21:19:20.922623+0000","last_fullsized":"2024-05-03T21:19:20.922623+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T21:18:38.766727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T21:18:38.766727+0000","last_clean_scrub_stamp":"2024-05-03T21:18:38.766727+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T06:44:55.036889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1013,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":25,"num_read_kb":3,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":16,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-03T21:18:44.825008+0000","last_change":"2024-05-03T21:18:42.807486+0000","last_active":"2024-05-03T21:18:44.825008+0000","last_peered":"2024-05-03T21:18:44.825008+0000","last_clean":"2024-05-03T21:18:44.825008+0000","last_became_active":"2024-05-03T21:18:42.807120+0000","last_became_peered":"2024-05-03T21:18:42.807120+0000","last_unstale":"2024-05-03T21:18:44.825008+0000","last_undegraded":"2024-05-03T21:18:44.825008+0000","last_fullsized":"2024-05-03T21:18:44.825008+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_clean_scrub_stamp":"2024-05-03T21:18:41.788726+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T05:32:38.621330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1a","version":"82'3","reported_seq":24,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-03T21:18:57.873629+0000","last_change":"2024-05-03T21:18:39.791943+0000","last_active":"2024-05-03T21:18:57.873629+0000","last_peered":"2024-05-03T21:18:57.873629+0000","last_clean":"2024-05-03T21:18:57.873629+0000","last_became_active":"2024-05-03T21:18:39.791587+0000","last_became_peered":"2024-05-03T21:18:39.791587+0000","last_unstale":"2024-05-03T21:18:57.873629+0000","last_undegraded":"2024-05-03T21:18:57.873629+0000","last_fullsized":"2024-05-03T21:18:57.873629+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T21:18:38.766727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T21:18:38.766727+0000","last_clean_scrub_stamp":"2024-05-03T21:18:38.766727+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T05:01:39.458019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":16,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-03T21:18:44.810502+0000","last_change":"2024-05-03T21:18:42.815435+0000","last_active":"2024-05-03T21:18:44.810502+0000","last_peered":"2024-05-03T21:18:44.810502+0000","last_clean":"2024-05-03T21:18:44.810502+0000","last_became_active":"2024-05-03T21:18:42.815064+0000","last_became_peered":"2024-05-03T21:18:42.815064+0000","last_unstale":"2024-05-03T21:18:44.810502+0000","last_undegraded":"2024-05-03T21:18:44.810502+0000","last_fullsized":"2024-05-03T21:18:44.810502+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_clean_scrub_stamp":"2024-05-03T21:18:41.788726+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T06:08:50.435421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"82'2","reported_seq":23,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-03T21:18:57.873460+0000","last_change":"2024-05-03T21:18:39.784859+0000","last_active":"2024-05-03T21:18:57.873460+0000","last_peered":"2024-05-03T21:18:57.873460+0000","last_clean":"2024-05-03T21:18:57.873460+0000","last_became_active":"2024-05-03T21:18:39.784428+0000","last_became_peered":"2024-05-03T21:18:39.784428+0000","last_unstale":"2024-05-03T21:18:57.873460+0000","last_undegraded":"2024-05-03T21:18:57.873460+0000","last_fullsized":"2024-05-03T21:18:57.873460+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T21:18:38.766727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T21:18:38.766727+0000","last_clean_scrub_stamp":"2024-05-03T21:18:38.766727+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T05:56:18.368399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":16,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-03T21:18:44.827308+0000","last_change":"2024-05-03T21:18:42.813285+0000","last_active":"2024-05-03T21:18:44.827308+0000","last_peered":"2024-05-03T21:18:44.827308+0000","last_clean":"2024-05-03T21:18:44.827308+0000","last_became_active":"2024-05-03T21:18:42.812954+0000","last_became_peered":"2024-05-03T21:18:42.812954+0000","last_unstale":"2024-05-03T21:18:44.827308+0000","last_undegraded":"2024-05-03T21:18:44.827308+0000","last_fullsized":"2024-05-03T21:18:44.827308+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_clean_scrub_stamp":"2024-05-03T21:18:41.788726+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T00:19:26.841514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"82'1","reported_seq":23,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-03T21:18:44.810087+0000","last_change":"2024-05-03T21:18:39.793485+0000","last_active":"2024-05-03T21:18:44.810087+0000","last_peered":"2024-05-03T21:18:44.810087+0000","last_clean":"2024-05-03T21:18:44.810087+0000","last_became_active":"2024-05-03T21:18:39.793055+0000","last_became_peered":"2024-05-03T21:18:39.793055+0000","last_unstale":"2024-05-03T21:18:44.810087+0000","last_undegraded":"2024-05-03T21:18:44.810087+0000","last_fullsized":"2024-05-03T21:18:44.810087+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T21:18:38.766727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T21:18:38.766727+0000","last_clean_scrub_stamp":"2024-05-03T21:18:38.766727+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T03:10:59.880330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":16,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-03T21:18:44.825940+0000","last_change":"2024-05-03T21:18:42.815721+0000","last_active":"2024-05-03T21:18:44.825940+0000","last_peered":"2024-05-03T21:18:44.825940+0000","last_clean":"2024-05-03T21:18:44.825940+0000","last_became_active":"2024-05-03T21:18:42.815408+0000","last_became_peered":"2024-05-03T21:18:42.815408+0000","last_unstale":"2024-05-03T21:18:44.825940+0000","last_undegraded":"2024-05-03T21:18:44.825940+0000","last_fullsized":"2024-05-03T21:18:44.825940+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_clean_scrub_stamp":"2024-05-03T21:18:41.788726+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T03:03:46.851667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"82'3","reported_seq":49,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-03T21:19:23.926323+0000","last_change":"2024-05-03T21:18:39.792060+0000","last_active":"2024-05-03T21:19:23.926323+0000","last_peered":"2024-05-03T21:19:23.926323+0000","last_clean":"2024-05-03T21:19:23.926323+0000","last_became_active":"2024-05-03T21:18:39.791704+0000","last_became_peered":"2024-05-03T21:18:39.791704+0000","last_unstale":"2024-05-03T21:19:23.926323+0000","last_undegraded":"2024-05-03T21:19:23.926323+0000","last_fullsized":"2024-05-03T21:19:23.926323+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T21:18:38.766727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T21:18:38.766727+0000","last_clean_scrub_stamp":"2024-05-03T21:18:38.766727+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T03:46:43.784554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":25,"num_read_kb":25,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":16,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-03T21:18:44.824351+0000","last_change":"2024-05-03T21:18:42.823016+0000","last_active":"2024-05-03T21:18:44.824351+0000","last_peered":"2024-05-03T21:18:44.824351+0000","last_clean":"2024-05-03T21:18:44.824351+0000","last_became_active":"2024-05-03T21:18:42.822363+0000","last_became_peered":"2024-05-03T21:18:42.822363+0000","last_unstale":"2024-05-03T21:18:44.824351+0000","last_undegraded":"2024-05-03T21:18:44.824351+0000","last_fullsized":"2024-05-03T21:18:44.824351+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_clean_scrub_stamp":"2024-05-03T21:18:41.788726+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T07:46:36.213746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"82'2","reported_seq":23,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-03T21:18:56.865853+0000","last_change":"2024-05-03T21:18:39.791944+0000","last_active":"2024-05-03T21:18:56.865853+0000","last_peered":"2024-05-03T21:18:56.865853+0000","last_clean":"2024-05-03T21:18:56.865853+0000","last_became_active":"2024-05-03T21:18:39.791576+0000","last_became_peered":"2024-05-03T21:18:39.791576+0000","last_unstale":"2024-05-03T21:18:56.865853+0000","last_undegraded":"2024-05-03T21:18:56.865853+0000","last_fullsized":"2024-05-03T21:18:56.865853+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T21:18:38.766727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T21:18:38.766727+0000","last_clean_scrub_stamp":"2024-05-03T21:18:38.766727+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T07:05:58.190107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":16,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-03T21:18:44.825830+0000","last_change":"2024-05-03T21:18:42.819587+0000","last_active":"2024-05-03T21:18:44.825830+0000","last_peered":"2024-05-03T21:18:44.825830+0000","last_clean":"2024-05-03T21:18:44.825830+0000","last_became_active":"2024-05-03T21:18:42.819286+0000","last_became_peered":"2024-05-03T21:18:42.819286+0000","last_unstale":"2024-05-03T21:18:44.825830+0000","last_undegraded":"2024-05-03T21:18:44.825830+0000","last_fullsized":"2024-05-03T21:18:44.825830+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_clean_scrub_stamp":"2024-05-03T21:18:41.788726+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T01:31:34.367410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"82'4","reported_seq":27,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-03T21:18:56.864959+0000","last_change":"2024-05-03T21:18:39.815394+0000","last_active":"2024-05-03T21:18:56.864959+0000","last_peered":"2024-05-03T21:18:56.864959+0000","last_clean":"2024-05-03T21:18:56.864959+0000","last_became_active":"2024-05-03T21:18:39.814874+0000","last_became_peered":"2024-05-03T21:18:39.814874+0000","last_unstale":"2024-05-03T21:18:56.864959+0000","last_undegraded":"2024-05-03T21:18:56.864959+0000","last_fullsized":"2024-05-03T21:18:56.864959+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T21:18:38.766727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T21:18:38.766727+0000","last_clean_scrub_stamp":"2024-05-03T21:18:38.766727+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T08:42:50.532572+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":16,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-03T21:18:44.823698+0000","last_change":"2024-05-03T21:18:42.804483+0000","last_active":"2024-05-03T21:18:44.823698+0000","last_peered":"2024-05-03T21:18:44.823698+0000","last_clean":"2024-05-03T21:18:44.823698+0000","last_became_active":"2024-05-03T21:18:42.804064+0000","last_became_peered":"2024-05-03T21:18:42.804064+0000","last_unstale":"2024-05-03T21:18:44.823698+0000","last_undegraded":"2024-05-03T21:18:44.823698+0000","last_fullsized":"2024-05-03T21:18:44.823698+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_clean_scrub_stamp":"2024-05-03T21:18:41.788726+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T01:02:22.249152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"82'25","reported_seq":102,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-03T21:19:24.930975+0000","last_change":"2024-05-03T21:18:39.792864+0000","last_active":"2024-05-03T21:19:24.930975+0000","last_peered":"2024-05-03T21:19:24.930975+0000","last_clean":"2024-05-03T21:19:24.930975+0000","last_became_active":"2024-05-03T21:18:39.792469+0000","last_became_peered":"2024-05-03T21:18:39.792469+0000","last_unstale":"2024-05-03T21:19:24.930975+0000","last_undegraded":"2024-05-03T21:19:24.930975+0000","last_fullsized":"2024-05-03T21:19:24.930975+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T21:18:38.766727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T21:18:38.766727+0000","last_clean_scrub_stamp":"2024-05-03T21:18:38.766727+0000","objects_scrubbed":0,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-04T22:31:09.180649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":56,"num_read_kb":76,"num_write":26,"num_write_kb":56,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":16,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-03T21:18:44.825715+0000","last_change":"2024-05-03T21:18:42.816299+0000","last_active":"2024-05-03T21:18:44.825715+0000","last_peered":"2024-05-03T21:18:44.825715+0000","last_clean":"2024-05-03T21:18:44.825715+0000","last_became_active":"2024-05-03T21:18:42.815947+0000","last_became_peered":"2024-05-03T21:18:42.815947+0000","last_unstale":"2024-05-03T21:18:44.825715+0000","last_undegraded":"2024-05-03T21:18:44.825715+0000","last_fullsized":"2024-05-03T21:18:44.825715+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_clean_scrub_stamp":"2024-05-03T21:18:41.788726+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T07:52:21.620885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"82'6","reported_seq":28,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-03T21:18:57.873519+0000","last_change":"2024-05-03T21:18:39.794552+0000","last_active":"2024-05-03T21:18:57.873519+0000","last_peered":"2024-05-03T21:18:57.873519+0000","last_clean":"2024-05-03T21:18:57.873519+0000","last_became_active":"2024-05-03T21:18:39.794183+0000","last_became_peered":"2024-05-03T21:18:39.794183+0000","last_unstale":"2024-05-03T21:18:57.873519+0000","last_undegraded":"2024-05-03T21:18:57.873519+0000","last_fullsized":"2024-05-03T21:18:57.873519+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T21:18:38.766727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T21:18:38.766727+0000","last_clean_scrub_stamp":"2024-05-03T21:18:38.766727+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T00:16:32.820261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":16,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-03T21:18:44.823220+0000","last_change":"2024-05-03T21:18:42.811542+0000","last_active":"2024-05-03T21:18:44.823220+0000","last_peered":"2024-05-03T21:18:44.823220+0000","last_clean":"2024-05-03T21:18:44.823220+0000","last_became_active":"2024-05-03T21:18:42.810301+0000","last_became_peered":"2024-05-03T21:18:42.810301+0000","last_unstale":"2024-05-03T21:18:44.823220+0000","last_undegraded":"2024-05-03T21:18:44.823220+0000","last_fullsized":"2024-05-03T21:18:44.823220+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_clean_scrub_stamp":"2024-05-03T21:18:41.788726+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T03:01:30.503201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"82'3","reported_seq":25,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-03T21:18:56.863772+0000","last_change":"2024-05-03T21:18:39.796028+0000","last_active":"2024-05-03T21:18:56.863772+0000","last_peered":"2024-05-03T21:18:56.863772+0000","last_clean":"2024-05-03T21:18:56.863772+0000","last_became_active":"2024-05-03T21:18:39.795646+0000","last_became_peered":"2024-05-03T21:18:39.795646+0000","last_unstale":"2024-05-03T21:18:56.863772+0000","last_undegraded":"2024-05-03T21:18:56.863772+0000","last_fullsized":"2024-05-03T21:18:56.863772+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T21:18:38.766727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T21:18:38.766727+0000","last_clean_scrub_stamp":"2024-05-03T21:18:38.766727+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T01:56:31.128184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":16,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-03T21:18:44.826284+0000","last_change":"2024-05-03T21:18:42.810690+0000","last_active":"2024-05-03T21:18:44.826284+0000","last_peered":"2024-05-03T21:18:44.826284+0000","last_clean":"2024-05-03T21:18:44.826284+0000","last_became_active":"2024-05-03T21:18:42.810397+0000","last_became_peered":"2024-05-03T21:18:42.810397+0000","last_unstale":"2024-05-03T21:18:44.826284+0000","last_undegraded":"2024-05-03T21:18:44.826284+0000","last_fullsized":"2024-05-03T21:18:44.826284+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_clean_scrub_stamp":"2024-05-03T21:18:41.788726+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T09:14:43.292353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"82'4","reported_seq":51,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-03T21:19:20.922654+0000","last_change":"2024-05-03T21:18:39.793102+0000","last_active":"2024-05-03T21:19:20.922654+0000","last_peered":"2024-05-03T21:19:20.922654+0000","last_clean":"2024-05-03T21:19:20.922654+0000","last_became_active":"2024-05-03T21:18:39.792700+0000","last_became_peered":"2024-05-03T21:18:39.792700+0000","last_unstale":"2024-05-03T21:19:20.922654+0000","last_undegraded":"2024-05-03T21:19:20.922654+0000","last_fullsized":"2024-05-03T21:19:20.922654+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T21:18:38.766727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T21:18:38.766727+0000","last_clean_scrub_stamp":"2024-05-03T21:18:38.766727+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-04T21:32:36.761116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":979,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":4,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":16,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-03T21:18:44.826908+0000","last_change":"2024-05-03T21:18:42.815975+0000","last_active":"2024-05-03T21:18:44.826908+0000","last_peered":"2024-05-03T21:18:44.826908+0000","last_clean":"2024-05-03T21:18:44.826908+0000","last_became_active":"2024-05-03T21:18:42.814667+0000","last_became_peered":"2024-05-03T21:18:42.814667+0000","last_unstale":"2024-05-03T21:18:44.826908+0000","last_undegraded":"2024-05-03T21:18:44.826908+0000","last_fullsized":"2024-05-03T21:18:44.826908+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_clean_scrub_stamp":"2024-05-03T21:18:41.788726+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T00:50:47.991863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"82'15","reported_seq":38,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-03T21:18:57.875431+0000","last_change":"2024-05-03T21:18:39.815018+0000","last_active":"2024-05-03T21:18:57.875431+0000","last_peered":"2024-05-03T21:18:57.875431+0000","last_clean":"2024-05-03T21:18:57.875431+0000","last_became_active":"2024-05-03T21:18:39.814319+0000","last_became_peered":"2024-05-03T21:18:39.814319+0000","last_unstale":"2024-05-03T21:18:57.875431+0000","last_undegraded":"2024-05-03T21:18:57.875431+0000","last_fullsized":"2024-05-03T21:18:57.875431+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T21:18:38.766727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T21:18:38.766727+0000","last_clean_scrub_stamp":"2024-05-03T21:18:38.766727+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T00:43:52.230521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":16,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-03T21:18:44.810137+0000","last_change":"2024-05-03T21:18:42.810313+0000","last_active":"2024-05-03T21:18:44.810137+0000","last_peered":"2024-05-03T21:18:44.810137+0000","last_clean":"2024-05-03T21:18:44.810137+0000","last_became_active":"2024-05-03T21:18:42.809732+0000","last_became_peered":"2024-05-03T21:18:42.809732+0000","last_unstale":"2024-05-03T21:18:44.810137+0000","last_undegraded":"2024-05-03T21:18:44.810137+0000","last_fullsized":"2024-05-03T21:18:44.810137+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_clean_scrub_stamp":"2024-05-03T21:18:41.788726+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T01:36:54.077645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"82'2","reported_seq":25,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-03T21:18:59.869422+0000","last_change":"2024-05-03T21:18:39.784067+0000","last_active":"2024-05-03T21:18:59.869422+0000","last_peered":"2024-05-03T21:18:59.869422+0000","last_clean":"2024-05-03T21:18:59.869422+0000","last_became_active":"2024-05-03T21:18:39.783596+0000","last_became_peered":"2024-05-03T21:18:39.783596+0000","last_unstale":"2024-05-03T21:18:59.869422+0000","last_undegraded":"2024-05-03T21:18:59.869422+0000","last_fullsized":"2024-05-03T21:18:59.869422+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T21:18:38.766727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T21:18:38.766727+0000","last_clean_scrub_stamp":"2024-05-03T21:18:38.766727+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T03:10:25.925757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":16,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-03T21:18:44.808547+0000","last_change":"2024-05-03T21:18:42.817617+0000","last_active":"2024-05-03T21:18:44.808547+0000","last_peered":"2024-05-03T21:18:44.808547+0000","last_clean":"2024-05-03T21:18:44.808547+0000","last_became_active":"2024-05-03T21:18:42.817267+0000","last_became_peered":"2024-05-03T21:18:42.817267+0000","last_unstale":"2024-05-03T21:18:44.808547+0000","last_undegraded":"2024-05-03T21:18:44.808547+0000","last_fullsized":"2024-05-03T21:18:44.808547+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_clean_scrub_stamp":"2024-05-03T21:18:41.788726+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T08:00:12.864215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":21,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-03T21:18:44.825193+0000","last_change":"2024-05-03T21:18:39.814213+0000","last_active":"2024-05-03T21:18:44.825193+0000","last_peered":"2024-05-03T21:18:44.825193+0000","last_clean":"2024-05-03T21:18:44.825193+0000","last_became_active":"2024-05-03T21:18:39.813787+0000","last_became_peered":"2024-05-03T21:18:39.813787+0000","last_unstale":"2024-05-03T21:18:44.825193+0000","last_undegraded":"2024-05-03T21:18:44.825193+0000","last_fullsized":"2024-05-03T21:18:44.825193+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T21:18:38.766727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T21:18:38.766727+0000","last_clean_scrub_stamp":"2024-05-03T21:18:38.766727+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T08:45:17.963396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"82'1","reported_seq":17,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-03T21:18:45.303732+0000","last_change":"2024-05-03T21:18:42.821547+0000","last_active":"2024-05-03T21:18:45.303732+0000","last_peered":"2024-05-03T21:18:45.303732+0000","last_clean":"2024-05-03T21:18:45.303732+0000","last_became_active":"2024-05-03T21:18:42.821168+0000","last_became_peered":"2024-05-03T21:18:42.821168+0000","last_unstale":"2024-05-03T21:18:45.303732+0000","last_undegraded":"2024-05-03T21:18:45.303732+0000","last_fullsized":"2024-05-03T21:18:45.303732+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_clean_scrub_stamp":"2024-05-03T21:18:41.788726+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T07:28:25.037648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"82'1","reported_seq":22,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-03T21:18:44.826618+0000","last_change":"2024-05-03T21:18:39.792994+0000","last_active":"2024-05-03T21:18:44.826618+0000","last_peered":"2024-05-03T21:18:44.826618+0000","last_clean":"2024-05-03T21:18:44.826618+0000","last_became_active":"2024-05-03T21:18:39.792125+0000","last_became_peered":"2024-05-03T21:18:39.792125+0000","last_unstale":"2024-05-03T21:18:44.826618+0000","last_undegraded":"2024-05-03T21:18:44.826618+0000","last_fullsized":"2024-05-03T21:18:44.826618+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T21:18:38.766727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T21:18:38.766727+0000","last_clean_scrub_stamp":"2024-05-03T21:18:38.766727+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T03:09:45.917263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":16,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-03T21:18:44.808517+0000","last_change":"2024-05-03T21:18:42.814513+0000","last_active":"2024-05-03T21:18:44.808517+0000","last_peered":"2024-05-03T21:18:44.808517+0000","last_clean":"2024-05-03T21:18:44.808517+0000","last_became_active":"2024-05-03T21:18:42.814097+0000","last_became_peered":"2024-05-03T21:18:42.814097+0000","last_unstale":"2024-05-03T21:18:44.808517+0000","last_undegraded":"2024-05-03T21:18:44.808517+0000","last_fullsized":"2024-05-03T21:18:44.808517+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_clean_scrub_stamp":"2024-05-03T21:18:41.788726+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T06:52:46.731543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":16,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-03T21:18:44.810452+0000","last_change":"2024-05-03T21:18:42.812413+0000","last_active":"2024-05-03T21:18:44.810452+0000","last_peered":"2024-05-03T21:18:44.810452+0000","last_clean":"2024-05-03T21:18:44.810452+0000","last_became_active":"2024-05-03T21:18:42.812065+0000","last_became_peered":"2024-05-03T21:18:42.812065+0000","last_unstale":"2024-05-03T21:18:44.810452+0000","last_undegraded":"2024-05-03T21:18:44.810452+0000","last_fullsized":"2024-05-03T21:18:44.810452+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_clean_scrub_stamp":"2024-05-03T21:18:41.788726+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T07:52:44.489097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"82'4","reported_seq":47,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-03T21:19:20.921970+0000","last_change":"2024-05-03T21:18:39.813982+0000","last_active":"2024-05-03T21:19:20.921970+0000","last_peered":"2024-05-03T21:19:20.921970+0000","last_clean":"2024-05-03T21:19:20.921970+0000","last_became_active":"2024-05-03T21:18:39.813590+0000","last_became_peered":"2024-05-03T21:18:39.813590+0000","last_unstale":"2024-05-03T21:19:20.921970+0000","last_undegraded":"2024-05-03T21:19:20.921970+0000","last_fullsized":"2024-05-03T21:19:20.921970+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T21:18:38.766727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T21:18:38.766727+0000","last_clean_scrub_stamp":"2024-05-03T21:18:38.766727+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T02:30:24.733268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":22,"num_read_kb":22,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":16,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-03T21:18:44.827080+0000","last_change":"2024-05-03T21:18:42.808357+0000","last_active":"2024-05-03T21:18:44.827080+0000","last_peered":"2024-05-03T21:18:44.827080+0000","last_clean":"2024-05-03T21:18:44.827080+0000","last_became_active":"2024-05-03T21:18:42.808023+0000","last_became_peered":"2024-05-03T21:18:42.808023+0000","last_unstale":"2024-05-03T21:18:44.827080+0000","last_undegraded":"2024-05-03T21:18:44.827080+0000","last_fullsized":"2024-05-03T21:18:44.827080+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_clean_scrub_stamp":"2024-05-03T21:18:41.788726+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T08:33:11.713491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"82'3","reported_seq":27,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-03T21:19:00.890481+0000","last_change":"2024-05-03T21:18:39.787052+0000","last_active":"2024-05-03T21:19:00.890481+0000","last_peered":"2024-05-03T21:19:00.890481+0000","last_clean":"2024-05-03T21:19:00.890481+0000","last_became_active":"2024-05-03T21:18:39.786390+0000","last_became_peered":"2024-05-03T21:18:39.786390+0000","last_unstale":"2024-05-03T21:19:00.890481+0000","last_undegraded":"2024-05-03T21:19:00.890481+0000","last_fullsized":"2024-05-03T21:19:00.890481+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T21:18:38.766727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T21:18:38.766727+0000","last_clean_scrub_stamp":"2024-05-03T21:18:38.766727+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T08:18:27.832378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"21'92","reported_seq":95,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-03T21:18:44.826380+0000","last_change":"2024-05-03T21:17:17.301485+0000","last_active":"2024-05-03T21:18:44.826380+0000","last_peered":"2024-05-03T21:18:44.826380+0000","last_clean":"2024-05-03T21:18:44.826380+0000","last_became_active":"2024-05-03T21:17:17.296458+0000","last_became_peered":"2024-05-03T21:17:17.296458+0000","last_unstale":"2024-05-03T21:18:44.826380+0000","last_undegraded":"2024-05-03T21:18:44.826380+0000","last_fullsized":"2024-05-03T21:18:44.826380+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T21:14:00.058823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T21:14:00.058823+0000","last_clean_scrub_stamp":"2024-05-03T21:14:00.058823+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T03:13:14.815888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":16,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-03T21:18:44.822472+0000","last_change":"2024-05-03T21:18:42.818715+0000","last_active":"2024-05-03T21:18:44.822472+0000","last_peered":"2024-05-03T21:18:44.822472+0000","last_clean":"2024-05-03T21:18:44.822472+0000","last_became_active":"2024-05-03T21:18:42.818327+0000","last_became_peered":"2024-05-03T21:18:42.818327+0000","last_unstale":"2024-05-03T21:18:44.822472+0000","last_undegraded":"2024-05-03T21:18:44.822472+0000","last_fullsized":"2024-05-03T21:18:44.822472+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_clean_scrub_stamp":"2024-05-03T21:18:41.788726+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T08:42:31.151515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"82'3","reported_seq":24,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-03T21:18:57.875103+0000","last_change":"2024-05-03T21:18:39.791884+0000","last_active":"2024-05-03T21:18:57.875103+0000","last_peered":"2024-05-03T21:18:57.875103+0000","last_clean":"2024-05-03T21:18:57.875103+0000","last_became_active":"2024-05-03T21:18:39.791465+0000","last_became_peered":"2024-05-03T21:18:39.791465+0000","last_unstale":"2024-05-03T21:18:57.875103+0000","last_undegraded":"2024-05-03T21:18:57.875103+0000","last_fullsized":"2024-05-03T21:18:57.875103+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T21:18:38.766727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T21:18:38.766727+0000","last_clean_scrub_stamp":"2024-05-03T21:18:38.766727+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T07:25:39.572012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"82'1","reported_seq":24,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-03T21:18:56.864376+0000","last_change":"2024-05-03T21:18:39.814671+0000","last_active":"2024-05-03T21:18:56.864376+0000","last_peered":"2024-05-03T21:18:56.864376+0000","last_clean":"2024-05-03T21:18:56.864376+0000","last_became_active":"2024-05-03T21:18:39.814326+0000","last_became_peered":"2024-05-03T21:18:39.814326+0000","last_unstale":"2024-05-03T21:18:56.864376+0000","last_undegraded":"2024-05-03T21:18:56.864376+0000","last_fullsized":"2024-05-03T21:18:56.864376+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T21:18:38.766727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T21:18:38.766727+0000","last_clean_scrub_stamp":"2024-05-03T21:18:38.766727+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T01:21:53.058203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":16,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-03T21:18:44.823177+0000","last_change":"2024-05-03T21:18:42.815525+0000","last_active":"2024-05-03T21:18:44.823177+0000","last_peered":"2024-05-03T21:18:44.823177+0000","last_clean":"2024-05-03T21:18:44.823177+0000","last_became_active":"2024-05-03T21:18:42.815209+0000","last_became_peered":"2024-05-03T21:18:42.815209+0000","last_unstale":"2024-05-03T21:18:44.823177+0000","last_undegraded":"2024-05-03T21:18:44.823177+0000","last_fullsized":"2024-05-03T21:18:44.823177+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_clean_scrub_stamp":"2024-05-03T21:18:41.788726+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T01:02:51.545090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"82'9","reported_seq":30,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-03T21:19:15.825284+0000","last_change":"2024-05-03T21:18:39.784592+0000","last_active":"2024-05-03T21:19:15.825284+0000","last_peered":"2024-05-03T21:19:15.825284+0000","last_clean":"2024-05-03T21:19:15.825284+0000","last_became_active":"2024-05-03T21:18:39.784242+0000","last_became_peered":"2024-05-03T21:18:39.784242+0000","last_unstale":"2024-05-03T21:19:15.825284+0000","last_undegraded":"2024-05-03T21:19:15.825284+0000","last_fullsized":"2024-05-03T21:19:15.825284+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T21:18:38.766727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T21:18:38.766727+0000","last_clean_scrub_stamp":"2024-05-03T21:18:38.766727+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-04T22:23:01.914553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":16,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-03T21:18:44.826924+0000","last_change":"2024-05-03T21:18:42.814361+0000","last_active":"2024-05-03T21:18:44.826924+0000","last_peered":"2024-05-03T21:18:44.826924+0000","last_clean":"2024-05-03T21:18:44.826924+0000","last_became_active":"2024-05-03T21:18:42.813448+0000","last_became_peered":"2024-05-03T21:18:42.813448+0000","last_unstale":"2024-05-03T21:18:44.826924+0000","last_undegraded":"2024-05-03T21:18:44.826924+0000","last_fullsized":"2024-05-03T21:18:44.826924+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_clean_scrub_stamp":"2024-05-03T21:18:41.788726+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T06:21:11.974428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"82'4","reported_seq":26,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-03T21:18:56.863301+0000","last_change":"2024-05-03T21:18:39.905533+0000","last_active":"2024-05-03T21:18:56.863301+0000","last_peered":"2024-05-03T21:18:56.863301+0000","last_clean":"2024-05-03T21:18:56.863301+0000","last_became_active":"2024-05-03T21:18:39.905256+0000","last_became_peered":"2024-05-03T21:18:39.905256+0000","last_unstale":"2024-05-03T21:18:56.863301+0000","last_undegraded":"2024-05-03T21:18:56.863301+0000","last_fullsized":"2024-05-03T21:18:56.863301+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T21:18:38.766727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T21:18:38.766727+0000","last_clean_scrub_stamp":"2024-05-03T21:18:38.766727+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T07:07:12.162628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":16,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-03T21:18:44.825270+0000","last_change":"2024-05-03T21:18:42.821052+0000","last_active":"2024-05-03T21:18:44.825270+0000","last_peered":"2024-05-03T21:18:44.825270+0000","last_clean":"2024-05-03T21:18:44.825270+0000","last_became_active":"2024-05-03T21:18:42.820751+0000","last_became_peered":"2024-05-03T21:18:42.820751+0000","last_unstale":"2024-05-03T21:18:44.825270+0000","last_undegraded":"2024-05-03T21:18:44.825270+0000","last_fullsized":"2024-05-03T21:18:44.825270+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_clean_scrub_stamp":"2024-05-03T21:18:41.788726+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T02:14:45.829778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"82'2","reported_seq":24,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-03T21:18:56.866318+0000","last_change":"2024-05-03T21:18:39.814793+0000","last_active":"2024-05-03T21:18:56.866318+0000","last_peered":"2024-05-03T21:18:56.866318+0000","last_clean":"2024-05-03T21:18:56.866318+0000","last_became_active":"2024-05-03T21:18:39.814427+0000","last_became_peered":"2024-05-03T21:18:39.814427+0000","last_unstale":"2024-05-03T21:18:56.866318+0000","last_undegraded":"2024-05-03T21:18:56.866318+0000","last_fullsized":"2024-05-03T21:18:56.866318+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T21:18:38.766727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T21:18:38.766727+0000","last_clean_scrub_stamp":"2024-05-03T21:18:38.766727+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T01:01:34.673401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":16,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-03T21:18:44.823103+0000","last_change":"2024-05-03T21:18:42.812123+0000","last_active":"2024-05-03T21:18:44.823103+0000","last_peered":"2024-05-03T21:18:44.823103+0000","last_clean":"2024-05-03T21:18:44.823103+0000","last_became_active":"2024-05-03T21:18:42.811799+0000","last_became_peered":"2024-05-03T21:18:42.811799+0000","last_unstale":"2024-05-03T21:18:44.823103+0000","last_undegraded":"2024-05-03T21:18:44.823103+0000","last_fullsized":"2024-05-03T21:18:44.823103+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_clean_scrub_stamp":"2024-05-03T21:18:41.788726+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T05:47:42.995000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"82'3","reported_seq":24,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-03T21:18:56.864980+0000","last_change":"2024-05-03T21:18:39.786729+0000","last_active":"2024-05-03T21:18:56.864980+0000","last_peered":"2024-05-03T21:18:56.864980+0000","last_clean":"2024-05-03T21:18:56.864980+0000","last_became_active":"2024-05-03T21:18:39.786276+0000","last_became_peered":"2024-05-03T21:18:39.786276+0000","last_unstale":"2024-05-03T21:18:56.864980+0000","last_undegraded":"2024-05-03T21:18:56.864980+0000","last_fullsized":"2024-05-03T21:18:56.864980+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T21:18:38.766727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T21:18:38.766727+0000","last_clean_scrub_stamp":"2024-05-03T21:18:38.766727+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T00:47:34.921468+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":16,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-03T21:18:44.808384+0000","last_change":"2024-05-03T21:18:42.818135+0000","last_active":"2024-05-03T21:18:44.808384+0000","last_peered":"2024-05-03T21:18:44.808384+0000","last_clean":"2024-05-03T21:18:44.808384+0000","last_became_active":"2024-05-03T21:18:42.816868+0000","last_became_peered":"2024-05-03T21:18:42.816868+0000","last_unstale":"2024-05-03T21:18:44.808384+0000","last_undegraded":"2024-05-03T21:18:44.808384+0000","last_fullsized":"2024-05-03T21:18:44.808384+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_clean_scrub_stamp":"2024-05-03T21:18:41.788726+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T00:36:26.783820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"82'1","reported_seq":22,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-03T21:18:44.824149+0000","last_change":"2024-05-03T21:18:39.787608+0000","last_active":"2024-05-03T21:18:44.824149+0000","last_peered":"2024-05-03T21:18:44.824149+0000","last_clean":"2024-05-03T21:18:44.824149+0000","last_became_active":"2024-05-03T21:18:39.787041+0000","last_became_peered":"2024-05-03T21:18:39.787041+0000","last_unstale":"2024-05-03T21:18:44.824149+0000","last_undegraded":"2024-05-03T21:18:44.824149+0000","last_fullsized":"2024-05-03T21:18:44.824149+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T21:18:38.766727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T21:18:38.766727+0000","last_clean_scrub_stamp":"2024-05-03T21:18:38.766727+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-04T22:42:48.948711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":16,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-03T21:18:44.826847+0000","last_change":"2024-05-03T21:18:42.819671+0000","last_active":"2024-05-03T21:18:44.826847+0000","last_peered":"2024-05-03T21:18:44.826847+0000","last_clean":"2024-05-03T21:18:44.826847+0000","last_became_active":"2024-05-03T21:18:42.819336+0000","last_became_peered":"2024-05-03T21:18:42.819336+0000","last_unstale":"2024-05-03T21:18:44.826847+0000","last_undegraded":"2024-05-03T21:18:44.826847+0000","last_fullsized":"2024-05-03T21:18:44.826847+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_clean_scrub_stamp":"2024-05-03T21:18:41.788726+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T04:54:03.726522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"82'4","reported_seq":27,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-03T21:18:56.865488+0000","last_change":"2024-05-03T21:18:39.792012+0000","last_active":"2024-05-03T21:18:56.865488+0000","last_peered":"2024-05-03T21:18:56.865488+0000","last_clean":"2024-05-03T21:18:56.865488+0000","last_became_active":"2024-05-03T21:18:39.791638+0000","last_became_peered":"2024-05-03T21:18:39.791638+0000","last_unstale":"2024-05-03T21:18:56.865488+0000","last_undegraded":"2024-05-03T21:18:56.865488+0000","last_fullsized":"2024-05-03T21:18:56.865488+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T21:18:38.766727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T21:18:38.766727+0000","last_clean_scrub_stamp":"2024-05-03T21:18:38.766727+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T07:12:17.441761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":16,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-03T21:18:44.826808+0000","last_change":"2024-05-03T21:18:42.810587+0000","last_active":"2024-05-03T21:18:44.826808+0000","last_peered":"2024-05-03T21:18:44.826808+0000","last_clean":"2024-05-03T21:18:44.826808+0000","last_became_active":"2024-05-03T21:18:42.810274+0000","last_became_peered":"2024-05-03T21:18:42.810274+0000","last_unstale":"2024-05-03T21:18:44.826808+0000","last_undegraded":"2024-05-03T21:18:44.826808+0000","last_fullsized":"2024-05-03T21:18:44.826808+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_clean_scrub_stamp":"2024-05-03T21:18:41.788726+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T02:45:20.891119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"82'3","reported_seq":25,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-03T21:18:57.874152+0000","last_change":"2024-05-03T21:18:39.905882+0000","last_active":"2024-05-03T21:18:57.874152+0000","last_peered":"2024-05-03T21:18:57.874152+0000","last_clean":"2024-05-03T21:18:57.874152+0000","last_became_active":"2024-05-03T21:18:39.905488+0000","last_became_peered":"2024-05-03T21:18:39.905488+0000","last_unstale":"2024-05-03T21:18:57.874152+0000","last_undegraded":"2024-05-03T21:18:57.874152+0000","last_fullsized":"2024-05-03T21:18:57.874152+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T21:18:38.766727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T21:18:38.766727+0000","last_clean_scrub_stamp":"2024-05-03T21:18:38.766727+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T02:02:17.913931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":16,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-03T21:18:44.828561+0000","last_change":"2024-05-03T21:18:42.818693+0000","last_active":"2024-05-03T21:18:44.828561+0000","last_peered":"2024-05-03T21:18:44.828561+0000","last_clean":"2024-05-03T21:18:44.828561+0000","last_became_active":"2024-05-03T21:18:42.818177+0000","last_became_peered":"2024-05-03T21:18:42.818177+0000","last_unstale":"2024-05-03T21:18:44.828561+0000","last_undegraded":"2024-05-03T21:18:44.828561+0000","last_fullsized":"2024-05-03T21:18:44.828561+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_clean_scrub_stamp":"2024-05-03T21:18:41.788726+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T03:05:10.708866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"82'1","reported_seq":24,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-03T21:18:44.825739+0000","last_change":"2024-05-03T21:18:39.792399+0000","last_active":"2024-05-03T21:18:44.825739+0000","last_peered":"2024-05-03T21:18:44.825739+0000","last_clean":"2024-05-03T21:18:44.825739+0000","last_became_active":"2024-05-03T21:18:39.791462+0000","last_became_peered":"2024-05-03T21:18:39.791462+0000","last_unstale":"2024-05-03T21:18:44.825739+0000","last_undegraded":"2024-05-03T21:18:44.825739+0000","last_fullsized":"2024-05-03T21:18:44.825739+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T21:18:38.766727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T21:18:38.766727+0000","last_clean_scrub_stamp":"2024-05-03T21:18:38.766727+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T08:24:23.045438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"82'1","reported_seq":17,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-03T21:18:45.108763+0000","last_change":"2024-05-03T21:18:42.813275+0000","last_active":"2024-05-03T21:18:45.108763+0000","last_peered":"2024-05-03T21:18:45.108763+0000","last_clean":"2024-05-03T21:18:45.108763+0000","last_became_active":"2024-05-03T21:18:42.812640+0000","last_became_peered":"2024-05-03T21:18:42.812640+0000","last_unstale":"2024-05-03T21:18:45.108763+0000","last_undegraded":"2024-05-03T21:18:45.108763+0000","last_fullsized":"2024-05-03T21:18:45.108763+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_clean_scrub_stamp":"2024-05-03T21:18:41.788726+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T01:34:27.071382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"82'2","reported_seq":23,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-03T21:18:44.826343+0000","last_change":"2024-05-03T21:18:39.787487+0000","last_active":"2024-05-03T21:18:44.826343+0000","last_peered":"2024-05-03T21:18:44.826343+0000","last_clean":"2024-05-03T21:18:44.826343+0000","last_became_active":"2024-05-03T21:18:39.787135+0000","last_became_peered":"2024-05-03T21:18:39.787135+0000","last_unstale":"2024-05-03T21:18:44.826343+0000","last_undegraded":"2024-05-03T21:18:44.826343+0000","last_fullsized":"2024-05-03T21:18:44.826343+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T21:18:38.766727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T21:18:38.766727+0000","last_clean_scrub_stamp":"2024-05-03T21:18:38.766727+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T05:21:13.304600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":16,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-03T21:18:44.827864+0000","last_change":"2024-05-03T21:18:42.815682+0000","last_active":"2024-05-03T21:18:44.827864+0000","last_peered":"2024-05-03T21:18:44.827864+0000","last_clean":"2024-05-03T21:18:44.827864+0000","last_became_active":"2024-05-03T21:18:42.813011+0000","last_became_peered":"2024-05-03T21:18:42.813011+0000","last_unstale":"2024-05-03T21:18:44.827864+0000","last_undegraded":"2024-05-03T21:18:44.827864+0000","last_fullsized":"2024-05-03T21:18:44.827864+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_clean_scrub_stamp":"2024-05-03T21:18:41.788726+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T03:47:21.680152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":16,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-03T21:18:44.808499+0000","last_change":"2024-05-03T21:18:42.817574+0000","last_active":"2024-05-03T21:18:44.808499+0000","last_peered":"2024-05-03T21:18:44.808499+0000","last_clean":"2024-05-03T21:18:44.808499+0000","last_became_active":"2024-05-03T21:18:42.817227+0000","last_became_peered":"2024-05-03T21:18:42.817227+0000","last_unstale":"2024-05-03T21:18:44.808499+0000","last_undegraded":"2024-05-03T21:18:44.808499+0000","last_fullsized":"2024-05-03T21:18:44.808499+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_clean_scrub_stamp":"2024-05-03T21:18:41.788726+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-04T23:15:34.784285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":16,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-03T21:18:44.823494+0000","last_change":"2024-05-03T21:18:42.822400+0000","last_active":"2024-05-03T21:18:44.823494+0000","last_peered":"2024-05-03T21:18:44.823494+0000","last_clean":"2024-05-03T21:18:44.823494+0000","last_became_active":"2024-05-03T21:18:42.822000+0000","last_became_peered":"2024-05-03T21:18:42.822000+0000","last_unstale":"2024-05-03T21:18:44.823494+0000","last_undegraded":"2024-05-03T21:18:44.823494+0000","last_fullsized":"2024-05-03T21:18:44.823494+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_clean_scrub_stamp":"2024-05-03T21:18:41.788726+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T06:06:23.492826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":16,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-03T21:18:44.823927+0000","last_change":"2024-05-03T21:18:42.823980+0000","last_active":"2024-05-03T21:18:44.823927+0000","last_peered":"2024-05-03T21:18:44.823927+0000","last_clean":"2024-05-03T21:18:44.823927+0000","last_became_active":"2024-05-03T21:18:42.823619+0000","last_became_peered":"2024-05-03T21:18:42.823619+0000","last_unstale":"2024-05-03T21:18:44.823927+0000","last_undegraded":"2024-05-03T21:18:44.823927+0000","last_fullsized":"2024-05-03T21:18:44.823927+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_clean_scrub_stamp":"2024-05-03T21:18:41.788726+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T08:13:53.104446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":16,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-03T21:18:44.824020+0000","last_change":"2024-05-03T21:18:42.813596+0000","last_active":"2024-05-03T21:18:44.824020+0000","last_peered":"2024-05-03T21:18:44.824020+0000","last_clean":"2024-05-03T21:18:44.824020+0000","last_became_active":"2024-05-03T21:18:42.813238+0000","last_became_peered":"2024-05-03T21:18:42.813238+0000","last_unstale":"2024-05-03T21:18:44.824020+0000","last_undegraded":"2024-05-03T21:18:44.824020+0000","last_fullsized":"2024-05-03T21:18:44.824020+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_clean_scrub_stamp":"2024-05-03T21:18:41.788726+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T05:14:52.712566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":16,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-03T21:18:44.822965+0000","last_change":"2024-05-03T21:18:42.812252+0000","last_active":"2024-05-03T21:18:44.822965+0000","last_peered":"2024-05-03T21:18:44.822965+0000","last_clean":"2024-05-03T21:18:44.822965+0000","last_became_active":"2024-05-03T21:18:42.811917+0000","last_became_peered":"2024-05-03T21:18:42.811917+0000","last_unstale":"2024-05-03T21:18:44.822965+0000","last_undegraded":"2024-05-03T21:18:44.822965+0000","last_fullsized":"2024-05-03T21:18:44.822965+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_clean_scrub_stamp":"2024-05-03T21:18:41.788726+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T00:55:09.512412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":16,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-03T21:18:44.808264+0000","last_change":"2024-05-03T21:18:42.819345+0000","last_active":"2024-05-03T21:18:44.808264+0000","last_peered":"2024-05-03T21:18:44.808264+0000","last_clean":"2024-05-03T21:18:44.808264+0000","last_became_active":"2024-05-03T21:18:42.817901+0000","last_became_peered":"2024-05-03T21:18:42.817901+0000","last_unstale":"2024-05-03T21:18:44.808264+0000","last_undegraded":"2024-05-03T21:18:44.808264+0000","last_fullsized":"2024-05-03T21:18:44.808264+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_clean_scrub_stamp":"2024-05-03T21:18:41.788726+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T07:25:30.498261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":16,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-03T21:18:44.825269+0000","last_change":"2024-05-03T21:18:42.825009+0000","last_active":"2024-05-03T21:18:44.825269+0000","last_peered":"2024-05-03T21:18:44.825269+0000","last_clean":"2024-05-03T21:18:44.825269+0000","last_became_active":"2024-05-03T21:18:42.824626+0000","last_became_peered":"2024-05-03T21:18:42.824626+0000","last_unstale":"2024-05-03T21:18:44.825269+0000","last_undegraded":"2024-05-03T21:18:44.825269+0000","last_fullsized":"2024-05-03T21:18:44.825269+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_clean_scrub_stamp":"2024-05-03T21:18:41.788726+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T06:48:17.760340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":16,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-03T21:18:44.826847+0000","last_change":"2024-05-03T21:18:42.812305+0000","last_active":"2024-05-03T21:18:44.826847+0000","last_peered":"2024-05-03T21:18:44.826847+0000","last_clean":"2024-05-03T21:18:44.826847+0000","last_became_active":"2024-05-03T21:18:42.811738+0000","last_became_peered":"2024-05-03T21:18:42.811738+0000","last_unstale":"2024-05-03T21:18:44.826847+0000","last_undegraded":"2024-05-03T21:18:44.826847+0000","last_fullsized":"2024-05-03T21:18:44.826847+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_clean_scrub_stamp":"2024-05-03T21:18:41.788726+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-04T21:41:24.312517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":16,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-03T21:18:44.810058+0000","last_change":"2024-05-03T21:18:42.817306+0000","last_active":"2024-05-03T21:18:44.810058+0000","last_peered":"2024-05-03T21:18:44.810058+0000","last_clean":"2024-05-03T21:18:44.810058+0000","last_became_active":"2024-05-03T21:18:42.816768+0000","last_became_peered":"2024-05-03T21:18:42.816768+0000","last_unstale":"2024-05-03T21:18:44.810058+0000","last_undegraded":"2024-05-03T21:18:44.810058+0000","last_fullsized":"2024-05-03T21:18:44.810058+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_clean_scrub_stamp":"2024-05-03T21:18:41.788726+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-04T23:24:42.055568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":16,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-03T21:18:44.826798+0000","last_change":"2024-05-03T21:18:42.816008+0000","last_active":"2024-05-03T21:18:44.826798+0000","last_peered":"2024-05-03T21:18:44.826798+0000","last_clean":"2024-05-03T21:18:44.826798+0000","last_became_active":"2024-05-03T21:18:42.815653+0000","last_became_peered":"2024-05-03T21:18:42.815653+0000","last_unstale":"2024-05-03T21:18:44.826798+0000","last_undegraded":"2024-05-03T21:18:44.826798+0000","last_fullsized":"2024-05-03T21:18:44.826798+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_clean_scrub_stamp":"2024-05-03T21:18:41.788726+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-04T23:15:22.986562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":16,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-03T21:18:44.823562+0000","last_change":"2024-05-03T21:18:42.814820+0000","last_active":"2024-05-03T21:18:44.823562+0000","last_peered":"2024-05-03T21:18:44.823562+0000","last_clean":"2024-05-03T21:18:44.823562+0000","last_became_active":"2024-05-03T21:18:42.814498+0000","last_became_peered":"2024-05-03T21:18:42.814498+0000","last_unstale":"2024-05-03T21:18:44.823562+0000","last_undegraded":"2024-05-03T21:18:44.823562+0000","last_fullsized":"2024-05-03T21:18:44.823562+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_clean_scrub_stamp":"2024-05-03T21:18:41.788726+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T01:35:15.727585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":16,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-03T21:18:44.823542+0000","last_change":"2024-05-03T21:18:42.814701+0000","last_active":"2024-05-03T21:18:44.823542+0000","last_peered":"2024-05-03T21:18:44.823542+0000","last_clean":"2024-05-03T21:18:44.823542+0000","last_became_active":"2024-05-03T21:18:42.814394+0000","last_became_peered":"2024-05-03T21:18:42.814394+0000","last_unstale":"2024-05-03T21:18:44.823542+0000","last_undegraded":"2024-05-03T21:18:44.823542+0000","last_fullsized":"2024-05-03T21:18:44.823542+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_clean_scrub_stamp":"2024-05-03T21:18:41.788726+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T04:50:05.941873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":16,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-03T21:18:44.808193+0000","last_change":"2024-05-03T21:18:42.815987+0000","last_active":"2024-05-03T21:18:44.808193+0000","last_peered":"2024-05-03T21:18:44.808193+0000","last_clean":"2024-05-03T21:18:44.808193+0000","last_became_active":"2024-05-03T21:18:42.815613+0000","last_became_peered":"2024-05-03T21:18:42.815613+0000","last_unstale":"2024-05-03T21:18:44.808193+0000","last_undegraded":"2024-05-03T21:18:44.808193+0000","last_fullsized":"2024-05-03T21:18:44.808193+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_clean_scrub_stamp":"2024-05-03T21:18:41.788726+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T03:57:01.997504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4751360,"data_stored":4722944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":92,"ondisk_log_size":92,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":54502,"num_objects":60,"num_object_clones":0,"num_object_copies":180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":0,"num_read":167,"num_read_kb":143,"num_write":156,"num_write_kb":97,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":393216,"data_stored":163506,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":134,"ondisk_log_size":134,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":74,"seq":317827579927,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27876,"kb_used_data":860,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717276,"statfs":{"total":95995035648,"available":95966490624,"internally_reserved":0,"allocated":880640,"data_stored":424993,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":68,"seq":292057776156,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28548,"kb_used_data":1532,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716604,"statfs":{"total":95995035648,"available":95965802496,"internally_reserved":0,"allocated":1568768,"data_stored":1066975,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":63,"seq":270582939681,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32052,"kb_used_data":876,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713100,"statfs":{"total":95995035648,"available":95962214400,"internally_reserved":0,"allocated":897024,"data_stored":425229,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":57,"seq":244813135911,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32052,"kb_used_data":876,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713100,"statfs":{"total":95995035648,"available":95962214400,"internally_reserved":0,"allocated":897024,"data_stored":425149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":50,"seq":214748364843,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28472,"kb_used_data":1456,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716680,"statfs":{"total":95995035648,"available":95965880320,"internally_reserved":0,"allocated":1490944,"data_stored":1017470,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":45,"seq":193273528368,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27880,"kb_used_data":864,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717272,"statfs":{"total":95995035648,"available":95966486528,"internally_reserved":0,"allocated":884736,"data_stored":424618,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574922,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36252,"kb_used_data":916,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708900,"statfs":{"total":95995035648,"available":95957913600,"internally_reserved":0,"allocated":937984,"data_stored":474521,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738446,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36220,"kb_used_data":884,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708932,"statfs":{"total":95995035648,"available":95957946368,"internally_reserved":0,"allocated":905216,"data_stored":425889,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411396,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36276,"kb_used_data":940,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708876,"statfs":{"total":95995035648,"available":95957889024,"internally_reserved":0,"allocated":962560,"data_stored":476262,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":27,"seq":115964117055,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36216,"kb_used_data":880,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708936,"statfs":{"total":95995035648,"available":95957950464,"internally_reserved":0,"allocated":901120,"data_stored":424929,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920825,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27884,"kb_used_data":868,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717268,"statfs":{"total":95995035648,"available":95966482432,"internally_reserved":0,"allocated":888832,"data_stored":425446,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724598,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32644,"kb_used_data":1468,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712508,"statfs":{"total":95995035648,"available":95961608192,"internally_reserved":0,"allocated":1503232,"data_stored":1016763,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":50110,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":51851,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1035,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1867,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2574,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":94208,"data_stored":52079,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-05-03T21:19:27.540 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9a047fe7599a549042d06a3fb5b6813a203eabea shell --fsid 713bc57a-0991-11ef-bc97-c7b262605968 -- ceph pg dump --format=json 2024-05-03T21:19:27.793 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/713bc57a-0991-11ef-bc97-c7b262605968/mon.a/config 2024-05-03T21:19:28.178 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:19:27 smithi067 ceph-mon[29730]: from='client.14736 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-03T21:19:28.245 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:19:27 smithi082 ceph-mon[32933]: from='client.14736 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-03T21:19:28.324 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:19:27 smithi138 ceph-mon[33110]: from='client.14736 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-03T21:19:29.180 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:19:28 smithi067 ceph-mon[29730]: pgmap v309: 97 pgs: 97 active+clean; 630 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 2.2 KiB/s wr, 7 op/s 2024-05-03T21:19:29.245 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:19:28 smithi082 ceph-mon[32933]: pgmap v309: 97 pgs: 97 active+clean; 630 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 2.2 KiB/s wr, 7 op/s 2024-05-03T21:19:29.324 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:19:28 smithi138 ceph-mon[33110]: pgmap v309: 97 pgs: 97 active+clean; 630 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 2.2 KiB/s wr, 7 op/s 2024-05-03T21:19:29.447 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:29.445+0000 7fbb76f84700 1 -- 172.21.15.67:0/18943230 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbb70102630 msgr2=0x7fbb70102a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:19:29.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:29.445+0000 7fbb76f84700 1 --2- 172.21.15.67:0/18943230 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbb70102630 0x7fbb70102a10 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fbb640099c0 tx=0x7fbb64009cd0 comp rx=0 tx=0).stop 2024-05-03T21:19:29.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:29.445+0000 7fbb76f84700 1 -- 172.21.15.67:0/18943230 shutdown_connections 2024-05-03T21:19:29.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:29.445+0000 7fbb76f84700 1 --2- 172.21.15.67:0/18943230 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbb70102630 0x7fbb70102a10 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:19:29.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:29.445+0000 7fbb76f84700 1 --2- 172.21.15.67:0/18943230 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fbb701076c0 0x7fbb7010f290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:19:29.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:29.445+0000 7fbb76f84700 1 --2- 172.21.15.67:0/18943230 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fbb70102fe0 0x7fbb70106fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:19:29.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:29.445+0000 7fbb76f84700 1 -- 172.21.15.67:0/18943230 >> 172.21.15.67:0/18943230 conn(0x7fbb70079350 msgr2=0x7fbb70079760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T21:19:29.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:29.445+0000 7fbb76f84700 1 -- 172.21.15.67:0/18943230 shutdown_connections 2024-05-03T21:19:29.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:29.445+0000 7fbb76f84700 1 -- 172.21.15.67:0/18943230 wait complete. 2024-05-03T21:19:29.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:29.445+0000 7fbb76f84700 1 Processor -- start 2024-05-03T21:19:29.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:29.446+0000 7fbb76f84700 1 -- start start 2024-05-03T21:19:29.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:29.446+0000 7fbb76f84700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbb70102630 0x7fbb70199d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:19:29.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:29.446+0000 7fbb76f84700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fbb70102fe0 0x7fbb7019a2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:19:29.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:29.446+0000 7fbb76f84700 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fbb701076c0 0x7fbb7019e630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:19:29.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:29.446+0000 7fbb76f84700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fbb701127e0 con 0x7fbb70102630 2024-05-03T21:19:29.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:29.446+0000 7fbb76f84700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7fbb70112660 con 0x7fbb70102fe0 2024-05-03T21:19:29.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:29.446+0000 7fbb76f84700 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7fbb70112960 con 0x7fbb701076c0 2024-05-03T21:19:29.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:29.446+0000 7fbb74d20700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbb70102630 0x7fbb70199d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:19:29.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:29.446+0000 7fbb74d20700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbb70102630 0x7fbb70199d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:50242/0 (socket says 172.21.15.67:50242) 2024-05-03T21:19:29.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:29.446+0000 7fbb74d20700 1 -- 172.21.15.67:0/2682698542 learned_addr learned my addr 172.21.15.67:0/2682698542 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-03T21:19:29.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:29.446+0000 7fbb6ffff700 1 --2- 172.21.15.67:0/2682698542 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fbb70102fe0 0x7fbb7019a2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:19:29.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:29.446+0000 7fbb75521700 1 --2- 172.21.15.67:0/2682698542 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fbb701076c0 0x7fbb7019e630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:19:29.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:29.446+0000 7fbb74d20700 1 -- 172.21.15.67:0/2682698542 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fbb701076c0 msgr2=0x7fbb7019e630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:19:29.451 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:29.446+0000 7fbb74d20700 1 --2- 172.21.15.67:0/2682698542 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fbb701076c0 0x7fbb7019e630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:19:29.451 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:29.446+0000 7fbb74d20700 1 -- 172.21.15.67:0/2682698542 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fbb70102fe0 msgr2=0x7fbb7019a2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:19:29.451 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:29.446+0000 7fbb74d20700 1 --2- 172.21.15.67:0/2682698542 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fbb70102fe0 0x7fbb7019a2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:19:29.451 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:29.446+0000 7fbb74d20700 1 -- 172.21.15.67:0/2682698542 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbb7019ed70 con 0x7fbb70102630 2024-05-03T21:19:29.451 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:29.447+0000 7fbb74d20700 1 --2- 172.21.15.67:0/2682698542 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbb70102630 0x7fbb70199d60 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fbb640059d0 tx=0x7fbb6400bb00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T21:19:29.451 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:29.447+0000 7fbb6dffb700 1 -- 172.21.15.67:0/2682698542 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb64027070 con 0x7fbb70102630 2024-05-03T21:19:29.451 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:29.447+0000 7fbb76f84700 1 -- 172.21.15.67:0/2682698542 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbb7019f000 con 0x7fbb70102630 2024-05-03T21:19:29.451 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:29.447+0000 7fbb6dffb700 1 -- 172.21.15.67:0/2682698542 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(28 keys) ==== 1170+0+0 (secure 0 0 0) 0x7fbb64003f70 con 0x7fbb70102630 2024-05-03T21:19:29.451 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:29.447+0000 7fbb6dffb700 1 -- 172.21.15.67:0/2682698542 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb64022dd0 con 0x7fbb70102630 2024-05-03T21:19:29.451 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:29.447+0000 7fbb76f84700 1 -- 172.21.15.67:0/2682698542 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbb701a68e0 con 0x7fbb70102630 2024-05-03T21:19:29.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:29.448+0000 7fbb76f84700 1 -- 172.21.15.67:0/2682698542 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbb70103ac0 con 0x7fbb70102630 2024-05-03T21:19:29.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:29.449+0000 7fbb6dffb700 1 -- 172.21.15.67:0/2682698542 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 17) ==== 149703+0+0 (secure 0 0 0) 0x7fbb6402a070 con 0x7fbb70102630 2024-05-03T21:19:29.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:29.450+0000 7fbb6dffb700 1 --2- 172.21.15.67:0/2682698542 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7fbb600b2910 0x7fbb600b4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:19:29.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:29.450+0000 7fbb6ffff700 1 --2- 172.21.15.67:0/2682698542 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7fbb600b2910 0x7fbb600b4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:19:29.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:29.450+0000 7fbb6dffb700 1 -- 172.21.15.67:0/2682698542 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7979+0+0 (secure 0 0 0) 0x7fbb640e2ff0 con 0x7fbb70102630 2024-05-03T21:19:29.453 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:29.451+0000 7fbb6ffff700 1 --2- 172.21.15.67:0/2682698542 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7fbb600b2910 0x7fbb600b4dd0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fbb70108cb0 tx=0x7fbb5c00a300 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T21:19:29.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:29.454+0000 7fbb6dffb700 1 -- 172.21.15.67:0/2682698542 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191839 (secure 0 0 0) 0x7fbb64031080 con 0x7fbb70102630 2024-05-03T21:19:29.671 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:29.668+0000 7fbb76f84700 1 -- 172.21.15.67:0/2682698542 --> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fbb7019b0c0 con 0x7fbb600b2910 2024-05-03T21:19:29.678 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:29.676+0000 7fbb6dffb700 1 -- 172.21.15.67:0/2682698542 <== mgr.14150 v2:172.21.15.67:6800/604451592 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+258695 (secure 0 0 0) 0x7fbb7019b0c0 con 0x7fbb600b2910 2024-05-03T21:19:29.683 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-05-03T21:19:29.685 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:29.679+0000 7fbb76f84700 1 -- 172.21.15.67:0/2682698542 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7fbb600b2910 msgr2=0x7fbb600b4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:19:29.685 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:29.679+0000 7fbb76f84700 1 --2- 172.21.15.67:0/2682698542 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7fbb600b2910 0x7fbb600b4dd0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fbb70108cb0 tx=0x7fbb5c00a300 comp rx=0 tx=0).stop 2024-05-03T21:19:29.685 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:29.679+0000 7fbb76f84700 1 -- 172.21.15.67:0/2682698542 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbb70102630 msgr2=0x7fbb70199d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:19:29.685 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:29.679+0000 7fbb76f84700 1 --2- 172.21.15.67:0/2682698542 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbb70102630 0x7fbb70199d60 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fbb640059d0 tx=0x7fbb6400bb00 comp rx=0 tx=0).stop 2024-05-03T21:19:29.685 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:29.679+0000 7fbb76f84700 1 -- 172.21.15.67:0/2682698542 shutdown_connections 2024-05-03T21:19:29.685 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:29.679+0000 7fbb76f84700 1 --2- 172.21.15.67:0/2682698542 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7fbb600b2910 0x7fbb600b4dd0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:19:29.685 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:29.679+0000 7fbb76f84700 1 --2- 172.21.15.67:0/2682698542 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbb70102630 0x7fbb70199d60 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:19:29.686 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:29.679+0000 7fbb76f84700 1 --2- 172.21.15.67:0/2682698542 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fbb701076c0 0x7fbb7019e630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:19:29.686 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:29.679+0000 7fbb76f84700 1 --2- 172.21.15.67:0/2682698542 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fbb70102fe0 0x7fbb7019a2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:19:29.686 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:29.679+0000 7fbb76f84700 1 -- 172.21.15.67:0/2682698542 >> 172.21.15.67:0/2682698542 conn(0x7fbb70079350 msgr2=0x7fbb7006bee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T21:19:29.686 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:29.679+0000 7fbb76f84700 1 -- 172.21.15.67:0/2682698542 shutdown_connections 2024-05-03T21:19:29.686 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:29.679+0000 7fbb76f84700 1 -- 172.21.15.67:0/2682698542 wait complete. 2024-05-03T21:19:29.686 INFO:teuthology.orchestra.run.smithi067.stderr:dumped all 2024-05-03T21:19:30.350 INFO:teuthology.orchestra.run.smithi067.stdout:{"pg_ready":true,"pg_map":{"version":309,"stamp":"2024-05-03T21:19:27.816515+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":645104,"num_objects":64,"num_object_clones":0,"num_object_copies":192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":64,"num_whiteouts":0,"num_read":258,"num_read_kb":212,"num_write":281,"num_write_kb":2249,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":228,"ondisk_log_size":228,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":291,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":382372,"kb_used_data":12420,"kb_used_omap":18,"kb_used_meta":369837,"kb_avail":1124559452,"statfs":{"total":1151940427776,"available":1151548878848,"internally_reserved":0,"allocated":12718080,"data_stored":7028244,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19143,"internal_metadata":378713401},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":23282,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":76,"num_read_kb":58,"num_write":15,"num_write_kb":27,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003230"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":16,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-03T21:18:44.808490+0000","last_change":"2024-05-03T21:18:42.819790+0000","last_active":"2024-05-03T21:18:44.808490+0000","last_peered":"2024-05-03T21:18:44.808490+0000","last_clean":"2024-05-03T21:18:44.808490+0000","last_became_active":"2024-05-03T21:18:42.818989+0000","last_became_peered":"2024-05-03T21:18:42.818989+0000","last_unstale":"2024-05-03T21:18:44.808490+0000","last_undegraded":"2024-05-03T21:18:44.808490+0000","last_fullsized":"2024-05-03T21:18:44.808490+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_clean_scrub_stamp":"2024-05-03T21:18:41.788726+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T08:38:43.699198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":16,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-03T21:18:44.825312+0000","last_change":"2024-05-03T21:18:42.812454+0000","last_active":"2024-05-03T21:18:44.825312+0000","last_peered":"2024-05-03T21:18:44.825312+0000","last_clean":"2024-05-03T21:18:44.825312+0000","last_became_active":"2024-05-03T21:18:42.812140+0000","last_became_peered":"2024-05-03T21:18:42.812140+0000","last_unstale":"2024-05-03T21:18:44.825312+0000","last_undegraded":"2024-05-03T21:18:44.825312+0000","last_fullsized":"2024-05-03T21:18:44.825312+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_clean_scrub_stamp":"2024-05-03T21:18:41.788726+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-04T22:30:36.057079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":16,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-03T21:18:44.824785+0000","last_change":"2024-05-03T21:18:42.815547+0000","last_active":"2024-05-03T21:18:44.824785+0000","last_peered":"2024-05-03T21:18:44.824785+0000","last_clean":"2024-05-03T21:18:44.824785+0000","last_became_active":"2024-05-03T21:18:42.815192+0000","last_became_peered":"2024-05-03T21:18:42.815192+0000","last_unstale":"2024-05-03T21:18:44.824785+0000","last_undegraded":"2024-05-03T21:18:44.824785+0000","last_fullsized":"2024-05-03T21:18:44.824785+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_clean_scrub_stamp":"2024-05-03T21:18:41.788726+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T04:15:05.786275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":16,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-03T21:18:44.826855+0000","last_change":"2024-05-03T21:18:42.819376+0000","last_active":"2024-05-03T21:18:44.826855+0000","last_peered":"2024-05-03T21:18:44.826855+0000","last_clean":"2024-05-03T21:18:44.826855+0000","last_became_active":"2024-05-03T21:18:42.819027+0000","last_became_peered":"2024-05-03T21:18:42.819027+0000","last_unstale":"2024-05-03T21:18:44.826855+0000","last_undegraded":"2024-05-03T21:18:44.826855+0000","last_fullsized":"2024-05-03T21:18:44.826855+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_clean_scrub_stamp":"2024-05-03T21:18:41.788726+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T01:41:06.639885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":16,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-03T21:18:44.823042+0000","last_change":"2024-05-03T21:18:42.807225+0000","last_active":"2024-05-03T21:18:44.823042+0000","last_peered":"2024-05-03T21:18:44.823042+0000","last_clean":"2024-05-03T21:18:44.823042+0000","last_became_active":"2024-05-03T21:18:42.806856+0000","last_became_peered":"2024-05-03T21:18:42.806856+0000","last_unstale":"2024-05-03T21:18:44.823042+0000","last_undegraded":"2024-05-03T21:18:44.823042+0000","last_fullsized":"2024-05-03T21:18:44.823042+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_clean_scrub_stamp":"2024-05-03T21:18:41.788726+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-04T23:59:15.221592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":16,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-03T21:18:44.808536+0000","last_change":"2024-05-03T21:18:42.819099+0000","last_active":"2024-05-03T21:18:44.808536+0000","last_peered":"2024-05-03T21:18:44.808536+0000","last_clean":"2024-05-03T21:18:44.808536+0000","last_became_active":"2024-05-03T21:18:42.818775+0000","last_became_peered":"2024-05-03T21:18:42.818775+0000","last_unstale":"2024-05-03T21:18:44.808536+0000","last_undegraded":"2024-05-03T21:18:44.808536+0000","last_fullsized":"2024-05-03T21:18:44.808536+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_clean_scrub_stamp":"2024-05-03T21:18:41.788726+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T03:46:43.762816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":16,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-03T21:18:44.826168+0000","last_change":"2024-05-03T21:18:42.811779+0000","last_active":"2024-05-03T21:18:44.826168+0000","last_peered":"2024-05-03T21:18:44.826168+0000","last_clean":"2024-05-03T21:18:44.826168+0000","last_became_active":"2024-05-03T21:18:42.811376+0000","last_became_peered":"2024-05-03T21:18:42.811376+0000","last_unstale":"2024-05-03T21:18:44.826168+0000","last_undegraded":"2024-05-03T21:18:44.826168+0000","last_fullsized":"2024-05-03T21:18:44.826168+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_clean_scrub_stamp":"2024-05-03T21:18:41.788726+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-04T22:08:00.629106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":16,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-03T21:18:44.827151+0000","last_change":"2024-05-03T21:18:42.820643+0000","last_active":"2024-05-03T21:18:44.827151+0000","last_peered":"2024-05-03T21:18:44.827151+0000","last_clean":"2024-05-03T21:18:44.827151+0000","last_became_active":"2024-05-03T21:18:42.820317+0000","last_became_peered":"2024-05-03T21:18:42.820317+0000","last_unstale":"2024-05-03T21:18:44.827151+0000","last_undegraded":"2024-05-03T21:18:44.827151+0000","last_fullsized":"2024-05-03T21:18:44.827151+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_clean_scrub_stamp":"2024-05-03T21:18:41.788726+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-04T21:39:44.319396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":16,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-03T21:18:44.824601+0000","last_change":"2024-05-03T21:18:42.819895+0000","last_active":"2024-05-03T21:18:44.824601+0000","last_peered":"2024-05-03T21:18:44.824601+0000","last_clean":"2024-05-03T21:18:44.824601+0000","last_became_active":"2024-05-03T21:18:42.818360+0000","last_became_peered":"2024-05-03T21:18:42.818360+0000","last_unstale":"2024-05-03T21:18:44.824601+0000","last_undegraded":"2024-05-03T21:18:44.824601+0000","last_fullsized":"2024-05-03T21:18:44.824601+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_clean_scrub_stamp":"2024-05-03T21:18:41.788726+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T06:49:09.368610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":16,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-03T21:18:44.808891+0000","last_change":"2024-05-03T21:18:42.817064+0000","last_active":"2024-05-03T21:18:44.808891+0000","last_peered":"2024-05-03T21:18:44.808891+0000","last_clean":"2024-05-03T21:18:44.808891+0000","last_became_active":"2024-05-03T21:18:42.816727+0000","last_became_peered":"2024-05-03T21:18:42.816727+0000","last_unstale":"2024-05-03T21:18:44.808891+0000","last_undegraded":"2024-05-03T21:18:44.808891+0000","last_fullsized":"2024-05-03T21:18:44.808891+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_clean_scrub_stamp":"2024-05-03T21:18:41.788726+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T06:30:20.093765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":16,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-03T21:18:44.827690+0000","last_change":"2024-05-03T21:18:42.818728+0000","last_active":"2024-05-03T21:18:44.827690+0000","last_peered":"2024-05-03T21:18:44.827690+0000","last_clean":"2024-05-03T21:18:44.827690+0000","last_became_active":"2024-05-03T21:18:42.818250+0000","last_became_peered":"2024-05-03T21:18:42.818250+0000","last_unstale":"2024-05-03T21:18:44.827690+0000","last_undegraded":"2024-05-03T21:18:44.827690+0000","last_fullsized":"2024-05-03T21:18:44.827690+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_clean_scrub_stamp":"2024-05-03T21:18:41.788726+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T03:52:04.613842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":16,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-03T21:18:44.826558+0000","last_change":"2024-05-03T21:18:42.816897+0000","last_active":"2024-05-03T21:18:44.826558+0000","last_peered":"2024-05-03T21:18:44.826558+0000","last_clean":"2024-05-03T21:18:44.826558+0000","last_became_active":"2024-05-03T21:18:42.816274+0000","last_became_peered":"2024-05-03T21:18:42.816274+0000","last_unstale":"2024-05-03T21:18:44.826558+0000","last_undegraded":"2024-05-03T21:18:44.826558+0000","last_fullsized":"2024-05-03T21:18:44.826558+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_clean_scrub_stamp":"2024-05-03T21:18:41.788726+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-04T21:48:17.744413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":16,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-03T21:18:44.822961+0000","last_change":"2024-05-03T21:18:42.820523+0000","last_active":"2024-05-03T21:18:44.822961+0000","last_peered":"2024-05-03T21:18:44.822961+0000","last_clean":"2024-05-03T21:18:44.822961+0000","last_became_active":"2024-05-03T21:18:42.820133+0000","last_became_peered":"2024-05-03T21:18:42.820133+0000","last_unstale":"2024-05-03T21:18:44.822961+0000","last_undegraded":"2024-05-03T21:18:44.822961+0000","last_fullsized":"2024-05-03T21:18:44.822961+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_clean_scrub_stamp":"2024-05-03T21:18:41.788726+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-04T23:25:34.304369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":16,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-03T21:18:44.823838+0000","last_change":"2024-05-03T21:18:42.814467+0000","last_active":"2024-05-03T21:18:44.823838+0000","last_peered":"2024-05-03T21:18:44.823838+0000","last_clean":"2024-05-03T21:18:44.823838+0000","last_became_active":"2024-05-03T21:18:42.814124+0000","last_became_peered":"2024-05-03T21:18:42.814124+0000","last_unstale":"2024-05-03T21:18:44.823838+0000","last_undegraded":"2024-05-03T21:18:44.823838+0000","last_fullsized":"2024-05-03T21:18:44.823838+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_clean_scrub_stamp":"2024-05-03T21:18:41.788726+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T08:41:03.567620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":16,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-03T21:18:44.823508+0000","last_change":"2024-05-03T21:18:42.822158+0000","last_active":"2024-05-03T21:18:44.823508+0000","last_peered":"2024-05-03T21:18:44.823508+0000","last_clean":"2024-05-03T21:18:44.823508+0000","last_became_active":"2024-05-03T21:18:42.821793+0000","last_became_peered":"2024-05-03T21:18:42.821793+0000","last_unstale":"2024-05-03T21:18:44.823508+0000","last_undegraded":"2024-05-03T21:18:44.823508+0000","last_fullsized":"2024-05-03T21:18:44.823508+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_clean_scrub_stamp":"2024-05-03T21:18:41.788726+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T06:44:09.269513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":16,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-03T21:18:44.822931+0000","last_change":"2024-05-03T21:18:42.813904+0000","last_active":"2024-05-03T21:18:44.822931+0000","last_peered":"2024-05-03T21:18:44.822931+0000","last_clean":"2024-05-03T21:18:44.822931+0000","last_became_active":"2024-05-03T21:18:42.813488+0000","last_became_peered":"2024-05-03T21:18:42.813488+0000","last_unstale":"2024-05-03T21:18:44.822931+0000","last_undegraded":"2024-05-03T21:18:44.822931+0000","last_fullsized":"2024-05-03T21:18:44.822931+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_clean_scrub_stamp":"2024-05-03T21:18:41.788726+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T00:52:20.288003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":16,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-03T21:18:44.808670+0000","last_change":"2024-05-03T21:18:42.820157+0000","last_active":"2024-05-03T21:18:44.808670+0000","last_peered":"2024-05-03T21:18:44.808670+0000","last_clean":"2024-05-03T21:18:44.808670+0000","last_became_active":"2024-05-03T21:18:42.819450+0000","last_became_peered":"2024-05-03T21:18:42.819450+0000","last_unstale":"2024-05-03T21:18:44.808670+0000","last_undegraded":"2024-05-03T21:18:44.808670+0000","last_fullsized":"2024-05-03T21:18:44.808670+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_clean_scrub_stamp":"2024-05-03T21:18:41.788726+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T04:34:19.252499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":16,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-03T21:18:44.825529+0000","last_change":"2024-05-03T21:18:42.816582+0000","last_active":"2024-05-03T21:18:44.825529+0000","last_peered":"2024-05-03T21:18:44.825529+0000","last_clean":"2024-05-03T21:18:44.825529+0000","last_became_active":"2024-05-03T21:18:42.816247+0000","last_became_peered":"2024-05-03T21:18:42.816247+0000","last_unstale":"2024-05-03T21:18:44.825529+0000","last_undegraded":"2024-05-03T21:18:44.825529+0000","last_fullsized":"2024-05-03T21:18:44.825529+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_clean_scrub_stamp":"2024-05-03T21:18:41.788726+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T00:09:02.620176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":16,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-03T21:18:44.824827+0000","last_change":"2024-05-03T21:18:42.821587+0000","last_active":"2024-05-03T21:18:44.824827+0000","last_peered":"2024-05-03T21:18:44.824827+0000","last_clean":"2024-05-03T21:18:44.824827+0000","last_became_active":"2024-05-03T21:18:42.821236+0000","last_became_peered":"2024-05-03T21:18:42.821236+0000","last_unstale":"2024-05-03T21:18:44.824827+0000","last_undegraded":"2024-05-03T21:18:44.824827+0000","last_fullsized":"2024-05-03T21:18:44.824827+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_clean_scrub_stamp":"2024-05-03T21:18:41.788726+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-04T22:23:46.746787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"82'9","reported_seq":31,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-03T21:18:57.874729+0000","last_change":"2024-05-03T21:18:39.792001+0000","last_active":"2024-05-03T21:18:57.874729+0000","last_peered":"2024-05-03T21:18:57.874729+0000","last_clean":"2024-05-03T21:18:57.874729+0000","last_became_active":"2024-05-03T21:18:39.791644+0000","last_became_peered":"2024-05-03T21:18:39.791644+0000","last_unstale":"2024-05-03T21:18:57.874729+0000","last_undegraded":"2024-05-03T21:18:57.874729+0000","last_fullsized":"2024-05-03T21:18:57.874729+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T21:18:38.766727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T21:18:38.766727+0000","last_clean_scrub_stamp":"2024-05-03T21:18:38.766727+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-04T23:15:36.169243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":16,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-03T21:18:44.827474+0000","last_change":"2024-05-03T21:18:42.815115+0000","last_active":"2024-05-03T21:18:44.827474+0000","last_peered":"2024-05-03T21:18:44.827474+0000","last_clean":"2024-05-03T21:18:44.827474+0000","last_became_active":"2024-05-03T21:18:42.814734+0000","last_became_peered":"2024-05-03T21:18:42.814734+0000","last_unstale":"2024-05-03T21:18:44.827474+0000","last_undegraded":"2024-05-03T21:18:44.827474+0000","last_fullsized":"2024-05-03T21:18:44.827474+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_clean_scrub_stamp":"2024-05-03T21:18:41.788726+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-04T23:25:18.232982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"82'2","reported_seq":23,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-03T21:18:44.829754+0000","last_change":"2024-05-03T21:18:39.814107+0000","last_active":"2024-05-03T21:18:44.829754+0000","last_peered":"2024-05-03T21:18:44.829754+0000","last_clean":"2024-05-03T21:18:44.829754+0000","last_became_active":"2024-05-03T21:18:39.813615+0000","last_became_peered":"2024-05-03T21:18:39.813615+0000","last_unstale":"2024-05-03T21:18:44.829754+0000","last_undegraded":"2024-05-03T21:18:44.829754+0000","last_fullsized":"2024-05-03T21:18:44.829754+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T21:18:38.766727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T21:18:38.766727+0000","last_clean_scrub_stamp":"2024-05-03T21:18:38.766727+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-04T23:24:00.914805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":16,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-03T21:18:44.825794+0000","last_change":"2024-05-03T21:18:42.816925+0000","last_active":"2024-05-03T21:18:44.825794+0000","last_peered":"2024-05-03T21:18:44.825794+0000","last_clean":"2024-05-03T21:18:44.825794+0000","last_became_active":"2024-05-03T21:18:42.814731+0000","last_became_peered":"2024-05-03T21:18:42.814731+0000","last_unstale":"2024-05-03T21:18:44.825794+0000","last_undegraded":"2024-05-03T21:18:44.825794+0000","last_fullsized":"2024-05-03T21:18:44.825794+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_clean_scrub_stamp":"2024-05-03T21:18:41.788726+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T00:18:26.859784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"82'4","reported_seq":26,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-03T21:18:57.873396+0000","last_change":"2024-05-03T21:18:39.792101+0000","last_active":"2024-05-03T21:18:57.873396+0000","last_peered":"2024-05-03T21:18:57.873396+0000","last_clean":"2024-05-03T21:18:57.873396+0000","last_became_active":"2024-05-03T21:18:39.791725+0000","last_became_peered":"2024-05-03T21:18:39.791725+0000","last_unstale":"2024-05-03T21:18:57.873396+0000","last_undegraded":"2024-05-03T21:18:57.873396+0000","last_fullsized":"2024-05-03T21:18:57.873396+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T21:18:38.766727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T21:18:38.766727+0000","last_clean_scrub_stamp":"2024-05-03T21:18:38.766727+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T05:09:42.900318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":16,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-03T21:18:44.823496+0000","last_change":"2024-05-03T21:18:42.823318+0000","last_active":"2024-05-03T21:18:44.823496+0000","last_peered":"2024-05-03T21:18:44.823496+0000","last_clean":"2024-05-03T21:18:44.823496+0000","last_became_active":"2024-05-03T21:18:42.823011+0000","last_became_peered":"2024-05-03T21:18:42.823011+0000","last_unstale":"2024-05-03T21:18:44.823496+0000","last_undegraded":"2024-05-03T21:18:44.823496+0000","last_fullsized":"2024-05-03T21:18:44.823496+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_clean_scrub_stamp":"2024-05-03T21:18:41.788726+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T01:04:51.967269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"82'2","reported_seq":24,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-03T21:18:44.828425+0000","last_change":"2024-05-03T21:18:39.787730+0000","last_active":"2024-05-03T21:18:44.828425+0000","last_peered":"2024-05-03T21:18:44.828425+0000","last_clean":"2024-05-03T21:18:44.828425+0000","last_became_active":"2024-05-03T21:18:39.787403+0000","last_became_peered":"2024-05-03T21:18:39.787403+0000","last_unstale":"2024-05-03T21:18:44.828425+0000","last_undegraded":"2024-05-03T21:18:44.828425+0000","last_fullsized":"2024-05-03T21:18:44.828425+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T21:18:38.766727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T21:18:38.766727+0000","last_clean_scrub_stamp":"2024-05-03T21:18:38.766727+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T06:28:41.454461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":16,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-03T21:18:44.810654+0000","last_change":"2024-05-03T21:18:42.818987+0000","last_active":"2024-05-03T21:18:44.810654+0000","last_peered":"2024-05-03T21:18:44.810654+0000","last_clean":"2024-05-03T21:18:44.810654+0000","last_became_active":"2024-05-03T21:18:42.818645+0000","last_became_peered":"2024-05-03T21:18:42.818645+0000","last_unstale":"2024-05-03T21:18:44.810654+0000","last_undegraded":"2024-05-03T21:18:44.810654+0000","last_fullsized":"2024-05-03T21:18:44.810654+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_clean_scrub_stamp":"2024-05-03T21:18:41.788726+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T02:26:27.470390+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"82'6","reported_seq":58,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-03T21:19:25.929714+0000","last_change":"2024-05-03T21:18:39.793807+0000","last_active":"2024-05-03T21:19:25.929714+0000","last_peered":"2024-05-03T21:19:25.929714+0000","last_clean":"2024-05-03T21:19:25.929714+0000","last_became_active":"2024-05-03T21:18:39.793447+0000","last_became_peered":"2024-05-03T21:18:39.793447+0000","last_unstale":"2024-05-03T21:19:25.929714+0000","last_undegraded":"2024-05-03T21:19:25.929714+0000","last_fullsized":"2024-05-03T21:19:25.929714+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T21:18:38.766727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T21:18:38.766727+0000","last_clean_scrub_stamp":"2024-05-03T21:18:38.766727+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T06:44:55.036889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1013,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":30,"num_read_kb":3,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":16,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-03T21:18:44.825008+0000","last_change":"2024-05-03T21:18:42.807486+0000","last_active":"2024-05-03T21:18:44.825008+0000","last_peered":"2024-05-03T21:18:44.825008+0000","last_clean":"2024-05-03T21:18:44.825008+0000","last_became_active":"2024-05-03T21:18:42.807120+0000","last_became_peered":"2024-05-03T21:18:42.807120+0000","last_unstale":"2024-05-03T21:18:44.825008+0000","last_undegraded":"2024-05-03T21:18:44.825008+0000","last_fullsized":"2024-05-03T21:18:44.825008+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_clean_scrub_stamp":"2024-05-03T21:18:41.788726+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T05:32:38.621330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1a","version":"82'3","reported_seq":24,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-03T21:18:57.873629+0000","last_change":"2024-05-03T21:18:39.791943+0000","last_active":"2024-05-03T21:18:57.873629+0000","last_peered":"2024-05-03T21:18:57.873629+0000","last_clean":"2024-05-03T21:18:57.873629+0000","last_became_active":"2024-05-03T21:18:39.791587+0000","last_became_peered":"2024-05-03T21:18:39.791587+0000","last_unstale":"2024-05-03T21:18:57.873629+0000","last_undegraded":"2024-05-03T21:18:57.873629+0000","last_fullsized":"2024-05-03T21:18:57.873629+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T21:18:38.766727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T21:18:38.766727+0000","last_clean_scrub_stamp":"2024-05-03T21:18:38.766727+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T05:01:39.458019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":16,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-03T21:18:44.810502+0000","last_change":"2024-05-03T21:18:42.815435+0000","last_active":"2024-05-03T21:18:44.810502+0000","last_peered":"2024-05-03T21:18:44.810502+0000","last_clean":"2024-05-03T21:18:44.810502+0000","last_became_active":"2024-05-03T21:18:42.815064+0000","last_became_peered":"2024-05-03T21:18:42.815064+0000","last_unstale":"2024-05-03T21:18:44.810502+0000","last_undegraded":"2024-05-03T21:18:44.810502+0000","last_fullsized":"2024-05-03T21:18:44.810502+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_clean_scrub_stamp":"2024-05-03T21:18:41.788726+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T06:08:50.435421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"82'2","reported_seq":23,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-03T21:18:57.873460+0000","last_change":"2024-05-03T21:18:39.784859+0000","last_active":"2024-05-03T21:18:57.873460+0000","last_peered":"2024-05-03T21:18:57.873460+0000","last_clean":"2024-05-03T21:18:57.873460+0000","last_became_active":"2024-05-03T21:18:39.784428+0000","last_became_peered":"2024-05-03T21:18:39.784428+0000","last_unstale":"2024-05-03T21:18:57.873460+0000","last_undegraded":"2024-05-03T21:18:57.873460+0000","last_fullsized":"2024-05-03T21:18:57.873460+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T21:18:38.766727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T21:18:38.766727+0000","last_clean_scrub_stamp":"2024-05-03T21:18:38.766727+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T05:56:18.368399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":16,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-03T21:18:44.827308+0000","last_change":"2024-05-03T21:18:42.813285+0000","last_active":"2024-05-03T21:18:44.827308+0000","last_peered":"2024-05-03T21:18:44.827308+0000","last_clean":"2024-05-03T21:18:44.827308+0000","last_became_active":"2024-05-03T21:18:42.812954+0000","last_became_peered":"2024-05-03T21:18:42.812954+0000","last_unstale":"2024-05-03T21:18:44.827308+0000","last_undegraded":"2024-05-03T21:18:44.827308+0000","last_fullsized":"2024-05-03T21:18:44.827308+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_clean_scrub_stamp":"2024-05-03T21:18:41.788726+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T00:19:26.841514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"82'1","reported_seq":23,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-03T21:18:44.810087+0000","last_change":"2024-05-03T21:18:39.793485+0000","last_active":"2024-05-03T21:18:44.810087+0000","last_peered":"2024-05-03T21:18:44.810087+0000","last_clean":"2024-05-03T21:18:44.810087+0000","last_became_active":"2024-05-03T21:18:39.793055+0000","last_became_peered":"2024-05-03T21:18:39.793055+0000","last_unstale":"2024-05-03T21:18:44.810087+0000","last_undegraded":"2024-05-03T21:18:44.810087+0000","last_fullsized":"2024-05-03T21:18:44.810087+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T21:18:38.766727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T21:18:38.766727+0000","last_clean_scrub_stamp":"2024-05-03T21:18:38.766727+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T03:10:59.880330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":16,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-03T21:18:44.825940+0000","last_change":"2024-05-03T21:18:42.815721+0000","last_active":"2024-05-03T21:18:44.825940+0000","last_peered":"2024-05-03T21:18:44.825940+0000","last_clean":"2024-05-03T21:18:44.825940+0000","last_became_active":"2024-05-03T21:18:42.815408+0000","last_became_peered":"2024-05-03T21:18:42.815408+0000","last_unstale":"2024-05-03T21:18:44.825940+0000","last_undegraded":"2024-05-03T21:18:44.825940+0000","last_fullsized":"2024-05-03T21:18:44.825940+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_clean_scrub_stamp":"2024-05-03T21:18:41.788726+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T03:03:46.851667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"82'3","reported_seq":49,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-03T21:19:23.926323+0000","last_change":"2024-05-03T21:18:39.792060+0000","last_active":"2024-05-03T21:19:23.926323+0000","last_peered":"2024-05-03T21:19:23.926323+0000","last_clean":"2024-05-03T21:19:23.926323+0000","last_became_active":"2024-05-03T21:18:39.791704+0000","last_became_peered":"2024-05-03T21:18:39.791704+0000","last_unstale":"2024-05-03T21:19:23.926323+0000","last_undegraded":"2024-05-03T21:19:23.926323+0000","last_fullsized":"2024-05-03T21:19:23.926323+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T21:18:38.766727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T21:18:38.766727+0000","last_clean_scrub_stamp":"2024-05-03T21:18:38.766727+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T03:46:43.784554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":25,"num_read_kb":25,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":16,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-03T21:18:44.824351+0000","last_change":"2024-05-03T21:18:42.823016+0000","last_active":"2024-05-03T21:18:44.824351+0000","last_peered":"2024-05-03T21:18:44.824351+0000","last_clean":"2024-05-03T21:18:44.824351+0000","last_became_active":"2024-05-03T21:18:42.822363+0000","last_became_peered":"2024-05-03T21:18:42.822363+0000","last_unstale":"2024-05-03T21:18:44.824351+0000","last_undegraded":"2024-05-03T21:18:44.824351+0000","last_fullsized":"2024-05-03T21:18:44.824351+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_clean_scrub_stamp":"2024-05-03T21:18:41.788726+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T07:46:36.213746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"82'2","reported_seq":23,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-03T21:18:56.865853+0000","last_change":"2024-05-03T21:18:39.791944+0000","last_active":"2024-05-03T21:18:56.865853+0000","last_peered":"2024-05-03T21:18:56.865853+0000","last_clean":"2024-05-03T21:18:56.865853+0000","last_became_active":"2024-05-03T21:18:39.791576+0000","last_became_peered":"2024-05-03T21:18:39.791576+0000","last_unstale":"2024-05-03T21:18:56.865853+0000","last_undegraded":"2024-05-03T21:18:56.865853+0000","last_fullsized":"2024-05-03T21:18:56.865853+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T21:18:38.766727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T21:18:38.766727+0000","last_clean_scrub_stamp":"2024-05-03T21:18:38.766727+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T07:05:58.190107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":16,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-03T21:18:44.825830+0000","last_change":"2024-05-03T21:18:42.819587+0000","last_active":"2024-05-03T21:18:44.825830+0000","last_peered":"2024-05-03T21:18:44.825830+0000","last_clean":"2024-05-03T21:18:44.825830+0000","last_became_active":"2024-05-03T21:18:42.819286+0000","last_became_peered":"2024-05-03T21:18:42.819286+0000","last_unstale":"2024-05-03T21:18:44.825830+0000","last_undegraded":"2024-05-03T21:18:44.825830+0000","last_fullsized":"2024-05-03T21:18:44.825830+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_clean_scrub_stamp":"2024-05-03T21:18:41.788726+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T01:31:34.367410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"82'4","reported_seq":27,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-03T21:18:56.864959+0000","last_change":"2024-05-03T21:18:39.815394+0000","last_active":"2024-05-03T21:18:56.864959+0000","last_peered":"2024-05-03T21:18:56.864959+0000","last_clean":"2024-05-03T21:18:56.864959+0000","last_became_active":"2024-05-03T21:18:39.814874+0000","last_became_peered":"2024-05-03T21:18:39.814874+0000","last_unstale":"2024-05-03T21:18:56.864959+0000","last_undegraded":"2024-05-03T21:18:56.864959+0000","last_fullsized":"2024-05-03T21:18:56.864959+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T21:18:38.766727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T21:18:38.766727+0000","last_clean_scrub_stamp":"2024-05-03T21:18:38.766727+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T08:42:50.532572+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":16,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-03T21:18:44.823698+0000","last_change":"2024-05-03T21:18:42.804483+0000","last_active":"2024-05-03T21:18:44.823698+0000","last_peered":"2024-05-03T21:18:44.823698+0000","last_clean":"2024-05-03T21:18:44.823698+0000","last_became_active":"2024-05-03T21:18:42.804064+0000","last_became_peered":"2024-05-03T21:18:42.804064+0000","last_unstale":"2024-05-03T21:18:44.823698+0000","last_undegraded":"2024-05-03T21:18:44.823698+0000","last_fullsized":"2024-05-03T21:18:44.823698+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_clean_scrub_stamp":"2024-05-03T21:18:41.788726+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T01:02:22.249152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"82'25","reported_seq":102,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-03T21:19:24.930975+0000","last_change":"2024-05-03T21:18:39.792864+0000","last_active":"2024-05-03T21:19:24.930975+0000","last_peered":"2024-05-03T21:19:24.930975+0000","last_clean":"2024-05-03T21:19:24.930975+0000","last_became_active":"2024-05-03T21:18:39.792469+0000","last_became_peered":"2024-05-03T21:18:39.792469+0000","last_unstale":"2024-05-03T21:19:24.930975+0000","last_undegraded":"2024-05-03T21:19:24.930975+0000","last_fullsized":"2024-05-03T21:19:24.930975+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T21:18:38.766727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T21:18:38.766727+0000","last_clean_scrub_stamp":"2024-05-03T21:18:38.766727+0000","objects_scrubbed":0,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-04T22:31:09.180649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":56,"num_read_kb":76,"num_write":26,"num_write_kb":56,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":16,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-03T21:18:44.825715+0000","last_change":"2024-05-03T21:18:42.816299+0000","last_active":"2024-05-03T21:18:44.825715+0000","last_peered":"2024-05-03T21:18:44.825715+0000","last_clean":"2024-05-03T21:18:44.825715+0000","last_became_active":"2024-05-03T21:18:42.815947+0000","last_became_peered":"2024-05-03T21:18:42.815947+0000","last_unstale":"2024-05-03T21:18:44.825715+0000","last_undegraded":"2024-05-03T21:18:44.825715+0000","last_fullsized":"2024-05-03T21:18:44.825715+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_clean_scrub_stamp":"2024-05-03T21:18:41.788726+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T07:52:21.620885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"82'6","reported_seq":28,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-03T21:18:57.873519+0000","last_change":"2024-05-03T21:18:39.794552+0000","last_active":"2024-05-03T21:18:57.873519+0000","last_peered":"2024-05-03T21:18:57.873519+0000","last_clean":"2024-05-03T21:18:57.873519+0000","last_became_active":"2024-05-03T21:18:39.794183+0000","last_became_peered":"2024-05-03T21:18:39.794183+0000","last_unstale":"2024-05-03T21:18:57.873519+0000","last_undegraded":"2024-05-03T21:18:57.873519+0000","last_fullsized":"2024-05-03T21:18:57.873519+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T21:18:38.766727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T21:18:38.766727+0000","last_clean_scrub_stamp":"2024-05-03T21:18:38.766727+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T00:16:32.820261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":16,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-03T21:18:44.823220+0000","last_change":"2024-05-03T21:18:42.811542+0000","last_active":"2024-05-03T21:18:44.823220+0000","last_peered":"2024-05-03T21:18:44.823220+0000","last_clean":"2024-05-03T21:18:44.823220+0000","last_became_active":"2024-05-03T21:18:42.810301+0000","last_became_peered":"2024-05-03T21:18:42.810301+0000","last_unstale":"2024-05-03T21:18:44.823220+0000","last_undegraded":"2024-05-03T21:18:44.823220+0000","last_fullsized":"2024-05-03T21:18:44.823220+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_clean_scrub_stamp":"2024-05-03T21:18:41.788726+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T03:01:30.503201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"82'3","reported_seq":25,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-03T21:18:56.863772+0000","last_change":"2024-05-03T21:18:39.796028+0000","last_active":"2024-05-03T21:18:56.863772+0000","last_peered":"2024-05-03T21:18:56.863772+0000","last_clean":"2024-05-03T21:18:56.863772+0000","last_became_active":"2024-05-03T21:18:39.795646+0000","last_became_peered":"2024-05-03T21:18:39.795646+0000","last_unstale":"2024-05-03T21:18:56.863772+0000","last_undegraded":"2024-05-03T21:18:56.863772+0000","last_fullsized":"2024-05-03T21:18:56.863772+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T21:18:38.766727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T21:18:38.766727+0000","last_clean_scrub_stamp":"2024-05-03T21:18:38.766727+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T01:56:31.128184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":16,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-03T21:18:44.826284+0000","last_change":"2024-05-03T21:18:42.810690+0000","last_active":"2024-05-03T21:18:44.826284+0000","last_peered":"2024-05-03T21:18:44.826284+0000","last_clean":"2024-05-03T21:18:44.826284+0000","last_became_active":"2024-05-03T21:18:42.810397+0000","last_became_peered":"2024-05-03T21:18:42.810397+0000","last_unstale":"2024-05-03T21:18:44.826284+0000","last_undegraded":"2024-05-03T21:18:44.826284+0000","last_fullsized":"2024-05-03T21:18:44.826284+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_clean_scrub_stamp":"2024-05-03T21:18:41.788726+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T09:14:43.292353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"82'4","reported_seq":56,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-03T21:19:25.930637+0000","last_change":"2024-05-03T21:18:39.793102+0000","last_active":"2024-05-03T21:19:25.930637+0000","last_peered":"2024-05-03T21:19:25.930637+0000","last_clean":"2024-05-03T21:19:25.930637+0000","last_became_active":"2024-05-03T21:18:39.792700+0000","last_became_peered":"2024-05-03T21:18:39.792700+0000","last_unstale":"2024-05-03T21:19:25.930637+0000","last_undegraded":"2024-05-03T21:19:25.930637+0000","last_fullsized":"2024-05-03T21:19:25.930637+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T21:18:38.766727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T21:18:38.766727+0000","last_clean_scrub_stamp":"2024-05-03T21:18:38.766727+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-04T21:32:36.761116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":979,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":31,"num_read_kb":4,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":16,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-03T21:18:44.826908+0000","last_change":"2024-05-03T21:18:42.815975+0000","last_active":"2024-05-03T21:18:44.826908+0000","last_peered":"2024-05-03T21:18:44.826908+0000","last_clean":"2024-05-03T21:18:44.826908+0000","last_became_active":"2024-05-03T21:18:42.814667+0000","last_became_peered":"2024-05-03T21:18:42.814667+0000","last_unstale":"2024-05-03T21:18:44.826908+0000","last_undegraded":"2024-05-03T21:18:44.826908+0000","last_fullsized":"2024-05-03T21:18:44.826908+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_clean_scrub_stamp":"2024-05-03T21:18:41.788726+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T00:50:47.991863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"82'15","reported_seq":38,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-03T21:18:57.875431+0000","last_change":"2024-05-03T21:18:39.815018+0000","last_active":"2024-05-03T21:18:57.875431+0000","last_peered":"2024-05-03T21:18:57.875431+0000","last_clean":"2024-05-03T21:18:57.875431+0000","last_became_active":"2024-05-03T21:18:39.814319+0000","last_became_peered":"2024-05-03T21:18:39.814319+0000","last_unstale":"2024-05-03T21:18:57.875431+0000","last_undegraded":"2024-05-03T21:18:57.875431+0000","last_fullsized":"2024-05-03T21:18:57.875431+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T21:18:38.766727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T21:18:38.766727+0000","last_clean_scrub_stamp":"2024-05-03T21:18:38.766727+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T00:43:52.230521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":16,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-03T21:18:44.810137+0000","last_change":"2024-05-03T21:18:42.810313+0000","last_active":"2024-05-03T21:18:44.810137+0000","last_peered":"2024-05-03T21:18:44.810137+0000","last_clean":"2024-05-03T21:18:44.810137+0000","last_became_active":"2024-05-03T21:18:42.809732+0000","last_became_peered":"2024-05-03T21:18:42.809732+0000","last_unstale":"2024-05-03T21:18:44.810137+0000","last_undegraded":"2024-05-03T21:18:44.810137+0000","last_fullsized":"2024-05-03T21:18:44.810137+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_clean_scrub_stamp":"2024-05-03T21:18:41.788726+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T01:36:54.077645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"82'2","reported_seq":25,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-03T21:18:59.869422+0000","last_change":"2024-05-03T21:18:39.784067+0000","last_active":"2024-05-03T21:18:59.869422+0000","last_peered":"2024-05-03T21:18:59.869422+0000","last_clean":"2024-05-03T21:18:59.869422+0000","last_became_active":"2024-05-03T21:18:39.783596+0000","last_became_peered":"2024-05-03T21:18:39.783596+0000","last_unstale":"2024-05-03T21:18:59.869422+0000","last_undegraded":"2024-05-03T21:18:59.869422+0000","last_fullsized":"2024-05-03T21:18:59.869422+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T21:18:38.766727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T21:18:38.766727+0000","last_clean_scrub_stamp":"2024-05-03T21:18:38.766727+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T03:10:25.925757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":16,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-03T21:18:44.808547+0000","last_change":"2024-05-03T21:18:42.817617+0000","last_active":"2024-05-03T21:18:44.808547+0000","last_peered":"2024-05-03T21:18:44.808547+0000","last_clean":"2024-05-03T21:18:44.808547+0000","last_became_active":"2024-05-03T21:18:42.817267+0000","last_became_peered":"2024-05-03T21:18:42.817267+0000","last_unstale":"2024-05-03T21:18:44.808547+0000","last_undegraded":"2024-05-03T21:18:44.808547+0000","last_fullsized":"2024-05-03T21:18:44.808547+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_clean_scrub_stamp":"2024-05-03T21:18:41.788726+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T08:00:12.864215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":21,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-03T21:18:44.825193+0000","last_change":"2024-05-03T21:18:39.814213+0000","last_active":"2024-05-03T21:18:44.825193+0000","last_peered":"2024-05-03T21:18:44.825193+0000","last_clean":"2024-05-03T21:18:44.825193+0000","last_became_active":"2024-05-03T21:18:39.813787+0000","last_became_peered":"2024-05-03T21:18:39.813787+0000","last_unstale":"2024-05-03T21:18:44.825193+0000","last_undegraded":"2024-05-03T21:18:44.825193+0000","last_fullsized":"2024-05-03T21:18:44.825193+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T21:18:38.766727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T21:18:38.766727+0000","last_clean_scrub_stamp":"2024-05-03T21:18:38.766727+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T08:45:17.963396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"82'1","reported_seq":17,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-03T21:18:45.303732+0000","last_change":"2024-05-03T21:18:42.821547+0000","last_active":"2024-05-03T21:18:45.303732+0000","last_peered":"2024-05-03T21:18:45.303732+0000","last_clean":"2024-05-03T21:18:45.303732+0000","last_became_active":"2024-05-03T21:18:42.821168+0000","last_became_peered":"2024-05-03T21:18:42.821168+0000","last_unstale":"2024-05-03T21:18:45.303732+0000","last_undegraded":"2024-05-03T21:18:45.303732+0000","last_fullsized":"2024-05-03T21:18:45.303732+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_clean_scrub_stamp":"2024-05-03T21:18:41.788726+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T07:28:25.037648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"82'1","reported_seq":22,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-03T21:18:44.826618+0000","last_change":"2024-05-03T21:18:39.792994+0000","last_active":"2024-05-03T21:18:44.826618+0000","last_peered":"2024-05-03T21:18:44.826618+0000","last_clean":"2024-05-03T21:18:44.826618+0000","last_became_active":"2024-05-03T21:18:39.792125+0000","last_became_peered":"2024-05-03T21:18:39.792125+0000","last_unstale":"2024-05-03T21:18:44.826618+0000","last_undegraded":"2024-05-03T21:18:44.826618+0000","last_fullsized":"2024-05-03T21:18:44.826618+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T21:18:38.766727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T21:18:38.766727+0000","last_clean_scrub_stamp":"2024-05-03T21:18:38.766727+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T03:09:45.917263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":16,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-03T21:18:44.808517+0000","last_change":"2024-05-03T21:18:42.814513+0000","last_active":"2024-05-03T21:18:44.808517+0000","last_peered":"2024-05-03T21:18:44.808517+0000","last_clean":"2024-05-03T21:18:44.808517+0000","last_became_active":"2024-05-03T21:18:42.814097+0000","last_became_peered":"2024-05-03T21:18:42.814097+0000","last_unstale":"2024-05-03T21:18:44.808517+0000","last_undegraded":"2024-05-03T21:18:44.808517+0000","last_fullsized":"2024-05-03T21:18:44.808517+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_clean_scrub_stamp":"2024-05-03T21:18:41.788726+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T06:52:46.731543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":16,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-03T21:18:44.810452+0000","last_change":"2024-05-03T21:18:42.812413+0000","last_active":"2024-05-03T21:18:44.810452+0000","last_peered":"2024-05-03T21:18:44.810452+0000","last_clean":"2024-05-03T21:18:44.810452+0000","last_became_active":"2024-05-03T21:18:42.812065+0000","last_became_peered":"2024-05-03T21:18:42.812065+0000","last_unstale":"2024-05-03T21:18:44.810452+0000","last_undegraded":"2024-05-03T21:18:44.810452+0000","last_fullsized":"2024-05-03T21:18:44.810452+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_clean_scrub_stamp":"2024-05-03T21:18:41.788726+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T07:52:44.489097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"82'4","reported_seq":52,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-03T21:19:25.929861+0000","last_change":"2024-05-03T21:18:39.813982+0000","last_active":"2024-05-03T21:19:25.929861+0000","last_peered":"2024-05-03T21:19:25.929861+0000","last_clean":"2024-05-03T21:19:25.929861+0000","last_became_active":"2024-05-03T21:18:39.813590+0000","last_became_peered":"2024-05-03T21:18:39.813590+0000","last_unstale":"2024-05-03T21:19:25.929861+0000","last_undegraded":"2024-05-03T21:19:25.929861+0000","last_fullsized":"2024-05-03T21:19:25.929861+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T21:18:38.766727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T21:18:38.766727+0000","last_clean_scrub_stamp":"2024-05-03T21:18:38.766727+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T02:30:24.733268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":27,"num_read_kb":27,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":16,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-03T21:18:44.827080+0000","last_change":"2024-05-03T21:18:42.808357+0000","last_active":"2024-05-03T21:18:44.827080+0000","last_peered":"2024-05-03T21:18:44.827080+0000","last_clean":"2024-05-03T21:18:44.827080+0000","last_became_active":"2024-05-03T21:18:42.808023+0000","last_became_peered":"2024-05-03T21:18:42.808023+0000","last_unstale":"2024-05-03T21:18:44.827080+0000","last_undegraded":"2024-05-03T21:18:44.827080+0000","last_fullsized":"2024-05-03T21:18:44.827080+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_clean_scrub_stamp":"2024-05-03T21:18:41.788726+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T08:33:11.713491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"82'3","reported_seq":27,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-03T21:19:00.890481+0000","last_change":"2024-05-03T21:18:39.787052+0000","last_active":"2024-05-03T21:19:00.890481+0000","last_peered":"2024-05-03T21:19:00.890481+0000","last_clean":"2024-05-03T21:19:00.890481+0000","last_became_active":"2024-05-03T21:18:39.786390+0000","last_became_peered":"2024-05-03T21:18:39.786390+0000","last_unstale":"2024-05-03T21:19:00.890481+0000","last_undegraded":"2024-05-03T21:19:00.890481+0000","last_fullsized":"2024-05-03T21:19:00.890481+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T21:18:38.766727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T21:18:38.766727+0000","last_clean_scrub_stamp":"2024-05-03T21:18:38.766727+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T08:18:27.832378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"21'92","reported_seq":95,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-03T21:18:44.826380+0000","last_change":"2024-05-03T21:17:17.301485+0000","last_active":"2024-05-03T21:18:44.826380+0000","last_peered":"2024-05-03T21:18:44.826380+0000","last_clean":"2024-05-03T21:18:44.826380+0000","last_became_active":"2024-05-03T21:17:17.296458+0000","last_became_peered":"2024-05-03T21:17:17.296458+0000","last_unstale":"2024-05-03T21:18:44.826380+0000","last_undegraded":"2024-05-03T21:18:44.826380+0000","last_fullsized":"2024-05-03T21:18:44.826380+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T21:14:00.058823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T21:14:00.058823+0000","last_clean_scrub_stamp":"2024-05-03T21:14:00.058823+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T03:13:14.815888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":16,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-03T21:18:44.822472+0000","last_change":"2024-05-03T21:18:42.818715+0000","last_active":"2024-05-03T21:18:44.822472+0000","last_peered":"2024-05-03T21:18:44.822472+0000","last_clean":"2024-05-03T21:18:44.822472+0000","last_became_active":"2024-05-03T21:18:42.818327+0000","last_became_peered":"2024-05-03T21:18:42.818327+0000","last_unstale":"2024-05-03T21:18:44.822472+0000","last_undegraded":"2024-05-03T21:18:44.822472+0000","last_fullsized":"2024-05-03T21:18:44.822472+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_clean_scrub_stamp":"2024-05-03T21:18:41.788726+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T08:42:31.151515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"82'3","reported_seq":24,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-03T21:18:57.875103+0000","last_change":"2024-05-03T21:18:39.791884+0000","last_active":"2024-05-03T21:18:57.875103+0000","last_peered":"2024-05-03T21:18:57.875103+0000","last_clean":"2024-05-03T21:18:57.875103+0000","last_became_active":"2024-05-03T21:18:39.791465+0000","last_became_peered":"2024-05-03T21:18:39.791465+0000","last_unstale":"2024-05-03T21:18:57.875103+0000","last_undegraded":"2024-05-03T21:18:57.875103+0000","last_fullsized":"2024-05-03T21:18:57.875103+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T21:18:38.766727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T21:18:38.766727+0000","last_clean_scrub_stamp":"2024-05-03T21:18:38.766727+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T07:25:39.572012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"82'1","reported_seq":24,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-03T21:18:56.864376+0000","last_change":"2024-05-03T21:18:39.814671+0000","last_active":"2024-05-03T21:18:56.864376+0000","last_peered":"2024-05-03T21:18:56.864376+0000","last_clean":"2024-05-03T21:18:56.864376+0000","last_became_active":"2024-05-03T21:18:39.814326+0000","last_became_peered":"2024-05-03T21:18:39.814326+0000","last_unstale":"2024-05-03T21:18:56.864376+0000","last_undegraded":"2024-05-03T21:18:56.864376+0000","last_fullsized":"2024-05-03T21:18:56.864376+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T21:18:38.766727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T21:18:38.766727+0000","last_clean_scrub_stamp":"2024-05-03T21:18:38.766727+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T01:21:53.058203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":16,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-03T21:18:44.823177+0000","last_change":"2024-05-03T21:18:42.815525+0000","last_active":"2024-05-03T21:18:44.823177+0000","last_peered":"2024-05-03T21:18:44.823177+0000","last_clean":"2024-05-03T21:18:44.823177+0000","last_became_active":"2024-05-03T21:18:42.815209+0000","last_became_peered":"2024-05-03T21:18:42.815209+0000","last_unstale":"2024-05-03T21:18:44.823177+0000","last_undegraded":"2024-05-03T21:18:44.823177+0000","last_fullsized":"2024-05-03T21:18:44.823177+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_clean_scrub_stamp":"2024-05-03T21:18:41.788726+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T01:02:51.545090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"82'9","reported_seq":30,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-03T21:19:15.825284+0000","last_change":"2024-05-03T21:18:39.784592+0000","last_active":"2024-05-03T21:19:15.825284+0000","last_peered":"2024-05-03T21:19:15.825284+0000","last_clean":"2024-05-03T21:19:15.825284+0000","last_became_active":"2024-05-03T21:18:39.784242+0000","last_became_peered":"2024-05-03T21:18:39.784242+0000","last_unstale":"2024-05-03T21:19:15.825284+0000","last_undegraded":"2024-05-03T21:19:15.825284+0000","last_fullsized":"2024-05-03T21:19:15.825284+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T21:18:38.766727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T21:18:38.766727+0000","last_clean_scrub_stamp":"2024-05-03T21:18:38.766727+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-04T22:23:01.914553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":16,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-03T21:18:44.826924+0000","last_change":"2024-05-03T21:18:42.814361+0000","last_active":"2024-05-03T21:18:44.826924+0000","last_peered":"2024-05-03T21:18:44.826924+0000","last_clean":"2024-05-03T21:18:44.826924+0000","last_became_active":"2024-05-03T21:18:42.813448+0000","last_became_peered":"2024-05-03T21:18:42.813448+0000","last_unstale":"2024-05-03T21:18:44.826924+0000","last_undegraded":"2024-05-03T21:18:44.826924+0000","last_fullsized":"2024-05-03T21:18:44.826924+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_clean_scrub_stamp":"2024-05-03T21:18:41.788726+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T06:21:11.974428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"82'4","reported_seq":26,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-03T21:18:56.863301+0000","last_change":"2024-05-03T21:18:39.905533+0000","last_active":"2024-05-03T21:18:56.863301+0000","last_peered":"2024-05-03T21:18:56.863301+0000","last_clean":"2024-05-03T21:18:56.863301+0000","last_became_active":"2024-05-03T21:18:39.905256+0000","last_became_peered":"2024-05-03T21:18:39.905256+0000","last_unstale":"2024-05-03T21:18:56.863301+0000","last_undegraded":"2024-05-03T21:18:56.863301+0000","last_fullsized":"2024-05-03T21:18:56.863301+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T21:18:38.766727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T21:18:38.766727+0000","last_clean_scrub_stamp":"2024-05-03T21:18:38.766727+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T07:07:12.162628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":16,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-03T21:18:44.825270+0000","last_change":"2024-05-03T21:18:42.821052+0000","last_active":"2024-05-03T21:18:44.825270+0000","last_peered":"2024-05-03T21:18:44.825270+0000","last_clean":"2024-05-03T21:18:44.825270+0000","last_became_active":"2024-05-03T21:18:42.820751+0000","last_became_peered":"2024-05-03T21:18:42.820751+0000","last_unstale":"2024-05-03T21:18:44.825270+0000","last_undegraded":"2024-05-03T21:18:44.825270+0000","last_fullsized":"2024-05-03T21:18:44.825270+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_clean_scrub_stamp":"2024-05-03T21:18:41.788726+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T02:14:45.829778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"82'2","reported_seq":24,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-03T21:18:56.866318+0000","last_change":"2024-05-03T21:18:39.814793+0000","last_active":"2024-05-03T21:18:56.866318+0000","last_peered":"2024-05-03T21:18:56.866318+0000","last_clean":"2024-05-03T21:18:56.866318+0000","last_became_active":"2024-05-03T21:18:39.814427+0000","last_became_peered":"2024-05-03T21:18:39.814427+0000","last_unstale":"2024-05-03T21:18:56.866318+0000","last_undegraded":"2024-05-03T21:18:56.866318+0000","last_fullsized":"2024-05-03T21:18:56.866318+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T21:18:38.766727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T21:18:38.766727+0000","last_clean_scrub_stamp":"2024-05-03T21:18:38.766727+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T01:01:34.673401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":16,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-03T21:18:44.823103+0000","last_change":"2024-05-03T21:18:42.812123+0000","last_active":"2024-05-03T21:18:44.823103+0000","last_peered":"2024-05-03T21:18:44.823103+0000","last_clean":"2024-05-03T21:18:44.823103+0000","last_became_active":"2024-05-03T21:18:42.811799+0000","last_became_peered":"2024-05-03T21:18:42.811799+0000","last_unstale":"2024-05-03T21:18:44.823103+0000","last_undegraded":"2024-05-03T21:18:44.823103+0000","last_fullsized":"2024-05-03T21:18:44.823103+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_clean_scrub_stamp":"2024-05-03T21:18:41.788726+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T05:47:42.995000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"82'3","reported_seq":24,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-03T21:18:56.864980+0000","last_change":"2024-05-03T21:18:39.786729+0000","last_active":"2024-05-03T21:18:56.864980+0000","last_peered":"2024-05-03T21:18:56.864980+0000","last_clean":"2024-05-03T21:18:56.864980+0000","last_became_active":"2024-05-03T21:18:39.786276+0000","last_became_peered":"2024-05-03T21:18:39.786276+0000","last_unstale":"2024-05-03T21:18:56.864980+0000","last_undegraded":"2024-05-03T21:18:56.864980+0000","last_fullsized":"2024-05-03T21:18:56.864980+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T21:18:38.766727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T21:18:38.766727+0000","last_clean_scrub_stamp":"2024-05-03T21:18:38.766727+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T00:47:34.921468+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":16,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-03T21:18:44.808384+0000","last_change":"2024-05-03T21:18:42.818135+0000","last_active":"2024-05-03T21:18:44.808384+0000","last_peered":"2024-05-03T21:18:44.808384+0000","last_clean":"2024-05-03T21:18:44.808384+0000","last_became_active":"2024-05-03T21:18:42.816868+0000","last_became_peered":"2024-05-03T21:18:42.816868+0000","last_unstale":"2024-05-03T21:18:44.808384+0000","last_undegraded":"2024-05-03T21:18:44.808384+0000","last_fullsized":"2024-05-03T21:18:44.808384+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_clean_scrub_stamp":"2024-05-03T21:18:41.788726+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T00:36:26.783820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"82'1","reported_seq":22,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-03T21:18:44.824149+0000","last_change":"2024-05-03T21:18:39.787608+0000","last_active":"2024-05-03T21:18:44.824149+0000","last_peered":"2024-05-03T21:18:44.824149+0000","last_clean":"2024-05-03T21:18:44.824149+0000","last_became_active":"2024-05-03T21:18:39.787041+0000","last_became_peered":"2024-05-03T21:18:39.787041+0000","last_unstale":"2024-05-03T21:18:44.824149+0000","last_undegraded":"2024-05-03T21:18:44.824149+0000","last_fullsized":"2024-05-03T21:18:44.824149+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T21:18:38.766727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T21:18:38.766727+0000","last_clean_scrub_stamp":"2024-05-03T21:18:38.766727+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-04T22:42:48.948711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":16,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-03T21:18:44.826847+0000","last_change":"2024-05-03T21:18:42.819671+0000","last_active":"2024-05-03T21:18:44.826847+0000","last_peered":"2024-05-03T21:18:44.826847+0000","last_clean":"2024-05-03T21:18:44.826847+0000","last_became_active":"2024-05-03T21:18:42.819336+0000","last_became_peered":"2024-05-03T21:18:42.819336+0000","last_unstale":"2024-05-03T21:18:44.826847+0000","last_undegraded":"2024-05-03T21:18:44.826847+0000","last_fullsized":"2024-05-03T21:18:44.826847+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_clean_scrub_stamp":"2024-05-03T21:18:41.788726+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T04:54:03.726522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"82'4","reported_seq":27,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-03T21:18:56.865488+0000","last_change":"2024-05-03T21:18:39.792012+0000","last_active":"2024-05-03T21:18:56.865488+0000","last_peered":"2024-05-03T21:18:56.865488+0000","last_clean":"2024-05-03T21:18:56.865488+0000","last_became_active":"2024-05-03T21:18:39.791638+0000","last_became_peered":"2024-05-03T21:18:39.791638+0000","last_unstale":"2024-05-03T21:18:56.865488+0000","last_undegraded":"2024-05-03T21:18:56.865488+0000","last_fullsized":"2024-05-03T21:18:56.865488+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T21:18:38.766727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T21:18:38.766727+0000","last_clean_scrub_stamp":"2024-05-03T21:18:38.766727+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T07:12:17.441761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":16,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-03T21:18:44.826808+0000","last_change":"2024-05-03T21:18:42.810587+0000","last_active":"2024-05-03T21:18:44.826808+0000","last_peered":"2024-05-03T21:18:44.826808+0000","last_clean":"2024-05-03T21:18:44.826808+0000","last_became_active":"2024-05-03T21:18:42.810274+0000","last_became_peered":"2024-05-03T21:18:42.810274+0000","last_unstale":"2024-05-03T21:18:44.826808+0000","last_undegraded":"2024-05-03T21:18:44.826808+0000","last_fullsized":"2024-05-03T21:18:44.826808+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_clean_scrub_stamp":"2024-05-03T21:18:41.788726+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T02:45:20.891119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"82'3","reported_seq":25,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-03T21:18:57.874152+0000","last_change":"2024-05-03T21:18:39.905882+0000","last_active":"2024-05-03T21:18:57.874152+0000","last_peered":"2024-05-03T21:18:57.874152+0000","last_clean":"2024-05-03T21:18:57.874152+0000","last_became_active":"2024-05-03T21:18:39.905488+0000","last_became_peered":"2024-05-03T21:18:39.905488+0000","last_unstale":"2024-05-03T21:18:57.874152+0000","last_undegraded":"2024-05-03T21:18:57.874152+0000","last_fullsized":"2024-05-03T21:18:57.874152+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T21:18:38.766727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T21:18:38.766727+0000","last_clean_scrub_stamp":"2024-05-03T21:18:38.766727+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T02:02:17.913931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":16,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-03T21:18:44.828561+0000","last_change":"2024-05-03T21:18:42.818693+0000","last_active":"2024-05-03T21:18:44.828561+0000","last_peered":"2024-05-03T21:18:44.828561+0000","last_clean":"2024-05-03T21:18:44.828561+0000","last_became_active":"2024-05-03T21:18:42.818177+0000","last_became_peered":"2024-05-03T21:18:42.818177+0000","last_unstale":"2024-05-03T21:18:44.828561+0000","last_undegraded":"2024-05-03T21:18:44.828561+0000","last_fullsized":"2024-05-03T21:18:44.828561+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_clean_scrub_stamp":"2024-05-03T21:18:41.788726+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T03:05:10.708866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"82'1","reported_seq":24,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-03T21:18:44.825739+0000","last_change":"2024-05-03T21:18:39.792399+0000","last_active":"2024-05-03T21:18:44.825739+0000","last_peered":"2024-05-03T21:18:44.825739+0000","last_clean":"2024-05-03T21:18:44.825739+0000","last_became_active":"2024-05-03T21:18:39.791462+0000","last_became_peered":"2024-05-03T21:18:39.791462+0000","last_unstale":"2024-05-03T21:18:44.825739+0000","last_undegraded":"2024-05-03T21:18:44.825739+0000","last_fullsized":"2024-05-03T21:18:44.825739+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T21:18:38.766727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T21:18:38.766727+0000","last_clean_scrub_stamp":"2024-05-03T21:18:38.766727+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T08:24:23.045438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"82'1","reported_seq":17,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-03T21:18:45.108763+0000","last_change":"2024-05-03T21:18:42.813275+0000","last_active":"2024-05-03T21:18:45.108763+0000","last_peered":"2024-05-03T21:18:45.108763+0000","last_clean":"2024-05-03T21:18:45.108763+0000","last_became_active":"2024-05-03T21:18:42.812640+0000","last_became_peered":"2024-05-03T21:18:42.812640+0000","last_unstale":"2024-05-03T21:18:45.108763+0000","last_undegraded":"2024-05-03T21:18:45.108763+0000","last_fullsized":"2024-05-03T21:18:45.108763+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_clean_scrub_stamp":"2024-05-03T21:18:41.788726+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T01:34:27.071382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"82'2","reported_seq":23,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-03T21:18:44.826343+0000","last_change":"2024-05-03T21:18:39.787487+0000","last_active":"2024-05-03T21:18:44.826343+0000","last_peered":"2024-05-03T21:18:44.826343+0000","last_clean":"2024-05-03T21:18:44.826343+0000","last_became_active":"2024-05-03T21:18:39.787135+0000","last_became_peered":"2024-05-03T21:18:39.787135+0000","last_unstale":"2024-05-03T21:18:44.826343+0000","last_undegraded":"2024-05-03T21:18:44.826343+0000","last_fullsized":"2024-05-03T21:18:44.826343+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T21:18:38.766727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T21:18:38.766727+0000","last_clean_scrub_stamp":"2024-05-03T21:18:38.766727+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T05:21:13.304600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":16,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-03T21:18:44.827864+0000","last_change":"2024-05-03T21:18:42.815682+0000","last_active":"2024-05-03T21:18:44.827864+0000","last_peered":"2024-05-03T21:18:44.827864+0000","last_clean":"2024-05-03T21:18:44.827864+0000","last_became_active":"2024-05-03T21:18:42.813011+0000","last_became_peered":"2024-05-03T21:18:42.813011+0000","last_unstale":"2024-05-03T21:18:44.827864+0000","last_undegraded":"2024-05-03T21:18:44.827864+0000","last_fullsized":"2024-05-03T21:18:44.827864+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_clean_scrub_stamp":"2024-05-03T21:18:41.788726+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T03:47:21.680152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":16,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-03T21:18:44.808499+0000","last_change":"2024-05-03T21:18:42.817574+0000","last_active":"2024-05-03T21:18:44.808499+0000","last_peered":"2024-05-03T21:18:44.808499+0000","last_clean":"2024-05-03T21:18:44.808499+0000","last_became_active":"2024-05-03T21:18:42.817227+0000","last_became_peered":"2024-05-03T21:18:42.817227+0000","last_unstale":"2024-05-03T21:18:44.808499+0000","last_undegraded":"2024-05-03T21:18:44.808499+0000","last_fullsized":"2024-05-03T21:18:44.808499+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_clean_scrub_stamp":"2024-05-03T21:18:41.788726+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-04T23:15:34.784285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":16,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-03T21:18:44.823494+0000","last_change":"2024-05-03T21:18:42.822400+0000","last_active":"2024-05-03T21:18:44.823494+0000","last_peered":"2024-05-03T21:18:44.823494+0000","last_clean":"2024-05-03T21:18:44.823494+0000","last_became_active":"2024-05-03T21:18:42.822000+0000","last_became_peered":"2024-05-03T21:18:42.822000+0000","last_unstale":"2024-05-03T21:18:44.823494+0000","last_undegraded":"2024-05-03T21:18:44.823494+0000","last_fullsized":"2024-05-03T21:18:44.823494+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_clean_scrub_stamp":"2024-05-03T21:18:41.788726+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T06:06:23.492826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":16,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-03T21:18:44.823927+0000","last_change":"2024-05-03T21:18:42.823980+0000","last_active":"2024-05-03T21:18:44.823927+0000","last_peered":"2024-05-03T21:18:44.823927+0000","last_clean":"2024-05-03T21:18:44.823927+0000","last_became_active":"2024-05-03T21:18:42.823619+0000","last_became_peered":"2024-05-03T21:18:42.823619+0000","last_unstale":"2024-05-03T21:18:44.823927+0000","last_undegraded":"2024-05-03T21:18:44.823927+0000","last_fullsized":"2024-05-03T21:18:44.823927+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_clean_scrub_stamp":"2024-05-03T21:18:41.788726+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T08:13:53.104446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":16,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-03T21:18:44.824020+0000","last_change":"2024-05-03T21:18:42.813596+0000","last_active":"2024-05-03T21:18:44.824020+0000","last_peered":"2024-05-03T21:18:44.824020+0000","last_clean":"2024-05-03T21:18:44.824020+0000","last_became_active":"2024-05-03T21:18:42.813238+0000","last_became_peered":"2024-05-03T21:18:42.813238+0000","last_unstale":"2024-05-03T21:18:44.824020+0000","last_undegraded":"2024-05-03T21:18:44.824020+0000","last_fullsized":"2024-05-03T21:18:44.824020+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_clean_scrub_stamp":"2024-05-03T21:18:41.788726+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T05:14:52.712566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":16,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-03T21:18:44.822965+0000","last_change":"2024-05-03T21:18:42.812252+0000","last_active":"2024-05-03T21:18:44.822965+0000","last_peered":"2024-05-03T21:18:44.822965+0000","last_clean":"2024-05-03T21:18:44.822965+0000","last_became_active":"2024-05-03T21:18:42.811917+0000","last_became_peered":"2024-05-03T21:18:42.811917+0000","last_unstale":"2024-05-03T21:18:44.822965+0000","last_undegraded":"2024-05-03T21:18:44.822965+0000","last_fullsized":"2024-05-03T21:18:44.822965+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_clean_scrub_stamp":"2024-05-03T21:18:41.788726+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T00:55:09.512412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":16,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-03T21:18:44.808264+0000","last_change":"2024-05-03T21:18:42.819345+0000","last_active":"2024-05-03T21:18:44.808264+0000","last_peered":"2024-05-03T21:18:44.808264+0000","last_clean":"2024-05-03T21:18:44.808264+0000","last_became_active":"2024-05-03T21:18:42.817901+0000","last_became_peered":"2024-05-03T21:18:42.817901+0000","last_unstale":"2024-05-03T21:18:44.808264+0000","last_undegraded":"2024-05-03T21:18:44.808264+0000","last_fullsized":"2024-05-03T21:18:44.808264+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_clean_scrub_stamp":"2024-05-03T21:18:41.788726+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T07:25:30.498261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":16,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-03T21:18:44.825269+0000","last_change":"2024-05-03T21:18:42.825009+0000","last_active":"2024-05-03T21:18:44.825269+0000","last_peered":"2024-05-03T21:18:44.825269+0000","last_clean":"2024-05-03T21:18:44.825269+0000","last_became_active":"2024-05-03T21:18:42.824626+0000","last_became_peered":"2024-05-03T21:18:42.824626+0000","last_unstale":"2024-05-03T21:18:44.825269+0000","last_undegraded":"2024-05-03T21:18:44.825269+0000","last_fullsized":"2024-05-03T21:18:44.825269+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_clean_scrub_stamp":"2024-05-03T21:18:41.788726+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T06:48:17.760340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":16,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-03T21:18:44.826847+0000","last_change":"2024-05-03T21:18:42.812305+0000","last_active":"2024-05-03T21:18:44.826847+0000","last_peered":"2024-05-03T21:18:44.826847+0000","last_clean":"2024-05-03T21:18:44.826847+0000","last_became_active":"2024-05-03T21:18:42.811738+0000","last_became_peered":"2024-05-03T21:18:42.811738+0000","last_unstale":"2024-05-03T21:18:44.826847+0000","last_undegraded":"2024-05-03T21:18:44.826847+0000","last_fullsized":"2024-05-03T21:18:44.826847+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_clean_scrub_stamp":"2024-05-03T21:18:41.788726+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-04T21:41:24.312517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":16,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-03T21:18:44.810058+0000","last_change":"2024-05-03T21:18:42.817306+0000","last_active":"2024-05-03T21:18:44.810058+0000","last_peered":"2024-05-03T21:18:44.810058+0000","last_clean":"2024-05-03T21:18:44.810058+0000","last_became_active":"2024-05-03T21:18:42.816768+0000","last_became_peered":"2024-05-03T21:18:42.816768+0000","last_unstale":"2024-05-03T21:18:44.810058+0000","last_undegraded":"2024-05-03T21:18:44.810058+0000","last_fullsized":"2024-05-03T21:18:44.810058+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_clean_scrub_stamp":"2024-05-03T21:18:41.788726+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-04T23:24:42.055568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":16,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-03T21:18:44.826798+0000","last_change":"2024-05-03T21:18:42.816008+0000","last_active":"2024-05-03T21:18:44.826798+0000","last_peered":"2024-05-03T21:18:44.826798+0000","last_clean":"2024-05-03T21:18:44.826798+0000","last_became_active":"2024-05-03T21:18:42.815653+0000","last_became_peered":"2024-05-03T21:18:42.815653+0000","last_unstale":"2024-05-03T21:18:44.826798+0000","last_undegraded":"2024-05-03T21:18:44.826798+0000","last_fullsized":"2024-05-03T21:18:44.826798+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_clean_scrub_stamp":"2024-05-03T21:18:41.788726+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-04T23:15:22.986562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":16,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-03T21:18:44.823562+0000","last_change":"2024-05-03T21:18:42.814820+0000","last_active":"2024-05-03T21:18:44.823562+0000","last_peered":"2024-05-03T21:18:44.823562+0000","last_clean":"2024-05-03T21:18:44.823562+0000","last_became_active":"2024-05-03T21:18:42.814498+0000","last_became_peered":"2024-05-03T21:18:42.814498+0000","last_unstale":"2024-05-03T21:18:44.823562+0000","last_undegraded":"2024-05-03T21:18:44.823562+0000","last_fullsized":"2024-05-03T21:18:44.823562+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_clean_scrub_stamp":"2024-05-03T21:18:41.788726+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T01:35:15.727585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":16,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-03T21:18:44.823542+0000","last_change":"2024-05-03T21:18:42.814701+0000","last_active":"2024-05-03T21:18:44.823542+0000","last_peered":"2024-05-03T21:18:44.823542+0000","last_clean":"2024-05-03T21:18:44.823542+0000","last_became_active":"2024-05-03T21:18:42.814394+0000","last_became_peered":"2024-05-03T21:18:42.814394+0000","last_unstale":"2024-05-03T21:18:44.823542+0000","last_undegraded":"2024-05-03T21:18:44.823542+0000","last_fullsized":"2024-05-03T21:18:44.823542+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_clean_scrub_stamp":"2024-05-03T21:18:41.788726+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T04:50:05.941873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":16,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-03T21:18:44.808193+0000","last_change":"2024-05-03T21:18:42.815987+0000","last_active":"2024-05-03T21:18:44.808193+0000","last_peered":"2024-05-03T21:18:44.808193+0000","last_clean":"2024-05-03T21:18:44.808193+0000","last_became_active":"2024-05-03T21:18:42.815613+0000","last_became_peered":"2024-05-03T21:18:42.815613+0000","last_unstale":"2024-05-03T21:18:44.808193+0000","last_undegraded":"2024-05-03T21:18:44.808193+0000","last_fullsized":"2024-05-03T21:18:44.808193+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_clean_scrub_stamp":"2024-05-03T21:18:41.788726+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T03:57:01.997504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4751360,"data_stored":4722944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":92,"ondisk_log_size":92,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":54502,"num_objects":60,"num_object_clones":0,"num_object_copies":180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":0,"num_read":182,"num_read_kb":148,"num_write":156,"num_write_kb":97,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":393216,"data_stored":163506,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":134,"ondisk_log_size":134,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":74,"seq":317827579928,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27876,"kb_used_data":860,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717276,"statfs":{"total":95995035648,"available":95966490624,"internally_reserved":0,"allocated":880640,"data_stored":424993,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":68,"seq":292057776156,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28548,"kb_used_data":1532,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716604,"statfs":{"total":95995035648,"available":95965802496,"internally_reserved":0,"allocated":1568768,"data_stored":1066975,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":63,"seq":270582939682,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32052,"kb_used_data":876,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713100,"statfs":{"total":95995035648,"available":95962214400,"internally_reserved":0,"allocated":897024,"data_stored":425229,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":57,"seq":244813135912,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32052,"kb_used_data":876,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713100,"statfs":{"total":95995035648,"available":95962214400,"internally_reserved":0,"allocated":897024,"data_stored":425149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":50,"seq":214748364843,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28472,"kb_used_data":1456,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716680,"statfs":{"total":95995035648,"available":95965880320,"internally_reserved":0,"allocated":1490944,"data_stored":1017470,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":45,"seq":193273528369,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27880,"kb_used_data":864,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717272,"statfs":{"total":95995035648,"available":95966486528,"internally_reserved":0,"allocated":884736,"data_stored":424618,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574922,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36252,"kb_used_data":916,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708900,"statfs":{"total":95995035648,"available":95957913600,"internally_reserved":0,"allocated":937984,"data_stored":474521,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738447,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36220,"kb_used_data":884,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708932,"statfs":{"total":95995035648,"available":95957946368,"internally_reserved":0,"allocated":905216,"data_stored":425889,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411396,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36276,"kb_used_data":940,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708876,"statfs":{"total":95995035648,"available":95957889024,"internally_reserved":0,"allocated":962560,"data_stored":476262,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":27,"seq":115964117055,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36216,"kb_used_data":880,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708936,"statfs":{"total":95995035648,"available":95957950464,"internally_reserved":0,"allocated":901120,"data_stored":424929,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920826,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27884,"kb_used_data":868,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717268,"statfs":{"total":95995035648,"available":95966482432,"internally_reserved":0,"allocated":888832,"data_stored":425446,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724599,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32644,"kb_used_data":1468,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712508,"statfs":{"total":95995035648,"available":95961608192,"internally_reserved":0,"allocated":1503232,"data_stored":1016763,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":50110,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":51851,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1035,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1867,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2574,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":94208,"data_stored":52079,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-05-03T21:19:30.356 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-05-03T21:19:30.356 INFO:tasks.cephadm:Setting up client nodes... 2024-05-03T21:19:30.357 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9a047fe7599a549042d06a3fb5b6813a203eabea shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 713bc57a-0991-11ef-bc97-c7b262605968 -- ceph auth get-or-create client.0 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-05-03T21:19:30.608 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/713bc57a-0991-11ef-bc97-c7b262605968/mon.a/config 2024-05-03T21:19:30.983 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:19:30 smithi067 ceph-mon[29730]: from='client.14742 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-03T21:19:30.983 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:19:30 smithi067 ceph-mon[29730]: pgmap v310: 97 pgs: 97 active+clean; 630 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 2.2 KiB/s wr, 5 op/s 2024-05-03T21:19:31.245 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:19:30 smithi082 ceph-mon[32933]: from='client.14742 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-03T21:19:31.245 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:19:30 smithi082 ceph-mon[32933]: pgmap v310: 97 pgs: 97 active+clean; 630 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 2.2 KiB/s wr, 5 op/s 2024-05-03T21:19:31.324 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:19:30 smithi138 ceph-mon[33110]: from='client.14742 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-03T21:19:31.324 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:19:30 smithi138 ceph-mon[33110]: pgmap v310: 97 pgs: 97 active+clean; 630 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 2.2 KiB/s wr, 5 op/s 2024-05-03T21:19:32.281 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:32.276+0000 7f06dd65f700 1 -- 172.21.15.67:0/3397878893 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f06d8104750 msgr2=0x7f06d8104bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:19:32.281 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:32.276+0000 7f06dd65f700 1 --2- 172.21.15.67:0/3397878893 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f06d8104750 0x7f06d8104bb0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f06cc0099c0 tx=0x7f06cc009cd0 comp rx=0 tx=0).stop 2024-05-03T21:19:32.281 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:32.276+0000 7f06dd65f700 1 -- 172.21.15.67:0/3397878893 shutdown_connections 2024-05-03T21:19:32.281 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:32.276+0000 7f06dd65f700 1 --2- 172.21.15.67:0/3397878893 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f06d81050f0 0x7f06d810b9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:19:32.281 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:32.276+0000 7f06dd65f700 1 --2- 172.21.15.67:0/3397878893 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f06d8103550 0x7f06d8103970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:19:32.281 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:32.276+0000 7f06dd65f700 1 --2- 172.21.15.67:0/3397878893 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f06d8104750 0x7f06d8104bb0 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:19:32.281 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:32.276+0000 7f06dd65f700 1 -- 172.21.15.67:0/3397878893 >> 172.21.15.67:0/3397878893 conn(0x7f06d80feaf0 msgr2=0x7f06d8100f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T21:19:32.281 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:32.277+0000 7f06dd65f700 1 -- 172.21.15.67:0/3397878893 shutdown_connections 2024-05-03T21:19:32.281 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:32.277+0000 7f06dd65f700 1 -- 172.21.15.67:0/3397878893 wait complete. 2024-05-03T21:19:32.282 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:32.277+0000 7f06dd65f700 1 Processor -- start 2024-05-03T21:19:32.282 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:32.277+0000 7f06dd65f700 1 -- start start 2024-05-03T21:19:32.282 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:32.277+0000 7f06dd65f700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f06d8103550 0x7f06d8077240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:19:32.282 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:32.277+0000 7f06dd65f700 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f06d8104750 0x7f06d8075890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:19:32.282 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:32.277+0000 7f06dd65f700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f06d81050f0 0x7f06d8075dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:19:32.282 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:32.277+0000 7f06dd65f700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f06d810f2f0 con 0x7f06d8103550 2024-05-03T21:19:32.282 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:32.277+0000 7f06dd65f700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f06d810f170 con 0x7f06d81050f0 2024-05-03T21:19:32.282 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:32.277+0000 7f06dd65f700 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f06d810f470 con 0x7f06d8104750 2024-05-03T21:19:32.283 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:32.278+0000 7f06d6ffd700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f06d8103550 0x7f06d8077240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:19:32.283 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:32.278+0000 7f06d6ffd700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f06d8103550 0x7f06d8077240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:50268/0 (socket says 172.21.15.67:50268) 2024-05-03T21:19:32.283 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:32.278+0000 7f06d6ffd700 1 -- 172.21.15.67:0/935724302 learned_addr learned my addr 172.21.15.67:0/935724302 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-03T21:19:32.283 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:32.278+0000 7f06d67fc700 1 --2- 172.21.15.67:0/935724302 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f06d8104750 0x7f06d8075890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:19:32.283 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:32.278+0000 7f06d77fe700 1 --2- 172.21.15.67:0/935724302 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f06d81050f0 0x7f06d8075dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:19:32.283 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:32.278+0000 7f06d6ffd700 1 -- 172.21.15.67:0/935724302 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f06d8104750 msgr2=0x7f06d8075890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:19:32.283 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:32.278+0000 7f06d6ffd700 1 --2- 172.21.15.67:0/935724302 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f06d8104750 0x7f06d8075890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:19:32.283 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:32.278+0000 7f06d6ffd700 1 -- 172.21.15.67:0/935724302 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f06d81050f0 msgr2=0x7f06d8075dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:19:32.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:32.278+0000 7f06d6ffd700 1 --2- 172.21.15.67:0/935724302 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f06d81050f0 0x7f06d8075dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:19:32.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:32.278+0000 7f06d6ffd700 1 -- 172.21.15.67:0/935724302 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f06d8076680 con 0x7f06d8103550 2024-05-03T21:19:32.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:32.278+0000 7f06d6ffd700 1 --2- 172.21.15.67:0/935724302 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f06d8103550 0x7f06d8077240 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f06c800b500 tx=0x7f06c800b810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T21:19:32.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:32.278+0000 7f06bffff700 1 -- 172.21.15.67:0/935724302 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06c8015020 con 0x7f06d8103550 2024-05-03T21:19:32.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:32.278+0000 7f06dd65f700 1 -- 172.21.15.67:0/935724302 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f06d81a3980 con 0x7f06d8103550 2024-05-03T21:19:32.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:32.278+0000 7f06bffff700 1 -- 172.21.15.67:0/935724302 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(28 keys) ==== 1170+0+0 (secure 0 0 0) 0x7f06c8011d30 con 0x7f06d8103550 2024-05-03T21:19:32.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:32.278+0000 7f06bffff700 1 -- 172.21.15.67:0/935724302 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06c8010e00 con 0x7f06d8103550 2024-05-03T21:19:32.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:32.278+0000 7f06dd65f700 1 -- 172.21.15.67:0/935724302 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f06d81a3f40 con 0x7f06d8103550 2024-05-03T21:19:32.286 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:32.280+0000 7f06dd65f700 1 -- 172.21.15.67:0/935724302 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f06d80519e0 con 0x7f06d8103550 2024-05-03T21:19:32.286 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:32.282+0000 7f06bffff700 1 -- 172.21.15.67:0/935724302 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 17) ==== 149703+0+0 (secure 0 0 0) 0x7f06c8018020 con 0x7f06d8103550 2024-05-03T21:19:32.286 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:32.282+0000 7f06bffff700 1 --2- 172.21.15.67:0/935724302 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f06c00b29b0 0x7f06c00b4e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:19:32.286 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:32.283+0000 7f06d67fc700 1 --2- 172.21.15.67:0/935724302 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f06c00b29b0 0x7f06c00b4e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:19:32.286 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:32.283+0000 7f06bffff700 1 -- 172.21.15.67:0/935724302 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7979+0+0 (secure 0 0 0) 0x7f06c80e1de0 con 0x7f06d8103550 2024-05-03T21:19:32.286 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:32.283+0000 7f06d67fc700 1 --2- 172.21.15.67:0/935724302 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f06c00b29b0 0x7f06c00b4e70 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f06cc000c00 tx=0x7f06cc005fb0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T21:19:32.288 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:32.286+0000 7f06bffff700 1 -- 172.21.15.67:0/935724302 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191839 (secure 0 0 0) 0x7f06c801f020 con 0x7f06d8103550 2024-05-03T21:19:32.593 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:32.591+0000 7f06dd65f700 1 -- 172.21.15.67:0/935724302 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} v 0) -- 0x7f06d802e5e0 con 0x7f06d8103550 2024-05-03T21:19:32.595 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:32.593+0000 7f06bffff700 1 -- 172.21.15.67:0/935724302 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]=0 v32) ==== 170+0+59 (secure 0 0 0) 0x7f06c80a9c60 con 0x7f06d8103550 2024-05-03T21:19:32.595 INFO:teuthology.orchestra.run.smithi067.stdout:[client.0] 2024-05-03T21:19:32.596 INFO:teuthology.orchestra.run.smithi067.stdout: key = AQDkVDVmIU5VIxAAKqQkfrrWl4pKzYBUaGzU/A== 2024-05-03T21:19:32.598 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:32.596+0000 7f06dd65f700 1 -- 172.21.15.67:0/935724302 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f06c00b29b0 msgr2=0x7f06c00b4e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:19:32.598 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:32.596+0000 7f06dd65f700 1 --2- 172.21.15.67:0/935724302 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f06c00b29b0 0x7f06c00b4e70 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f06cc000c00 tx=0x7f06cc005fb0 comp rx=0 tx=0).stop 2024-05-03T21:19:32.598 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:32.596+0000 7f06dd65f700 1 -- 172.21.15.67:0/935724302 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f06d8103550 msgr2=0x7f06d8077240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:19:32.598 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:32.596+0000 7f06dd65f700 1 --2- 172.21.15.67:0/935724302 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f06d8103550 0x7f06d8077240 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f06c800b500 tx=0x7f06c800b810 comp rx=0 tx=0).stop 2024-05-03T21:19:32.598 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:32.596+0000 7f06dd65f700 1 -- 172.21.15.67:0/935724302 shutdown_connections 2024-05-03T21:19:32.598 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:32.596+0000 7f06dd65f700 1 --2- 172.21.15.67:0/935724302 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f06c00b29b0 0x7f06c00b4e70 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:19:32.599 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:32.596+0000 7f06dd65f700 1 --2- 172.21.15.67:0/935724302 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f06d8103550 0x7f06d8077240 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:19:32.599 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:32.596+0000 7f06dd65f700 1 --2- 172.21.15.67:0/935724302 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f06d81050f0 0x7f06d8075dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:19:32.599 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:32.596+0000 7f06dd65f700 1 --2- 172.21.15.67:0/935724302 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f06d8104750 0x7f06d8075890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:19:32.599 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:32.596+0000 7f06dd65f700 1 -- 172.21.15.67:0/935724302 >> 172.21.15.67:0/935724302 conn(0x7f06d80feaf0 msgr2=0x7f06d8107980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T21:19:32.599 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:32.596+0000 7f06dd65f700 1 -- 172.21.15.67:0/935724302 shutdown_connections 2024-05-03T21:19:32.599 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:32.596+0000 7f06dd65f700 1 -- 172.21.15.67:0/935724302 wait complete. 2024-05-03T21:19:33.139 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-05-03T21:19:33.139 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd of=/etc/ceph/ceph.client.0.keyring 2024-05-03T21:19:33.139 DEBUG:teuthology.orchestra.run.smithi067:> sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-05-03T21:19:33.176 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-05-03T21:19:33.176 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2024-05-03T21:19:33.176 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9a047fe7599a549042d06a3fb5b6813a203eabea shell --fsid 713bc57a-0991-11ef-bc97-c7b262605968 -- ceph mgr dump --format=json 2024-05-03T21:19:33.221 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:19:32 smithi067 ceph-mon[29730]: pgmap v311: 97 pgs: 97 active+clean; 632 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 2.4 KiB/s wr, 7 op/s 2024-05-03T21:19:33.221 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:19:32 smithi067 ceph-mon[29730]: from='client.? 172.21.15.67:0/935724302' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-05-03T21:19:33.221 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:19:32 smithi067 ceph-mon[29730]: from='client.? 172.21.15.67:0/935724302' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-05-03T21:19:33.245 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:19:32 smithi082 ceph-mon[32933]: pgmap v311: 97 pgs: 97 active+clean; 632 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 2.4 KiB/s wr, 7 op/s 2024-05-03T21:19:33.245 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:19:32 smithi082 ceph-mon[32933]: from='client.? 172.21.15.67:0/935724302' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-05-03T21:19:33.245 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:19:32 smithi082 ceph-mon[32933]: from='client.? 172.21.15.67:0/935724302' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-05-03T21:19:33.324 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:19:32 smithi138 ceph-mon[33110]: pgmap v311: 97 pgs: 97 active+clean; 632 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 2.4 KiB/s wr, 7 op/s 2024-05-03T21:19:33.324 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:19:32 smithi138 ceph-mon[33110]: from='client.? 172.21.15.67:0/935724302' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-05-03T21:19:33.324 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:19:32 smithi138 ceph-mon[33110]: from='client.? 172.21.15.67:0/935724302' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-05-03T21:19:33.463 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/713bc57a-0991-11ef-bc97-c7b262605968/mon.a/config 2024-05-03T21:19:34.221 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:19:33 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.0", "id": [8, 4]}]: dispatch 2024-05-03T21:19:34.221 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:19:33 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.a", "id": [2, 11]}]: dispatch 2024-05-03T21:19:34.221 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:19:33 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.b", "id": [2, 4]}]: dispatch 2024-05-03T21:19:34.221 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:19:33 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.10", "id": [8, 11]}]: dispatch 2024-05-03T21:19:34.221 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:19:33 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1", "id": [4, 9]}]: dispatch 2024-05-03T21:19:34.221 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:19:33 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1b", "id": [5, 9]}]: dispatch 2024-05-03T21:19:34.222 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:19:33 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.21", "id": [4, 8]}]: dispatch 2024-05-03T21:19:34.222 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:19:33 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3c", "id": [4, 9]}]: dispatch 2024-05-03T21:19:34.245 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:19:33 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.0", "id": [8, 4]}]: dispatch 2024-05-03T21:19:34.245 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:19:33 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.a", "id": [2, 11]}]: dispatch 2024-05-03T21:19:34.246 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:19:33 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.b", "id": [2, 4]}]: dispatch 2024-05-03T21:19:34.246 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:19:33 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.10", "id": [8, 11]}]: dispatch 2024-05-03T21:19:34.246 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:19:33 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1", "id": [4, 9]}]: dispatch 2024-05-03T21:19:34.246 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:19:33 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1b", "id": [5, 9]}]: dispatch 2024-05-03T21:19:34.246 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:19:33 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.21", "id": [4, 8]}]: dispatch 2024-05-03T21:19:34.246 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:19:33 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3c", "id": [4, 9]}]: dispatch 2024-05-03T21:19:34.324 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:19:33 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.0", "id": [8, 4]}]: dispatch 2024-05-03T21:19:34.325 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:19:33 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.a", "id": [2, 11]}]: dispatch 2024-05-03T21:19:34.325 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:19:33 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.b", "id": [2, 4]}]: dispatch 2024-05-03T21:19:34.325 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:19:33 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.10", "id": [8, 11]}]: dispatch 2024-05-03T21:19:34.325 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:19:33 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1", "id": [4, 9]}]: dispatch 2024-05-03T21:19:34.325 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:19:33 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1b", "id": [5, 9]}]: dispatch 2024-05-03T21:19:34.325 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:19:33 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.21", "id": [4, 8]}]: dispatch 2024-05-03T21:19:34.325 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:19:33 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3c", "id": [4, 9]}]: dispatch 2024-05-03T21:19:35.052 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:35.046+0000 7f7d54c33700 1 -- 172.21.15.67:0/770085872 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7d50102630 msgr2=0x7f7d50102a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:19:35.052 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:35.046+0000 7f7d54c33700 1 --2- 172.21.15.67:0/770085872 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7d50102630 0x7f7d50102a10 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f7d440099c0 tx=0x7f7d44009cd0 comp rx=0 tx=0).stop 2024-05-03T21:19:35.052 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:35.046+0000 7f7d54c33700 1 -- 172.21.15.67:0/770085872 shutdown_connections 2024-05-03T21:19:35.052 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:35.046+0000 7f7d54c33700 1 --2- 172.21.15.67:0/770085872 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7d50102630 0x7f7d50102a10 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:19:35.052 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:35.046+0000 7f7d54c33700 1 --2- 172.21.15.67:0/770085872 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f7d501076c0 0x7f7d5010f290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:19:35.053 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:35.046+0000 7f7d54c33700 1 --2- 172.21.15.67:0/770085872 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f7d50102fe0 0x7f7d50106fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:19:35.053 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:35.046+0000 7f7d54c33700 1 -- 172.21.15.67:0/770085872 >> 172.21.15.67:0/770085872 conn(0x7f7d50079350 msgr2=0x7f7d50079760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T21:19:35.053 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:35.046+0000 7f7d54c33700 1 -- 172.21.15.67:0/770085872 shutdown_connections 2024-05-03T21:19:35.053 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:35.046+0000 7f7d54c33700 1 -- 172.21.15.67:0/770085872 wait complete. 2024-05-03T21:19:35.053 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:35.047+0000 7f7d54c33700 1 Processor -- start 2024-05-03T21:19:35.054 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:35.047+0000 7f7d54c33700 1 -- start start 2024-05-03T21:19:35.054 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:35.047+0000 7f7d54c33700 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f7d50102630 0x7f7d5019ab40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:19:35.054 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:35.047+0000 7f7d54c33700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7d50102fe0 0x7f7d5019b080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:19:35.054 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:35.047+0000 7f7d54c33700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f7d501076c0 0x7f7d501a5bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:19:35.054 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:35.047+0000 7f7d54c33700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f7d50112800 con 0x7f7d50102fe0 2024-05-03T21:19:35.055 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:35.047+0000 7f7d54c33700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f7d50112680 con 0x7f7d501076c0 2024-05-03T21:19:35.055 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:35.047+0000 7f7d54c33700 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f7d50112980 con 0x7f7d50102630 2024-05-03T21:19:35.055 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:35.047+0000 7f7d4dd9b700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7d50102fe0 0x7f7d5019b080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:19:35.055 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:35.047+0000 7f7d4dd9b700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7d50102fe0 0x7f7d5019b080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:50282/0 (socket says 172.21.15.67:50282) 2024-05-03T21:19:35.055 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:35.047+0000 7f7d4dd9b700 1 -- 172.21.15.67:0/2722813823 learned_addr learned my addr 172.21.15.67:0/2722813823 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-03T21:19:35.055 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:35.048+0000 7f7d4e59c700 1 --2- 172.21.15.67:0/2722813823 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f7d50102630 0x7f7d5019ab40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:19:35.056 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:35.048+0000 7f7d4ed9d700 1 --2- 172.21.15.67:0/2722813823 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f7d501076c0 0x7f7d501a5bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:19:35.056 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:35.048+0000 7f7d4dd9b700 1 -- 172.21.15.67:0/2722813823 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f7d50102630 msgr2=0x7f7d5019ab40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:19:35.056 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:35.048+0000 7f7d4dd9b700 1 --2- 172.21.15.67:0/2722813823 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f7d50102630 0x7f7d5019ab40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:19:35.056 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:35.048+0000 7f7d4dd9b700 1 -- 172.21.15.67:0/2722813823 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f7d501076c0 msgr2=0x7f7d501a5bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:19:35.056 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:35.048+0000 7f7d4dd9b700 1 --2- 172.21.15.67:0/2722813823 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f7d501076c0 0x7f7d501a5bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:19:35.056 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:35.048+0000 7f7d4dd9b700 1 -- 172.21.15.67:0/2722813823 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7d501a6100 con 0x7f7d50102fe0 2024-05-03T21:19:35.057 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:35.048+0000 7f7d4dd9b700 1 --2- 172.21.15.67:0/2722813823 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7d50102fe0 0x7f7d5019b080 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f7d4000e9f0 tx=0x7f7d4000ed00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T21:19:35.057 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:35.048+0000 7f7d3f7fe700 1 -- 172.21.15.67:0/2722813823 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d40004a30 con 0x7f7d50102fe0 2024-05-03T21:19:35.057 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:35.048+0000 7f7d3f7fe700 1 -- 172.21.15.67:0/2722813823 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(28 keys) ==== 1170+0+0 (secure 0 0 0) 0x7f7d4000ccc0 con 0x7f7d50102fe0 2024-05-03T21:19:35.057 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:35.048+0000 7f7d54c33700 1 -- 172.21.15.67:0/2722813823 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7d501a6390 con 0x7f7d50102fe0 2024-05-03T21:19:35.057 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:35.048+0000 7f7d54c33700 1 -- 172.21.15.67:0/2722813823 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7d501a68a0 con 0x7f7d50102fe0 2024-05-03T21:19:35.057 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:35.048+0000 7f7d3f7fe700 1 -- 172.21.15.67:0/2722813823 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d40010430 con 0x7f7d50102fe0 2024-05-03T21:19:35.058 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:35.049+0000 7f7d54c33700 1 -- 172.21.15.67:0/2722813823 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7d5006c220 con 0x7f7d50102fe0 2024-05-03T21:19:35.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:35.051+0000 7f7d3f7fe700 1 -- 172.21.15.67:0/2722813823 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 17) ==== 149703+0+0 (secure 0 0 0) 0x7f7d4000c7c0 con 0x7f7d50102fe0 2024-05-03T21:19:35.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:35.051+0000 7f7d3f7fe700 1 --2- 172.21.15.67:0/2722813823 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f7d380b2990 0x7f7d380b4e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:19:35.060 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:35.052+0000 7f7d4e59c700 1 --2- 172.21.15.67:0/2722813823 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f7d380b2990 0x7f7d380b4e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:19:35.060 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:35.052+0000 7f7d3f7fe700 1 -- 172.21.15.67:0/2722813823 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 8211+0+0 (secure 0 0 0) 0x7f7d400e2760 con 0x7f7d50102fe0 2024-05-03T21:19:35.060 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:19:34 smithi067 ceph-mon[29730]: pgmap v312: 97 pgs: 97 active+clean; 632 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 255 B/s wr, 6 op/s 2024-05-03T21:19:35.060 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:19:34 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.0", "id": [8, 4]}]': finished 2024-05-03T21:19:35.060 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:19:34 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.a", "id": [2, 11]}]': finished 2024-05-03T21:19:35.060 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:19:34 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.b", "id": [2, 4]}]': finished 2024-05-03T21:19:35.060 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:19:34 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.10", "id": [8, 11]}]': finished 2024-05-03T21:19:35.060 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:19:34 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1", "id": [4, 9]}]': finished 2024-05-03T21:19:35.060 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:19:34 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1b", "id": [5, 9]}]': finished 2024-05-03T21:19:35.060 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:19:34 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.21", "id": [4, 8]}]': finished 2024-05-03T21:19:35.061 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:19:34 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3c", "id": [4, 9]}]': finished 2024-05-03T21:19:35.061 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:19:34 smithi067 ceph-mon[29730]: osdmap e83: 12 total, 12 up, 12 in 2024-05-03T21:19:35.061 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:35.052+0000 7f7d4e59c700 1 --2- 172.21.15.67:0/2722813823 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f7d380b2990 0x7f7d380b4e50 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f7d44000c00 tx=0x7f7d44005fb0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T21:19:35.061 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:35.055+0000 7f7d3f7fe700 1 -- 172.21.15.67:0/2722813823 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191839 (secure 0 0 0) 0x7f7d40054d70 con 0x7f7d50102fe0 2024-05-03T21:19:35.245 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:19:34 smithi082 ceph-mon[32933]: pgmap v312: 97 pgs: 97 active+clean; 632 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 255 B/s wr, 6 op/s 2024-05-03T21:19:35.246 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:19:34 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.0", "id": [8, 4]}]': finished 2024-05-03T21:19:35.246 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:19:34 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.a", "id": [2, 11]}]': finished 2024-05-03T21:19:35.246 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:19:34 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.b", "id": [2, 4]}]': finished 2024-05-03T21:19:35.246 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:19:34 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.10", "id": [8, 11]}]': finished 2024-05-03T21:19:35.246 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:19:34 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1", "id": [4, 9]}]': finished 2024-05-03T21:19:35.246 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:19:34 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1b", "id": [5, 9]}]': finished 2024-05-03T21:19:35.246 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:19:34 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.21", "id": [4, 8]}]': finished 2024-05-03T21:19:35.246 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:19:34 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3c", "id": [4, 9]}]': finished 2024-05-03T21:19:35.246 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:19:34 smithi082 ceph-mon[32933]: osdmap e83: 12 total, 12 up, 12 in 2024-05-03T21:19:35.325 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:19:34 smithi138 ceph-mon[33110]: pgmap v312: 97 pgs: 97 active+clean; 632 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 255 B/s wr, 6 op/s 2024-05-03T21:19:35.325 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:19:34 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.0", "id": [8, 4]}]': finished 2024-05-03T21:19:35.325 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:19:34 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.a", "id": [2, 11]}]': finished 2024-05-03T21:19:35.325 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:19:34 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.b", "id": [2, 4]}]': finished 2024-05-03T21:19:35.325 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:19:34 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.10", "id": [8, 11]}]': finished 2024-05-03T21:19:35.325 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:19:34 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1", "id": [4, 9]}]': finished 2024-05-03T21:19:35.325 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:19:34 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1b", "id": [5, 9]}]': finished 2024-05-03T21:19:35.325 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:19:34 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.21", "id": [4, 8]}]': finished 2024-05-03T21:19:35.325 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:19:34 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3c", "id": [4, 9]}]': finished 2024-05-03T21:19:35.325 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:19:34 smithi138 ceph-mon[33110]: osdmap e83: 12 total, 12 up, 12 in 2024-05-03T21:19:35.333 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:35.331+0000 7f7d54c33700 1 -- 172.21.15.67:0/2722813823 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) -- 0x7f7d5002e510 con 0x7f7d50102fe0 2024-05-03T21:19:35.338 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:35.336+0000 7f7d3f7fe700 1 -- 172.21.15.67:0/2722813823 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v17) ==== 74+0+288642 (secure 0 0 0) 0x7f7d400aa2d0 con 0x7f7d50102fe0 2024-05-03T21:19:35.342 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-05-03T21:19:35.344 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:35.340+0000 7f7d54c33700 1 -- 172.21.15.67:0/2722813823 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f7d380b2990 msgr2=0x7f7d380b4e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:19:35.344 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:35.340+0000 7f7d54c33700 1 --2- 172.21.15.67:0/2722813823 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f7d380b2990 0x7f7d380b4e50 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f7d44000c00 tx=0x7f7d44005fb0 comp rx=0 tx=0).stop 2024-05-03T21:19:35.344 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:35.340+0000 7f7d54c33700 1 -- 172.21.15.67:0/2722813823 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7d50102fe0 msgr2=0x7f7d5019b080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:19:35.345 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:35.340+0000 7f7d54c33700 1 --2- 172.21.15.67:0/2722813823 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7d50102fe0 0x7f7d5019b080 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f7d4000e9f0 tx=0x7f7d4000ed00 comp rx=0 tx=0).stop 2024-05-03T21:19:35.345 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:35.340+0000 7f7d54c33700 1 -- 172.21.15.67:0/2722813823 shutdown_connections 2024-05-03T21:19:35.345 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:35.340+0000 7f7d54c33700 1 --2- 172.21.15.67:0/2722813823 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f7d380b2990 0x7f7d380b4e50 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:19:35.345 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:35.340+0000 7f7d54c33700 1 --2- 172.21.15.67:0/2722813823 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f7d501076c0 0x7f7d501a5bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:19:35.345 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:35.340+0000 7f7d54c33700 1 --2- 172.21.15.67:0/2722813823 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f7d50102630 0x7f7d5019ab40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:19:35.345 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:35.340+0000 7f7d54c33700 1 --2- 172.21.15.67:0/2722813823 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7d50102fe0 0x7f7d5019b080 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:19:35.345 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:35.340+0000 7f7d54c33700 1 -- 172.21.15.67:0/2722813823 >> 172.21.15.67:0/2722813823 conn(0x7f7d50079350 msgr2=0x7f7d50101650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T21:19:35.345 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:35.340+0000 7f7d54c33700 1 -- 172.21.15.67:0/2722813823 shutdown_connections 2024-05-03T21:19:35.345 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:35.340+0000 7f7d54c33700 1 -- 172.21.15.67:0/2722813823 wait complete. 2024-05-03T21:19:36.064 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":17,"flags":0,"active_gid":14150,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":604451592},{"type":"v1","addr":"172.21.15.67:6801","nonce":604451592}]},"active_addr":"172.21.15.67:6801/604451592","active_change":"2024-05-03T21:11:33.721527+0000","active_mgr_features":4540701547738038271,"available":true,"standbys":[{"gid":24106,"name":"z","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.1","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"smb","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"update_orchestration":{"name":"update_orchestration","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically update orchestration when smb resources are changed","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]},{"gid":24116,"name":"y","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.1","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"smb","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"update_orchestration":{"name":"update_orchestration","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically update orchestration when smb resources are changed","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["cephadm","dashboard","iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.1","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"smb","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"update_orchestration":{"name":"update_orchestration","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically update orchestration when smb resources are changed","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://172.21.15.67:8443/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":3,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.67:0","nonce":867283717}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.67:0","nonce":1693089356}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.67:0","nonce":714236149}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.67:0","nonce":3954713205}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.67:0","nonce":1255251626}]}]} 2024-05-03T21:19:36.070 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2024-05-03T21:19:36.070 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-05-03T21:19:36.070 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9a047fe7599a549042d06a3fb5b6813a203eabea shell --fsid 713bc57a-0991-11ef-bc97-c7b262605968 -- ceph osd dump --format=json 2024-05-03T21:19:36.245 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:19:35 smithi082 ceph-mon[32933]: osdmap e84: 12 total, 12 up, 12 in 2024-05-03T21:19:36.245 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:19:35 smithi082 ceph-mon[32933]: from='client.? 172.21.15.67:0/2722813823' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-05-03T21:19:36.319 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/713bc57a-0991-11ef-bc97-c7b262605968/mon.a/config 2024-05-03T21:19:36.324 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:19:35 smithi138 ceph-mon[33110]: osdmap e84: 12 total, 12 up, 12 in 2024-05-03T21:19:36.324 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:19:35 smithi138 ceph-mon[33110]: from='client.? 172.21.15.67:0/2722813823' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-05-03T21:19:36.442 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:19:35 smithi067 ceph-mon[29730]: osdmap e84: 12 total, 12 up, 12 in 2024-05-03T21:19:36.442 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:19:35 smithi067 ceph-mon[29730]: from='client.? 172.21.15.67:0/2722813823' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-05-03T21:19:37.245 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:19:36 smithi082 ceph-mon[32933]: pgmap v315: 97 pgs: 1 activating+degraded, 96 active+clean; 632 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 383 B/s wr, 5 op/s; 1/192 objects degraded (0.521%) 2024-05-03T21:19:37.245 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:19:36 smithi082 ceph-mon[32933]: Health check failed: Degraded data redundancy: 1/192 objects degraded (0.521%), 1 pg degraded (PG_DEGRADED) 2024-05-03T21:19:37.324 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:19:36 smithi138 ceph-mon[33110]: pgmap v315: 97 pgs: 1 activating+degraded, 96 active+clean; 632 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 383 B/s wr, 5 op/s; 1/192 objects degraded (0.521%) 2024-05-03T21:19:37.325 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:19:36 smithi138 ceph-mon[33110]: Health check failed: Degraded data redundancy: 1/192 objects degraded (0.521%), 1 pg degraded (PG_DEGRADED) 2024-05-03T21:19:37.471 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:19:36 smithi067 ceph-mon[29730]: pgmap v315: 97 pgs: 1 activating+degraded, 96 active+clean; 632 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 383 B/s wr, 5 op/s; 1/192 objects degraded (0.521%) 2024-05-03T21:19:37.471 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:19:36 smithi067 ceph-mon[29730]: Health check failed: Degraded data redundancy: 1/192 objects degraded (0.521%), 1 pg degraded (PG_DEGRADED) 2024-05-03T21:19:38.094 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:38.091+0000 7fcd3b807700 1 -- 172.21.15.67:0/95771623 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcd34108850 msgr2=0x7fcd341103d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:19:38.094 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:38.091+0000 7fcd3b807700 1 --2- 172.21.15.67:0/95771623 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcd34108850 0x7fcd341103d0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fcd2c00b0d0 tx=0x7fcd2c00b3e0 comp rx=0 tx=0).stop 2024-05-03T21:19:38.094 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:38.091+0000 7fcd3b807700 1 -- 172.21.15.67:0/95771623 shutdown_connections 2024-05-03T21:19:38.094 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:38.091+0000 7fcd3b807700 1 --2- 172.21.15.67:0/95771623 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcd34108850 0x7fcd341103d0 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:19:38.094 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:38.091+0000 7fcd3b807700 1 --2- 172.21.15.67:0/95771623 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fcd34103750 0x7fcd34103b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:19:38.094 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:38.091+0000 7fcd3b807700 1 --2- 172.21.15.67:0/95771623 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fcd34104100 0x7fcd34108120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:19:38.094 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:38.091+0000 7fcd3b807700 1 -- 172.21.15.67:0/95771623 >> 172.21.15.67:0/95771623 conn(0x7fcd3407a600 msgr2=0x7fcd3407aa10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T21:19:38.094 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:38.091+0000 7fcd3b807700 1 -- 172.21.15.67:0/95771623 shutdown_connections 2024-05-03T21:19:38.094 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:38.091+0000 7fcd3b807700 1 -- 172.21.15.67:0/95771623 wait complete. 2024-05-03T21:19:38.094 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:38.092+0000 7fcd3b807700 1 Processor -- start 2024-05-03T21:19:38.094 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:38.092+0000 7fcd3b807700 1 -- start start 2024-05-03T21:19:38.096 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:38.092+0000 7fcd3b807700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcd34103750 0x7fcd3419f820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:19:38.096 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:38.092+0000 7fcd3b807700 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fcd34104100 0x7fcd3419fd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:19:38.096 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:38.092+0000 7fcd3b807700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fcd34108850 0x7fcd34199a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:19:38.096 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:38.092+0000 7fcd3b807700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fcd341138b0 con 0x7fcd34103750 2024-05-03T21:19:38.096 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:38.092+0000 7fcd3b807700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7fcd34113730 con 0x7fcd34108850 2024-05-03T21:19:38.096 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:38.092+0000 7fcd3b807700 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7fcd34113a30 con 0x7fcd34104100 2024-05-03T21:19:38.096 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:38.092+0000 7fcd395a3700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcd34103750 0x7fcd3419f820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:19:38.096 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:38.092+0000 7fcd395a3700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcd34103750 0x7fcd3419f820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:60528/0 (socket says 172.21.15.67:60528) 2024-05-03T21:19:38.097 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:38.092+0000 7fcd395a3700 1 -- 172.21.15.67:0/2324040481 learned_addr learned my addr 172.21.15.67:0/2324040481 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-03T21:19:38.097 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:38.092+0000 7fcd395a3700 1 -- 172.21.15.67:0/2324040481 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fcd34104100 msgr2=0x7fcd3419fd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:19:38.097 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:38.092+0000 7fcd395a3700 1 --2- 172.21.15.67:0/2324040481 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fcd34104100 0x7fcd3419fd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:19:38.097 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:38.092+0000 7fcd39da4700 1 --2- 172.21.15.67:0/2324040481 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fcd34108850 0x7fcd34199a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:19:38.097 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:38.092+0000 7fcd395a3700 1 -- 172.21.15.67:0/2324040481 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fcd34108850 msgr2=0x7fcd34199a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:19:38.097 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:38.092+0000 7fcd395a3700 1 --2- 172.21.15.67:0/2324040481 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fcd34108850 0x7fcd34199a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:19:38.097 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:38.092+0000 7fcd395a3700 1 -- 172.21.15.67:0/2324040481 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcd3419a190 con 0x7fcd34103750 2024-05-03T21:19:38.097 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:38.092+0000 7fcd39da4700 1 --2- 172.21.15.67:0/2324040481 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fcd34108850 0x7fcd34199a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-03T21:19:38.097 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:38.092+0000 7fcd395a3700 1 --2- 172.21.15.67:0/2324040481 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcd34103750 0x7fcd3419f820 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fcd3000c920 tx=0x7fcd3000cce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T21:19:38.097 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:38.093+0000 7fcd2a7fc700 1 -- 172.21.15.67:0/2324040481 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd30007b00 con 0x7fcd34103750 2024-05-03T21:19:38.097 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:38.093+0000 7fcd2a7fc700 1 -- 172.21.15.67:0/2324040481 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(28 keys) ==== 1170+0+0 (secure 0 0 0) 0x7fcd30004d30 con 0x7fcd34103750 2024-05-03T21:19:38.097 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:38.093+0000 7fcd2a7fc700 1 -- 172.21.15.67:0/2324040481 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd3000f450 con 0x7fcd34103750 2024-05-03T21:19:38.097 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:38.093+0000 7fcd3b807700 1 -- 172.21.15.67:0/2324040481 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcd3419a480 con 0x7fcd34103750 2024-05-03T21:19:38.097 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:38.093+0000 7fcd3b807700 1 -- 172.21.15.67:0/2324040481 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcd341a66f0 con 0x7fcd34103750 2024-05-03T21:19:38.098 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:38.094+0000 7fcd3b807700 1 -- 172.21.15.67:0/2324040481 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcd3406a190 con 0x7fcd34103750 2024-05-03T21:19:38.098 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:38.095+0000 7fcd2a7fc700 1 -- 172.21.15.67:0/2324040481 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 17) ==== 149703+0+0 (secure 0 0 0) 0x7fcd30004750 con 0x7fcd34103750 2024-05-03T21:19:38.098 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:38.096+0000 7fcd2a7fc700 1 --2- 172.21.15.67:0/2324040481 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7fcd200b2990 0x7fcd200b4e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:19:38.099 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:38.096+0000 7fcd38da2700 1 --2- 172.21.15.67:0/2324040481 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7fcd200b2990 0x7fcd200b4e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:19:38.099 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:38.096+0000 7fcd2a7fc700 1 -- 172.21.15.67:0/2324040481 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 8211+0+0 (secure 0 0 0) 0x7fcd300e1f50 con 0x7fcd34103750 2024-05-03T21:19:38.099 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:38.096+0000 7fcd38da2700 1 --2- 172.21.15.67:0/2324040481 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7fcd200b2990 0x7fcd200b4e50 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fcd24005fd0 tx=0x7fcd24005f60 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T21:19:38.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:38.101+0000 7fcd2a7fc700 1 -- 172.21.15.67:0/2324040481 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191839 (secure 0 0 0) 0x7fcd30014030 con 0x7fcd34103750 2024-05-03T21:19:38.315 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:38.313+0000 7fcd3b807700 1 -- 172.21.15.67:0/2324040481 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fcd3402e510 con 0x7fcd34103750 2024-05-03T21:19:38.317 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:38.315+0000 7fcd2a7fc700 1 -- 172.21.15.67:0/2324040481 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v84) ==== 74+0+22637 (secure 0 0 0) 0x7fcd300a9b20 con 0x7fcd34103750 2024-05-03T21:19:38.318 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-05-03T21:19:38.318 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":84,"fsid":"713bc57a-0991-11ef-bc97-c7b262605968","created":"2024-05-03T21:10:34.070178+0000","modified":"2024-05-03T21:19:34.967839+0000","last_up_change":"2024-05-03T21:17:39.993919+0000","last_in_change":"2024-05-03T21:17:19.978727+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-03T21:13:59.810446+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-05-03T21:18:37.953919+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"82","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-05-03T21:18:41.571707+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"82","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7939f31c-6fb7-42bd-af96-88a448d11bd2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":3692483807},{"type":"v1","addr":"172.21.15.67:6803","nonce":3692483807}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":3692483807},{"type":"v1","addr":"172.21.15.67:6805","nonce":3692483807}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":3692483807},{"type":"v1","addr":"172.21.15.67:6809","nonce":3692483807}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":3692483807},{"type":"v1","addr":"172.21.15.67:6807","nonce":3692483807}]},"public_addr":"172.21.15.67:6803/3692483807","cluster_addr":"172.21.15.67:6805/3692483807","heartbeat_back_addr":"172.21.15.67:6809/3692483807","heartbeat_front_addr":"172.21.15.67:6807/3692483807","state":["exists","up"]},{"osd":1,"uuid":"71858193-05a6-4c2c-9bb7-bc3d811d634e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6800","nonce":1139376397},{"type":"v1","addr":"172.21.15.82:6801","nonce":1139376397}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6802","nonce":1139376397},{"type":"v1","addr":"172.21.15.82:6803","nonce":1139376397}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6806","nonce":1139376397},{"type":"v1","addr":"172.21.15.82:6807","nonce":1139376397}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6804","nonce":1139376397},{"type":"v1","addr":"172.21.15.82:6805","nonce":1139376397}]},"public_addr":"172.21.15.82:6801/1139376397","cluster_addr":"172.21.15.82:6803/1139376397","heartbeat_back_addr":"172.21.15.82:6807/1139376397","heartbeat_front_addr":"172.21.15.82:6805/1139376397","state":["exists","up"]},{"osd":2,"uuid":"f9ece78d-6a19-4d46-9022-6ed184074745","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6800","nonce":2774330123},{"type":"v1","addr":"172.21.15.138:6801","nonce":2774330123}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6802","nonce":2774330123},{"type":"v1","addr":"172.21.15.138:6803","nonce":2774330123}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6806","nonce":2774330123},{"type":"v1","addr":"172.21.15.138:6807","nonce":2774330123}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6804","nonce":2774330123},{"type":"v1","addr":"172.21.15.138:6805","nonce":2774330123}]},"public_addr":"172.21.15.138:6801/2774330123","cluster_addr":"172.21.15.138:6803/2774330123","heartbeat_back_addr":"172.21.15.138:6807/2774330123","heartbeat_front_addr":"172.21.15.138:6805/2774330123","state":["exists","up"]},{"osd":3,"uuid":"3616df71-e7b3-4687-aaba-37b0b7766a3c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":27,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":3915934652},{"type":"v1","addr":"172.21.15.67:6811","nonce":3915934652}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":3915934652},{"type":"v1","addr":"172.21.15.67:6813","nonce":3915934652}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":3915934652},{"type":"v1","addr":"172.21.15.67:6817","nonce":3915934652}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":3915934652},{"type":"v1","addr":"172.21.15.67:6815","nonce":3915934652}]},"public_addr":"172.21.15.67:6811/3915934652","cluster_addr":"172.21.15.67:6813/3915934652","heartbeat_back_addr":"172.21.15.67:6817/3915934652","heartbeat_front_addr":"172.21.15.67:6815/3915934652","state":["exists","up"]},{"osd":4,"uuid":"c3fd0ffd-e23f-4279-b679-9d95a144cf32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6808","nonce":4031818951},{"type":"v1","addr":"172.21.15.82:6809","nonce":4031818951}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6810","nonce":4031818951},{"type":"v1","addr":"172.21.15.82:6811","nonce":4031818951}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6814","nonce":4031818951},{"type":"v1","addr":"172.21.15.82:6815","nonce":4031818951}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6812","nonce":4031818951},{"type":"v1","addr":"172.21.15.82:6813","nonce":4031818951}]},"public_addr":"172.21.15.82:6809/4031818951","cluster_addr":"172.21.15.82:6811/4031818951","heartbeat_back_addr":"172.21.15.82:6815/4031818951","heartbeat_front_addr":"172.21.15.82:6813/4031818951","state":["exists","up"]},{"osd":5,"uuid":"540ef0d9-a6dc-43c6-b09e-e87340ba24cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6808","nonce":2521889499},{"type":"v1","addr":"172.21.15.138:6809","nonce":2521889499}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6810","nonce":2521889499},{"type":"v1","addr":"172.21.15.138:6811","nonce":2521889499}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6814","nonce":2521889499},{"type":"v1","addr":"172.21.15.138:6815","nonce":2521889499}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6812","nonce":2521889499},{"type":"v1","addr":"172.21.15.138:6813","nonce":2521889499}]},"public_addr":"172.21.15.138:6809/2521889499","cluster_addr":"172.21.15.138:6811/2521889499","heartbeat_back_addr":"172.21.15.138:6815/2521889499","heartbeat_front_addr":"172.21.15.138:6813/2521889499","state":["exists","up"]},{"osd":6,"uuid":"49a89fd0-958a-4be9-a8db-6022f946153e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6818","nonce":3269682731},{"type":"v1","addr":"172.21.15.67:6819","nonce":3269682731}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":3269682731},{"type":"v1","addr":"172.21.15.67:6821","nonce":3269682731}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":3269682731},{"type":"v1","addr":"172.21.15.67:6825","nonce":3269682731}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":3269682731},{"type":"v1","addr":"172.21.15.67:6823","nonce":3269682731}]},"public_addr":"172.21.15.67:6819/3269682731","cluster_addr":"172.21.15.67:6821/3269682731","heartbeat_back_addr":"172.21.15.67:6825/3269682731","heartbeat_front_addr":"172.21.15.67:6823/3269682731","state":["exists","up"]},{"osd":7,"uuid":"8fa4cd64-ca6b-48ae-8db0-d369ce6102ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":50,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6816","nonce":3008589976},{"type":"v1","addr":"172.21.15.82:6817","nonce":3008589976}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6818","nonce":3008589976},{"type":"v1","addr":"172.21.15.82:6819","nonce":3008589976}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6822","nonce":3008589976},{"type":"v1","addr":"172.21.15.82:6823","nonce":3008589976}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6820","nonce":3008589976},{"type":"v1","addr":"172.21.15.82:6821","nonce":3008589976}]},"public_addr":"172.21.15.82:6817/3008589976","cluster_addr":"172.21.15.82:6819/3008589976","heartbeat_back_addr":"172.21.15.82:6823/3008589976","heartbeat_front_addr":"172.21.15.82:6821/3008589976","state":["exists","up"]},{"osd":8,"uuid":"fae39087-6c38-4771-a2b9-3f4fecb08edd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":57,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6816","nonce":2223920070},{"type":"v1","addr":"172.21.15.138:6817","nonce":2223920070}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6818","nonce":2223920070},{"type":"v1","addr":"172.21.15.138:6819","nonce":2223920070}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6822","nonce":2223920070},{"type":"v1","addr":"172.21.15.138:6823","nonce":2223920070}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6820","nonce":2223920070},{"type":"v1","addr":"172.21.15.138:6821","nonce":2223920070}]},"public_addr":"172.21.15.138:6817/2223920070","cluster_addr":"172.21.15.138:6819/2223920070","heartbeat_back_addr":"172.21.15.138:6823/2223920070","heartbeat_front_addr":"172.21.15.138:6821/2223920070","state":["exists","up"]},{"osd":9,"uuid":"3d387dbf-0163-41b7-98af-4fc45ed73ad4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":63,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":3939417129},{"type":"v1","addr":"172.21.15.67:6827","nonce":3939417129}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":3939417129},{"type":"v1","addr":"172.21.15.67:6829","nonce":3939417129}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":3939417129},{"type":"v1","addr":"172.21.15.67:6833","nonce":3939417129}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":3939417129},{"type":"v1","addr":"172.21.15.67:6831","nonce":3939417129}]},"public_addr":"172.21.15.67:6827/3939417129","cluster_addr":"172.21.15.67:6829/3939417129","heartbeat_back_addr":"172.21.15.67:6833/3939417129","heartbeat_front_addr":"172.21.15.67:6831/3939417129","state":["exists","up"]},{"osd":10,"uuid":"db0305a0-a83b-474b-837a-fbc239ac1738","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6824","nonce":2505507029},{"type":"v1","addr":"172.21.15.82:6825","nonce":2505507029}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6826","nonce":2505507029},{"type":"v1","addr":"172.21.15.82:6827","nonce":2505507029}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6830","nonce":2505507029},{"type":"v1","addr":"172.21.15.82:6831","nonce":2505507029}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6828","nonce":2505507029},{"type":"v1","addr":"172.21.15.82:6829","nonce":2505507029}]},"public_addr":"172.21.15.82:6825/2505507029","cluster_addr":"172.21.15.82:6827/2505507029","heartbeat_back_addr":"172.21.15.82:6831/2505507029","heartbeat_front_addr":"172.21.15.82:6829/2505507029","state":["exists","up"]},{"osd":11,"uuid":"479c2747-f28e-4704-87e4-d0c31782e2de","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":74,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6824","nonce":2528685018},{"type":"v1","addr":"172.21.15.138:6825","nonce":2528685018}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6826","nonce":2528685018},{"type":"v1","addr":"172.21.15.138:6827","nonce":2528685018}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6830","nonce":2528685018},{"type":"v1","addr":"172.21.15.138:6831","nonce":2528685018}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6828","nonce":2528685018},{"type":"v1","addr":"172.21.15.138:6829","nonce":2528685018}]},"public_addr":"172.21.15.138:6825/2528685018","cluster_addr":"172.21.15.138:6827/2528685018","heartbeat_back_addr":"172.21.15.138:6831/2528685018","heartbeat_front_addr":"172.21.15.138:6829/2528685018","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-03T21:13:07.768773+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-03T21:13:31.870232+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-03T21:13:55.394206+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-03T21:14:20.540281+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-03T21:14:44.318523+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-03T21:15:08.490829+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-03T21:15:34.144734+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-03T21:15:58.717240+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-03T21:16:22.743013+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-03T21:16:49.099588+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-03T21:17:13.321126+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-03T21:17:38.001978+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.0","mappings":[{"from":8,"to":4}]},{"pgid":"2.a","mappings":[{"from":2,"to":11}]},{"pgid":"2.b","mappings":[{"from":2,"to":4}]},{"pgid":"2.10","mappings":[{"from":8,"to":11}]},{"pgid":"3.1","mappings":[{"from":4,"to":9}]},{"pgid":"3.1b","mappings":[{"from":5,"to":9}]},{"pgid":"3.21","mappings":[{"from":4,"to":8}]},{"pgid":"3.3c","mappings":[{"from":4,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.67:0/1239643865":"2024-05-04T21:11:33.721416+0000","172.21.15.67:6801/2576113038":"2024-05-04T21:11:33.721416+0000","172.21.15.67:6800/2576113038":"2024-05-04T21:11:33.721416+0000","172.21.15.67:6800/1698171509":"2024-05-04T21:11:07.333651+0000","172.21.15.67:0/2217893923":"2024-05-04T21:11:33.721416+0000","172.21.15.67:0/20864811":"2024-05-04T21:11:07.333651+0000","172.21.15.67:6801/1698171509":"2024-05-04T21:11:07.333651+0000","172.21.15.67:0/773376553":"2024-05-04T21:11:33.721416+0000","172.21.15.67:0/847549114":"2024-05-04T21:11:07.333651+0000","172.21.15.67:0/1291778035":"2024-05-04T21:11:07.333651+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-03T21:19:38.320 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:38.317+0000 7fcd3b807700 1 -- 172.21.15.67:0/2324040481 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7fcd200b2990 msgr2=0x7fcd200b4e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:19:38.320 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:38.317+0000 7fcd3b807700 1 --2- 172.21.15.67:0/2324040481 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7fcd200b2990 0x7fcd200b4e50 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fcd24005fd0 tx=0x7fcd24005f60 comp rx=0 tx=0).stop 2024-05-03T21:19:38.320 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:38.317+0000 7fcd3b807700 1 -- 172.21.15.67:0/2324040481 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcd34103750 msgr2=0x7fcd3419f820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:19:38.320 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:38.317+0000 7fcd3b807700 1 --2- 172.21.15.67:0/2324040481 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcd34103750 0x7fcd3419f820 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fcd3000c920 tx=0x7fcd3000cce0 comp rx=0 tx=0).stop 2024-05-03T21:19:38.320 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:38.317+0000 7fcd3b807700 1 -- 172.21.15.67:0/2324040481 shutdown_connections 2024-05-03T21:19:38.320 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:38.317+0000 7fcd3b807700 1 --2- 172.21.15.67:0/2324040481 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7fcd200b2990 0x7fcd200b4e50 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:19:38.320 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:38.317+0000 7fcd3b807700 1 --2- 172.21.15.67:0/2324040481 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcd34103750 0x7fcd3419f820 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:19:38.320 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:38.317+0000 7fcd3b807700 1 --2- 172.21.15.67:0/2324040481 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fcd34108850 0x7fcd34199a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:19:38.320 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:38.317+0000 7fcd3b807700 1 --2- 172.21.15.67:0/2324040481 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fcd34104100 0x7fcd3419fd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:19:38.320 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:38.317+0000 7fcd3b807700 1 -- 172.21.15.67:0/2324040481 >> 172.21.15.67:0/2324040481 conn(0x7fcd3407a600 msgr2=0x7fcd34102690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T21:19:38.320 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:38.317+0000 7fcd3b807700 1 -- 172.21.15.67:0/2324040481 shutdown_connections 2024-05-03T21:19:38.320 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:38.318+0000 7fcd3b807700 1 -- 172.21.15.67:0/2324040481 wait complete. 2024-05-03T21:19:38.941 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-05-03T21:19:38.941 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9a047fe7599a549042d06a3fb5b6813a203eabea shell --fsid 713bc57a-0991-11ef-bc97-c7b262605968 -- ceph osd dump --format=json 2024-05-03T21:19:39.195 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/713bc57a-0991-11ef-bc97-c7b262605968/mon.a/config 2024-05-03T21:19:39.221 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:19:38 smithi067 ceph-mon[29730]: pgmap v316: 97 pgs: 1 activating+degraded, 96 active+clean; 632 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 383 B/s wr, 7 op/s; 1/192 objects degraded (0.521%); 13 B/s, 1 objects/s recovering 2024-05-03T21:19:39.221 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:19:38 smithi067 ceph-mon[29730]: from='client.? 172.21.15.67:0/2324040481' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-03T21:19:39.245 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:19:38 smithi082 ceph-mon[32933]: pgmap v316: 97 pgs: 1 activating+degraded, 96 active+clean; 632 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 383 B/s wr, 7 op/s; 1/192 objects degraded (0.521%); 13 B/s, 1 objects/s recovering 2024-05-03T21:19:39.245 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:19:38 smithi082 ceph-mon[32933]: from='client.? 172.21.15.67:0/2324040481' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-03T21:19:39.324 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:19:38 smithi138 ceph-mon[33110]: pgmap v316: 97 pgs: 1 activating+degraded, 96 active+clean; 632 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 383 B/s wr, 7 op/s; 1/192 objects degraded (0.521%); 13 B/s, 1 objects/s recovering 2024-05-03T21:19:39.324 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:19:38 smithi138 ceph-mon[33110]: from='client.? 172.21.15.67:0/2324040481' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-03T21:19:40.245 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:19:39 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T21:19:40.324 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:19:39 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T21:19:40.357 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:19:39 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T21:19:40.959 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:40.954+0000 7f03fa543700 1 -- 172.21.15.67:0/235913496 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f03f406c630 msgr2=0x7f03f4110360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:19:40.959 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:40.954+0000 7f03fa543700 1 --2- 172.21.15.67:0/235913496 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f03f406c630 0x7f03f4110360 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f03ec00b0d0 tx=0x7f03ec00b3e0 comp rx=0 tx=0).stop 2024-05-03T21:19:40.959 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:40.954+0000 7f03fa543700 1 -- 172.21.15.67:0/235913496 shutdown_connections 2024-05-03T21:19:40.959 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:40.954+0000 7f03fa543700 1 --2- 172.21.15.67:0/235913496 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f03f406c630 0x7f03f4110360 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:19:40.959 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:40.954+0000 7f03fa543700 1 --2- 172.21.15.67:0/235913496 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f03f4109590 0x7f03f4109970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:19:40.960 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:40.954+0000 7f03fa543700 1 --2- 172.21.15.67:0/235913496 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f03f406bc70 0x7f03f406c0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:19:40.960 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:40.954+0000 7f03fa543700 1 -- 172.21.15.67:0/235913496 >> 172.21.15.67:0/235913496 conn(0x7f03f4079440 msgr2=0x7f03f4079850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T21:19:40.960 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:40.954+0000 7f03fa543700 1 -- 172.21.15.67:0/235913496 shutdown_connections 2024-05-03T21:19:40.960 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:40.954+0000 7f03fa543700 1 -- 172.21.15.67:0/235913496 wait complete. 2024-05-03T21:19:40.960 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:40.954+0000 7f03fa543700 1 Processor -- start 2024-05-03T21:19:40.960 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:40.954+0000 7f03fa543700 1 -- start start 2024-05-03T21:19:40.960 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:40.955+0000 7f03fa543700 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f03f406bc70 0x7f03f4102410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:19:40.960 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:40.955+0000 7f03fa543700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f03f406c630 0x7f03f4102950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:19:40.960 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:40.955+0000 7f03fa543700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f03f4109590 0x7f03f4102e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:19:40.961 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:40.955+0000 7f03fa543700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f03f4113910 con 0x7f03f406c630 2024-05-03T21:19:40.961 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:40.955+0000 7f03fa543700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f03f4113790 con 0x7f03f4109590 2024-05-03T21:19:40.961 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:40.955+0000 7f03fa543700 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f03f4113a90 con 0x7f03f406bc70 2024-05-03T21:19:40.961 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:40.955+0000 7f03f37fe700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f03f406c630 0x7f03f4102950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:19:40.961 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:40.955+0000 7f03f37fe700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f03f406c630 0x7f03f4102950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:60544/0 (socket says 172.21.15.67:60544) 2024-05-03T21:19:40.961 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:40.955+0000 7f03f37fe700 1 -- 172.21.15.67:0/810052473 learned_addr learned my addr 172.21.15.67:0/810052473 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-03T21:19:40.961 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:40.955+0000 7f03f8ae0700 1 --2- 172.21.15.67:0/810052473 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f03f4109590 0x7f03f4102e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:19:40.961 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:40.955+0000 7f03f3fff700 1 --2- 172.21.15.67:0/810052473 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f03f406bc70 0x7f03f4102410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:19:40.961 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:40.955+0000 7f03f37fe700 1 -- 172.21.15.67:0/810052473 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f03f406bc70 msgr2=0x7f03f4102410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:19:40.962 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:40.955+0000 7f03f37fe700 1 --2- 172.21.15.67:0/810052473 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f03f406bc70 0x7f03f4102410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:19:40.962 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:40.955+0000 7f03f37fe700 1 -- 172.21.15.67:0/810052473 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f03f4109590 msgr2=0x7f03f4102e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:19:40.962 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:40.955+0000 7f03f37fe700 1 --2- 172.21.15.67:0/810052473 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f03f4109590 0x7f03f4102e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:19:40.962 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:40.955+0000 7f03f37fe700 1 -- 172.21.15.67:0/810052473 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f03f41a8170 con 0x7f03f406c630 2024-05-03T21:19:40.962 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:40.955+0000 7f03f8ae0700 1 --2- 172.21.15.67:0/810052473 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f03f4109590 0x7f03f4102e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-03T21:19:40.962 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:40.955+0000 7f03f37fe700 1 --2- 172.21.15.67:0/810052473 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f03f406c630 0x7f03f4102950 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f03e800d980 tx=0x7f03e800dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T21:19:40.962 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:40.955+0000 7f03f17fa700 1 -- 172.21.15.67:0/810052473 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03e8016070 con 0x7f03f406c630 2024-05-03T21:19:40.962 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:40.956+0000 7f03f17fa700 1 -- 172.21.15.67:0/810052473 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(28 keys) ==== 1170+0+0 (secure 0 0 0) 0x7f03e8004d30 con 0x7f03f406c630 2024-05-03T21:19:40.963 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:40.956+0000 7f03fa543700 1 -- 172.21.15.67:0/810052473 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f03f41a8460 con 0x7f03f406c630 2024-05-03T21:19:40.963 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:40.956+0000 7f03f17fa700 1 -- 172.21.15.67:0/810052473 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03e80056c0 con 0x7f03f406c630 2024-05-03T21:19:40.963 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:40.956+0000 7f03fa543700 1 -- 172.21.15.67:0/810052473 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f03f41a89a0 con 0x7f03f406c630 2024-05-03T21:19:40.963 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:40.957+0000 7f03fa543700 1 -- 172.21.15.67:0/810052473 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f03f406a190 con 0x7f03f406c630 2024-05-03T21:19:40.964 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:40.958+0000 7f03f17fa700 1 -- 172.21.15.67:0/810052473 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 17) ==== 149703+0+0 (secure 0 0 0) 0x7f03e8004750 con 0x7f03f406c630 2024-05-03T21:19:40.964 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:40.959+0000 7f03f17fa700 1 --2- 172.21.15.67:0/810052473 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f03dc0b2910 0x7f03dc0b4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:19:40.964 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:40.959+0000 7f03f3fff700 1 --2- 172.21.15.67:0/810052473 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f03dc0b2910 0x7f03dc0b4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:19:40.965 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:40.959+0000 7f03f17fa700 1 -- 172.21.15.67:0/810052473 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 8211+0+0 (secure 0 0 0) 0x7f03e80e25f0 con 0x7f03f406c630 2024-05-03T21:19:40.965 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:40.959+0000 7f03f3fff700 1 --2- 172.21.15.67:0/810052473 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f03dc0b2910 0x7f03dc0b4dd0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f03e4007b60 tx=0x7f03e40073d0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T21:19:40.965 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:40.962+0000 7f03f17fa700 1 -- 172.21.15.67:0/810052473 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191839 (secure 0 0 0) 0x7f03e801f020 con 0x7f03f406c630 2024-05-03T21:19:41.194 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:41.192+0000 7f03fa543700 1 -- 172.21.15.67:0/810052473 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f03f41a8cc0 con 0x7f03f406c630 2024-05-03T21:19:41.196 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:41.194+0000 7f03f17fa700 1 -- 172.21.15.67:0/810052473 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v84) ==== 74+0+22637 (secure 0 0 0) 0x7f03e80aa1c0 con 0x7f03f406c630 2024-05-03T21:19:41.196 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:19:40 smithi067 ceph-mon[29730]: pgmap v317: 97 pgs: 1 activating+degraded, 96 active+clean; 632 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 5 op/s; 1/192 objects degraded (0.521%); 13 B/s, 1 objects/s recovering 2024-05-03T21:19:41.197 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-05-03T21:19:41.197 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":84,"fsid":"713bc57a-0991-11ef-bc97-c7b262605968","created":"2024-05-03T21:10:34.070178+0000","modified":"2024-05-03T21:19:34.967839+0000","last_up_change":"2024-05-03T21:17:39.993919+0000","last_in_change":"2024-05-03T21:17:19.978727+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-03T21:13:59.810446+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-05-03T21:18:37.953919+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"82","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-05-03T21:18:41.571707+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"82","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7939f31c-6fb7-42bd-af96-88a448d11bd2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":3692483807},{"type":"v1","addr":"172.21.15.67:6803","nonce":3692483807}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":3692483807},{"type":"v1","addr":"172.21.15.67:6805","nonce":3692483807}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":3692483807},{"type":"v1","addr":"172.21.15.67:6809","nonce":3692483807}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":3692483807},{"type":"v1","addr":"172.21.15.67:6807","nonce":3692483807}]},"public_addr":"172.21.15.67:6803/3692483807","cluster_addr":"172.21.15.67:6805/3692483807","heartbeat_back_addr":"172.21.15.67:6809/3692483807","heartbeat_front_addr":"172.21.15.67:6807/3692483807","state":["exists","up"]},{"osd":1,"uuid":"71858193-05a6-4c2c-9bb7-bc3d811d634e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6800","nonce":1139376397},{"type":"v1","addr":"172.21.15.82:6801","nonce":1139376397}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6802","nonce":1139376397},{"type":"v1","addr":"172.21.15.82:6803","nonce":1139376397}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6806","nonce":1139376397},{"type":"v1","addr":"172.21.15.82:6807","nonce":1139376397}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6804","nonce":1139376397},{"type":"v1","addr":"172.21.15.82:6805","nonce":1139376397}]},"public_addr":"172.21.15.82:6801/1139376397","cluster_addr":"172.21.15.82:6803/1139376397","heartbeat_back_addr":"172.21.15.82:6807/1139376397","heartbeat_front_addr":"172.21.15.82:6805/1139376397","state":["exists","up"]},{"osd":2,"uuid":"f9ece78d-6a19-4d46-9022-6ed184074745","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6800","nonce":2774330123},{"type":"v1","addr":"172.21.15.138:6801","nonce":2774330123}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6802","nonce":2774330123},{"type":"v1","addr":"172.21.15.138:6803","nonce":2774330123}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6806","nonce":2774330123},{"type":"v1","addr":"172.21.15.138:6807","nonce":2774330123}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6804","nonce":2774330123},{"type":"v1","addr":"172.21.15.138:6805","nonce":2774330123}]},"public_addr":"172.21.15.138:6801/2774330123","cluster_addr":"172.21.15.138:6803/2774330123","heartbeat_back_addr":"172.21.15.138:6807/2774330123","heartbeat_front_addr":"172.21.15.138:6805/2774330123","state":["exists","up"]},{"osd":3,"uuid":"3616df71-e7b3-4687-aaba-37b0b7766a3c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":27,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":3915934652},{"type":"v1","addr":"172.21.15.67:6811","nonce":3915934652}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":3915934652},{"type":"v1","addr":"172.21.15.67:6813","nonce":3915934652}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":3915934652},{"type":"v1","addr":"172.21.15.67:6817","nonce":3915934652}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":3915934652},{"type":"v1","addr":"172.21.15.67:6815","nonce":3915934652}]},"public_addr":"172.21.15.67:6811/3915934652","cluster_addr":"172.21.15.67:6813/3915934652","heartbeat_back_addr":"172.21.15.67:6817/3915934652","heartbeat_front_addr":"172.21.15.67:6815/3915934652","state":["exists","up"]},{"osd":4,"uuid":"c3fd0ffd-e23f-4279-b679-9d95a144cf32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6808","nonce":4031818951},{"type":"v1","addr":"172.21.15.82:6809","nonce":4031818951}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6810","nonce":4031818951},{"type":"v1","addr":"172.21.15.82:6811","nonce":4031818951}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6814","nonce":4031818951},{"type":"v1","addr":"172.21.15.82:6815","nonce":4031818951}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6812","nonce":4031818951},{"type":"v1","addr":"172.21.15.82:6813","nonce":4031818951}]},"public_addr":"172.21.15.82:6809/4031818951","cluster_addr":"172.21.15.82:6811/4031818951","heartbeat_back_addr":"172.21.15.82:6815/4031818951","heartbeat_front_addr":"172.21.15.82:6813/4031818951","state":["exists","up"]},{"osd":5,"uuid":"540ef0d9-a6dc-43c6-b09e-e87340ba24cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6808","nonce":2521889499},{"type":"v1","addr":"172.21.15.138:6809","nonce":2521889499}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6810","nonce":2521889499},{"type":"v1","addr":"172.21.15.138:6811","nonce":2521889499}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6814","nonce":2521889499},{"type":"v1","addr":"172.21.15.138:6815","nonce":2521889499}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6812","nonce":2521889499},{"type":"v1","addr":"172.21.15.138:6813","nonce":2521889499}]},"public_addr":"172.21.15.138:6809/2521889499","cluster_addr":"172.21.15.138:6811/2521889499","heartbeat_back_addr":"172.21.15.138:6815/2521889499","heartbeat_front_addr":"172.21.15.138:6813/2521889499","state":["exists","up"]},{"osd":6,"uuid":"49a89fd0-958a-4be9-a8db-6022f946153e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6818","nonce":3269682731},{"type":"v1","addr":"172.21.15.67:6819","nonce":3269682731}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":3269682731},{"type":"v1","addr":"172.21.15.67:6821","nonce":3269682731}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":3269682731},{"type":"v1","addr":"172.21.15.67:6825","nonce":3269682731}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":3269682731},{"type":"v1","addr":"172.21.15.67:6823","nonce":3269682731}]},"public_addr":"172.21.15.67:6819/3269682731","cluster_addr":"172.21.15.67:6821/3269682731","heartbeat_back_addr":"172.21.15.67:6825/3269682731","heartbeat_front_addr":"172.21.15.67:6823/3269682731","state":["exists","up"]},{"osd":7,"uuid":"8fa4cd64-ca6b-48ae-8db0-d369ce6102ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":50,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6816","nonce":3008589976},{"type":"v1","addr":"172.21.15.82:6817","nonce":3008589976}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6818","nonce":3008589976},{"type":"v1","addr":"172.21.15.82:6819","nonce":3008589976}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6822","nonce":3008589976},{"type":"v1","addr":"172.21.15.82:6823","nonce":3008589976}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6820","nonce":3008589976},{"type":"v1","addr":"172.21.15.82:6821","nonce":3008589976}]},"public_addr":"172.21.15.82:6817/3008589976","cluster_addr":"172.21.15.82:6819/3008589976","heartbeat_back_addr":"172.21.15.82:6823/3008589976","heartbeat_front_addr":"172.21.15.82:6821/3008589976","state":["exists","up"]},{"osd":8,"uuid":"fae39087-6c38-4771-a2b9-3f4fecb08edd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":57,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6816","nonce":2223920070},{"type":"v1","addr":"172.21.15.138:6817","nonce":2223920070}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6818","nonce":2223920070},{"type":"v1","addr":"172.21.15.138:6819","nonce":2223920070}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6822","nonce":2223920070},{"type":"v1","addr":"172.21.15.138:6823","nonce":2223920070}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6820","nonce":2223920070},{"type":"v1","addr":"172.21.15.138:6821","nonce":2223920070}]},"public_addr":"172.21.15.138:6817/2223920070","cluster_addr":"172.21.15.138:6819/2223920070","heartbeat_back_addr":"172.21.15.138:6823/2223920070","heartbeat_front_addr":"172.21.15.138:6821/2223920070","state":["exists","up"]},{"osd":9,"uuid":"3d387dbf-0163-41b7-98af-4fc45ed73ad4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":63,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":3939417129},{"type":"v1","addr":"172.21.15.67:6827","nonce":3939417129}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":3939417129},{"type":"v1","addr":"172.21.15.67:6829","nonce":3939417129}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":3939417129},{"type":"v1","addr":"172.21.15.67:6833","nonce":3939417129}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":3939417129},{"type":"v1","addr":"172.21.15.67:6831","nonce":3939417129}]},"public_addr":"172.21.15.67:6827/3939417129","cluster_addr":"172.21.15.67:6829/3939417129","heartbeat_back_addr":"172.21.15.67:6833/3939417129","heartbeat_front_addr":"172.21.15.67:6831/3939417129","state":["exists","up"]},{"osd":10,"uuid":"db0305a0-a83b-474b-837a-fbc239ac1738","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6824","nonce":2505507029},{"type":"v1","addr":"172.21.15.82:6825","nonce":2505507029}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6826","nonce":2505507029},{"type":"v1","addr":"172.21.15.82:6827","nonce":2505507029}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6830","nonce":2505507029},{"type":"v1","addr":"172.21.15.82:6831","nonce":2505507029}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6828","nonce":2505507029},{"type":"v1","addr":"172.21.15.82:6829","nonce":2505507029}]},"public_addr":"172.21.15.82:6825/2505507029","cluster_addr":"172.21.15.82:6827/2505507029","heartbeat_back_addr":"172.21.15.82:6831/2505507029","heartbeat_front_addr":"172.21.15.82:6829/2505507029","state":["exists","up"]},{"osd":11,"uuid":"479c2747-f28e-4704-87e4-d0c31782e2de","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":74,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6824","nonce":2528685018},{"type":"v1","addr":"172.21.15.138:6825","nonce":2528685018}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6826","nonce":2528685018},{"type":"v1","addr":"172.21.15.138:6827","nonce":2528685018}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6830","nonce":2528685018},{"type":"v1","addr":"172.21.15.138:6831","nonce":2528685018}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6828","nonce":2528685018},{"type":"v1","addr":"172.21.15.138:6829","nonce":2528685018}]},"public_addr":"172.21.15.138:6825/2528685018","cluster_addr":"172.21.15.138:6827/2528685018","heartbeat_back_addr":"172.21.15.138:6831/2528685018","heartbeat_front_addr":"172.21.15.138:6829/2528685018","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-03T21:13:07.768773+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-03T21:13:31.870232+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-03T21:13:55.394206+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-03T21:14:20.540281+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-03T21:14:44.318523+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-03T21:15:08.490829+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-03T21:15:34.144734+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-03T21:15:58.717240+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-03T21:16:22.743013+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-03T21:16:49.099588+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-03T21:17:13.321126+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-03T21:17:38.001978+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.0","mappings":[{"from":8,"to":4}]},{"pgid":"2.a","mappings":[{"from":2,"to":11}]},{"pgid":"2.b","mappings":[{"from":2,"to":4}]},{"pgid":"2.10","mappings":[{"from":8,"to":11}]},{"pgid":"3.1","mappings":[{"from":4,"to":9}]},{"pgid":"3.1b","mappings":[{"from":5,"to":9}]},{"pgid":"3.21","mappings":[{"from":4,"to":8}]},{"pgid":"3.3c","mappings":[{"from":4,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.67:0/1239643865":"2024-05-04T21:11:33.721416+0000","172.21.15.67:6801/2576113038":"2024-05-04T21:11:33.721416+0000","172.21.15.67:6800/2576113038":"2024-05-04T21:11:33.721416+0000","172.21.15.67:6800/1698171509":"2024-05-04T21:11:07.333651+0000","172.21.15.67:0/2217893923":"2024-05-04T21:11:33.721416+0000","172.21.15.67:0/20864811":"2024-05-04T21:11:07.333651+0000","172.21.15.67:6801/1698171509":"2024-05-04T21:11:07.333651+0000","172.21.15.67:0/773376553":"2024-05-04T21:11:33.721416+0000","172.21.15.67:0/847549114":"2024-05-04T21:11:07.333651+0000","172.21.15.67:0/1291778035":"2024-05-04T21:11:07.333651+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-03T21:19:41.198 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:41.196+0000 7f03fa543700 1 -- 172.21.15.67:0/810052473 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f03dc0b2910 msgr2=0x7f03dc0b4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:19:41.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:41.196+0000 7f03fa543700 1 --2- 172.21.15.67:0/810052473 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f03dc0b2910 0x7f03dc0b4dd0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f03e4007b60 tx=0x7f03e40073d0 comp rx=0 tx=0).stop 2024-05-03T21:19:41.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:41.196+0000 7f03fa543700 1 -- 172.21.15.67:0/810052473 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f03f406c630 msgr2=0x7f03f4102950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:19:41.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:41.196+0000 7f03fa543700 1 --2- 172.21.15.67:0/810052473 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f03f406c630 0x7f03f4102950 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f03e800d980 tx=0x7f03e800dc90 comp rx=0 tx=0).stop 2024-05-03T21:19:41.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:41.196+0000 7f03fa543700 1 -- 172.21.15.67:0/810052473 shutdown_connections 2024-05-03T21:19:41.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:41.196+0000 7f03fa543700 1 --2- 172.21.15.67:0/810052473 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f03dc0b2910 0x7f03dc0b4dd0 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:19:41.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:41.196+0000 7f03fa543700 1 --2- 172.21.15.67:0/810052473 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f03f4109590 0x7f03f4102e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:19:41.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:41.196+0000 7f03fa543700 1 --2- 172.21.15.67:0/810052473 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f03f406bc70 0x7f03f4102410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:19:41.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:41.196+0000 7f03fa543700 1 --2- 172.21.15.67:0/810052473 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f03f406c630 0x7f03f4102950 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:19:41.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:41.196+0000 7f03fa543700 1 -- 172.21.15.67:0/810052473 >> 172.21.15.67:0/810052473 conn(0x7f03f4079440 msgr2=0x7f03f4100d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T21:19:41.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:41.197+0000 7f03fa543700 1 -- 172.21.15.67:0/810052473 shutdown_connections 2024-05-03T21:19:41.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:41.197+0000 7f03fa543700 1 -- 172.21.15.67:0/810052473 wait complete. 2024-05-03T21:19:41.245 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:19:40 smithi082 ceph-mon[32933]: pgmap v317: 97 pgs: 1 activating+degraded, 96 active+clean; 632 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 5 op/s; 1/192 objects degraded (0.521%); 13 B/s, 1 objects/s recovering 2024-05-03T21:19:41.324 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:19:40 smithi138 ceph-mon[33110]: pgmap v317: 97 pgs: 1 activating+degraded, 96 active+clean; 632 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 5 op/s; 1/192 objects degraded (0.521%); 13 B/s, 1 objects/s recovering 2024-05-03T21:19:42.221 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:19:41 smithi067 ceph-mon[29730]: from='client.? 172.21.15.67:0/810052473' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-03T21:19:42.221 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:19:41 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:19:42.222 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:19:41 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:19:42.222 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:19:41 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:19:42.222 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:19:41 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:19:42.245 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:19:41 smithi082 ceph-mon[32933]: from='client.? 172.21.15.67:0/810052473' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-03T21:19:42.245 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:19:41 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:19:42.245 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:19:41 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:19:42.246 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:19:41 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:19:42.246 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:19:41 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:19:42.286 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9a047fe7599a549042d06a3fb5b6813a203eabea shell --fsid 713bc57a-0991-11ef-bc97-c7b262605968 -- ceph tell osd.0 flush_pg_stats 2024-05-03T21:19:42.287 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9a047fe7599a549042d06a3fb5b6813a203eabea shell --fsid 713bc57a-0991-11ef-bc97-c7b262605968 -- ceph tell osd.1 flush_pg_stats 2024-05-03T21:19:42.287 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9a047fe7599a549042d06a3fb5b6813a203eabea shell --fsid 713bc57a-0991-11ef-bc97-c7b262605968 -- ceph tell osd.2 flush_pg_stats 2024-05-03T21:19:42.287 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9a047fe7599a549042d06a3fb5b6813a203eabea shell --fsid 713bc57a-0991-11ef-bc97-c7b262605968 -- ceph tell osd.3 flush_pg_stats 2024-05-03T21:19:42.288 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9a047fe7599a549042d06a3fb5b6813a203eabea shell --fsid 713bc57a-0991-11ef-bc97-c7b262605968 -- ceph tell osd.4 flush_pg_stats 2024-05-03T21:19:42.288 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9a047fe7599a549042d06a3fb5b6813a203eabea shell --fsid 713bc57a-0991-11ef-bc97-c7b262605968 -- ceph tell osd.5 flush_pg_stats 2024-05-03T21:19:42.289 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9a047fe7599a549042d06a3fb5b6813a203eabea shell --fsid 713bc57a-0991-11ef-bc97-c7b262605968 -- ceph tell osd.6 flush_pg_stats 2024-05-03T21:19:42.289 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9a047fe7599a549042d06a3fb5b6813a203eabea shell --fsid 713bc57a-0991-11ef-bc97-c7b262605968 -- ceph tell osd.7 flush_pg_stats 2024-05-03T21:19:42.289 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9a047fe7599a549042d06a3fb5b6813a203eabea shell --fsid 713bc57a-0991-11ef-bc97-c7b262605968 -- ceph tell osd.8 flush_pg_stats 2024-05-03T21:19:42.290 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9a047fe7599a549042d06a3fb5b6813a203eabea shell --fsid 713bc57a-0991-11ef-bc97-c7b262605968 -- ceph tell osd.9 flush_pg_stats 2024-05-03T21:19:42.290 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9a047fe7599a549042d06a3fb5b6813a203eabea shell --fsid 713bc57a-0991-11ef-bc97-c7b262605968 -- ceph tell osd.10 flush_pg_stats 2024-05-03T21:19:42.291 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9a047fe7599a549042d06a3fb5b6813a203eabea shell --fsid 713bc57a-0991-11ef-bc97-c7b262605968 -- ceph tell osd.11 flush_pg_stats 2024-05-03T21:19:42.324 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:19:41 smithi138 ceph-mon[33110]: from='client.? 172.21.15.67:0/810052473' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-03T21:19:42.324 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:19:41 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:19:42.324 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:19:41 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:19:42.324 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:19:41 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:19:42.325 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:19:41 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:19:42.978 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/713bc57a-0991-11ef-bc97-c7b262605968/mon.a/config 2024-05-03T21:19:42.989 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/713bc57a-0991-11ef-bc97-c7b262605968/mon.a/config 2024-05-03T21:19:43.302 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:19:42 smithi067 ceph-mon[29730]: pgmap v318: 97 pgs: 97 active+clean; 632 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s; 13 B/s, 1 objects/s recovering 2024-05-03T21:19:43.302 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:19:42 smithi067 ceph-mon[29730]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 1/192 objects degraded (0.521%), 1 pg degraded) 2024-05-03T21:19:43.302 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:19:42 smithi067 ceph-mon[29730]: Cluster is now healthy 2024-05-03T21:19:43.324 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:19:42 smithi138 ceph-mon[33110]: pgmap v318: 97 pgs: 97 active+clean; 632 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s; 13 B/s, 1 objects/s recovering 2024-05-03T21:19:43.324 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:19:42 smithi138 ceph-mon[33110]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 1/192 objects degraded (0.521%), 1 pg degraded) 2024-05-03T21:19:43.325 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:19:42 smithi138 ceph-mon[33110]: Cluster is now healthy 2024-05-03T21:19:43.495 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:19:42 smithi082 ceph-mon[32933]: pgmap v318: 97 pgs: 97 active+clean; 632 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s; 13 B/s, 1 objects/s recovering 2024-05-03T21:19:43.496 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:19:42 smithi082 ceph-mon[32933]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 1/192 objects degraded (0.521%), 1 pg degraded) 2024-05-03T21:19:43.496 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:19:42 smithi082 ceph-mon[32933]: Cluster is now healthy 2024-05-03T21:19:44.868 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/713bc57a-0991-11ef-bc97-c7b262605968/mon.a/config 2024-05-03T21:19:44.868 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/713bc57a-0991-11ef-bc97-c7b262605968/mon.a/config 2024-05-03T21:19:45.298 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:19:44 smithi082 ceph-mon[32933]: pgmap v319: 97 pgs: 97 active+clean; 632 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s; 11 B/s, 1 objects/s recovering 2024-05-03T21:19:45.322 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:19:44 smithi067 ceph-mon[29730]: pgmap v319: 97 pgs: 97 active+clean; 632 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s; 11 B/s, 1 objects/s recovering 2024-05-03T21:19:45.324 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:19:44 smithi138 ceph-mon[33110]: pgmap v319: 97 pgs: 97 active+clean; 632 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s; 11 B/s, 1 objects/s recovering 2024-05-03T21:19:46.278 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:44.852+0000 7f3e89432700 1 -- 172.21.15.67:0/148090491 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3e84109500 msgr2=0x7f3e841098e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:19:46.278 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:44.852+0000 7f3e89432700 1 --2- 172.21.15.67:0/148090491 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3e84109500 0x7f3e841098e0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f3e7c00b0a0 tx=0x7f3e7c00b3b0 comp rx=0 tx=0).stop 2024-05-03T21:19:46.278 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:44.852+0000 7f3e89432700 1 -- 172.21.15.67:0/148090491 shutdown_connections 2024-05-03T21:19:46.278 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:44.852+0000 7f3e89432700 1 --2- 172.21.15.67:0/148090491 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3e84109500 0x7f3e841098e0 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:19:46.279 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:44.852+0000 7f3e89432700 1 --2- 172.21.15.67:0/148090491 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f3e8406c3d0 0x7f3e84110320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:19:46.279 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:44.852+0000 7f3e89432700 1 --2- 172.21.15.67:0/148090491 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f3e8406ba10 0x7f3e8406be90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:19:46.279 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:44.852+0000 7f3e89432700 1 -- 172.21.15.67:0/148090491 >> 172.21.15.67:0/148090491 conn(0x7f3e84079310 msgr2=0x7f3e84079720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T21:19:46.279 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:44.852+0000 7f3e89432700 1 -- 172.21.15.67:0/148090491 shutdown_connections 2024-05-03T21:19:46.279 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:44.852+0000 7f3e89432700 1 -- 172.21.15.67:0/148090491 wait complete. 2024-05-03T21:19:46.279 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:44.852+0000 7f3e89432700 1 Processor -- start 2024-05-03T21:19:46.279 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:44.852+0000 7f3e89432700 1 -- start start 2024-05-03T21:19:46.279 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:44.853+0000 7f3e89432700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f3e8406ba10 0x7f3e84102460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:19:46.279 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:44.853+0000 7f3e89432700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3e8406c3d0 0x7f3e841029a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:19:46.279 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:44.853+0000 7f3e89432700 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f3e84109500 0x7f3e84102ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:19:46.280 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:44.853+0000 7f3e89432700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f3e841138c0 con 0x7f3e8406c3d0 2024-05-03T21:19:46.280 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:44.853+0000 7f3e89432700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f3e84113740 con 0x7f3e8406ba10 2024-05-03T21:19:46.280 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:44.853+0000 7f3e89432700 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f3e84113a40 con 0x7f3e84109500 2024-05-03T21:19:46.280 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:44.853+0000 7f3e827fc700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3e8406c3d0 0x7f3e841029a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:19:46.280 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:44.853+0000 7f3e827fc700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3e8406c3d0 0x7f3e841029a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:60566/0 (socket says 172.21.15.67:60566) 2024-05-03T21:19:46.280 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:44.853+0000 7f3e827fc700 1 -- 172.21.15.67:0/3505646241 learned_addr learned my addr 172.21.15.67:0/3505646241 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-03T21:19:46.280 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:44.853+0000 7f3e837fe700 1 --2- 172.21.15.67:0/3505646241 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f3e84109500 0x7f3e84102ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:19:46.280 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:44.853+0000 7f3e82ffd700 1 --2- 172.21.15.67:0/3505646241 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f3e8406ba10 0x7f3e84102460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:19:46.280 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:44.853+0000 7f3e827fc700 1 -- 172.21.15.67:0/3505646241 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f3e84109500 msgr2=0x7f3e84102ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:19:46.280 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:44.853+0000 7f3e827fc700 1 --2- 172.21.15.67:0/3505646241 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f3e84109500 0x7f3e84102ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:19:46.281 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:44.853+0000 7f3e827fc700 1 -- 172.21.15.67:0/3505646241 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f3e8406ba10 msgr2=0x7f3e84102460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:19:46.281 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:44.853+0000 7f3e827fc700 1 --2- 172.21.15.67:0/3505646241 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f3e8406ba10 0x7f3e84102460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:19:46.281 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:44.853+0000 7f3e827fc700 1 -- 172.21.15.67:0/3505646241 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3e841a8280 con 0x7f3e8406c3d0 2024-05-03T21:19:46.281 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:44.853+0000 7f3e827fc700 1 --2- 172.21.15.67:0/3505646241 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3e8406c3d0 0x7f3e841029a0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f3e6c00d980 tx=0x7f3e6c00dd40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T21:19:46.281 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:44.853+0000 7f3e7bfff700 1 -- 172.21.15.67:0/3505646241 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e6c017070 con 0x7f3e8406c3d0 2024-05-03T21:19:46.281 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:44.853+0000 7f3e89432700 1 -- 172.21.15.67:0/3505646241 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3e841a8570 con 0x7f3e8406c3d0 2024-05-03T21:19:46.282 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:44.854+0000 7f3e7bfff700 1 -- 172.21.15.67:0/3505646241 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(28 keys) ==== 1170+0+0 (secure 0 0 0) 0x7f3e6c004d30 con 0x7f3e8406c3d0 2024-05-03T21:19:46.282 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:44.854+0000 7f3e7bfff700 1 -- 172.21.15.67:0/3505646241 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e6c0053f0 con 0x7f3e8406c3d0 2024-05-03T21:19:46.282 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:44.854+0000 7f3e89432700 1 -- 172.21.15.67:0/3505646241 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3e841a8ab0 con 0x7f3e8406c3d0 2024-05-03T21:19:46.282 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:44.854+0000 7f3e89432700 1 -- 172.21.15.67:0/3505646241 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f3e84063bb0 con 0x7f3e8406c3d0 2024-05-03T21:19:46.282 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:44.856+0000 7f3e7bfff700 1 -- 172.21.15.67:0/3505646241 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 17) ==== 149703+0+0 (secure 0 0 0) 0x7f3e6c0055b0 con 0x7f3e8406c3d0 2024-05-03T21:19:46.282 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:44.856+0000 7f3e7bfff700 1 --2- 172.21.15.67:0/3505646241 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f3e700b2910 0x7f3e700b4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:19:46.282 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:44.857+0000 7f3e82ffd700 1 --2- 172.21.15.67:0/3505646241 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f3e700b2910 0x7f3e700b4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:19:46.282 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:44.857+0000 7f3e7bfff700 1 -- 172.21.15.67:0/3505646241 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 8211+0+0 (secure 0 0 0) 0x7f3e6c0e28f0 con 0x7f3e8406c3d0 2024-05-03T21:19:46.282 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:44.857+0000 7f3e7bfff700 1 --2- 172.21.15.67:0/3505646241 >> [v2:172.21.15.67:6802/3692483807,v1:172.21.15.67:6803/3692483807] conn(0x7f3e700bc450 0x7f3e700be8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:19:46.282 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:44.857+0000 7f3e7bfff700 1 -- 172.21.15.67:0/3505646241 --> [v2:172.21.15.67:6802/3692483807,v1:172.21.15.67:6803/3692483807] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f3e700befc0 con 0x7f3e700bc450 2024-05-03T21:19:46.282 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:44.857+0000 7f3e7bfff700 1 -- 172.21.15.67:0/3505646241 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_get_version_reply(handle=1 version=84) ==== 24+0+0 (secure 0 0 0) 0x7f3e6c0e2ca0 con 0x7f3e8406c3d0 2024-05-03T21:19:46.283 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:44.857+0000 7f3e837fe700 1 --2- 172.21.15.67:0/3505646241 >> [v2:172.21.15.67:6802/3692483807,v1:172.21.15.67:6803/3692483807] conn(0x7f3e700bc450 0x7f3e700be8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:19:46.283 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:44.857+0000 7f3e82ffd700 1 --2- 172.21.15.67:0/3505646241 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f3e700b2910 0x7f3e700b4dd0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f3e7c0062a0 tx=0x7f3e7c0061f0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T21:19:46.283 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:44.858+0000 7f3e837fe700 1 --2- 172.21.15.67:0/3505646241 >> [v2:172.21.15.67:6802/3692483807,v1:172.21.15.67:6803/3692483807] conn(0x7f3e700bc450 0x7f3e700be8f0 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T21:19:46.283 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:44.859+0000 7f3e7bfff700 1 -- 172.21.15.67:0/3505646241 <== osd.0 v2:172.21.15.67:6802/3692483807 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f3e700befc0 con 0x7f3e700bc450 2024-05-03T21:19:46.283 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:44.891+0000 7f3e89432700 1 -- 172.21.15.67:0/3505646241 --> [v2:172.21.15.67:6802/3692483807,v1:172.21.15.67:6803/3692483807] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f3e84064030 con 0x7f3e700bc450 2024-05-03T21:19:46.283 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:44.892+0000 7f3e7bfff700 1 -- 172.21.15.67:0/3505646241 <== osd.0 v2:172.21.15.67:6802/3692483807 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f3e84064030 con 0x7f3e700bc450 2024-05-03T21:19:46.283 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:44.892+0000 7f3e89432700 1 -- 172.21.15.67:0/3505646241 >> [v2:172.21.15.67:6802/3692483807,v1:172.21.15.67:6803/3692483807] conn(0x7f3e700bc450 msgr2=0x7f3e700be8f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:19:46.283 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:44.892+0000 7f3e89432700 1 --2- 172.21.15.67:0/3505646241 >> [v2:172.21.15.67:6802/3692483807,v1:172.21.15.67:6803/3692483807] conn(0x7f3e700bc450 0x7f3e700be8f0 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:19:46.283 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:44.892+0000 7f3e89432700 1 -- 172.21.15.67:0/3505646241 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f3e700b2910 msgr2=0x7f3e700b4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:19:46.283 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:44.892+0000 7f3e89432700 1 --2- 172.21.15.67:0/3505646241 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f3e700b2910 0x7f3e700b4dd0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f3e7c0062a0 tx=0x7f3e7c0061f0 comp rx=0 tx=0).stop 2024-05-03T21:19:46.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:44.892+0000 7f3e89432700 1 -- 172.21.15.67:0/3505646241 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3e8406c3d0 msgr2=0x7f3e841029a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:19:46.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:44.892+0000 7f3e89432700 1 --2- 172.21.15.67:0/3505646241 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3e8406c3d0 0x7f3e841029a0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f3e6c00d980 tx=0x7f3e6c00dd40 comp rx=0 tx=0).stop 2024-05-03T21:19:46.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:44.892+0000 7f3e837fe700 1 -- 172.21.15.67:0/3505646241 reap_dead start 2024-05-03T21:19:46.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:44.892+0000 7f3e89432700 1 -- 172.21.15.67:0/3505646241 shutdown_connections 2024-05-03T21:19:46.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:44.892+0000 7f3e89432700 1 -- 172.21.15.67:0/3505646241 >> 172.21.15.67:0/3505646241 conn(0x7f3e84079310 msgr2=0x7f3e84100f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T21:19:46.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:44.893+0000 7f3e89432700 1 -- 172.21.15.67:0/3505646241 shutdown_connections 2024-05-03T21:19:46.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:44.893+0000 7f3e89432700 1 -- 172.21.15.67:0/3505646241 wait complete. 2024-05-03T21:19:46.471 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:46.469+0000 7f807ab1b700 1 -- 172.21.15.67:0/394003701 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f807406c3d0 msgr2=0x7f8074110320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:19:46.472 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:46.469+0000 7f807ab1b700 1 --2- 172.21.15.67:0/394003701 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f807406c3d0 0x7f8074110320 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f806c00b0a0 tx=0x7f806c00b3b0 comp rx=0 tx=0).stop 2024-05-03T21:19:46.472 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:46.469+0000 7f807ab1b700 1 -- 172.21.15.67:0/394003701 shutdown_connections 2024-05-03T21:19:46.472 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:46.469+0000 7f807ab1b700 1 --2- 172.21.15.67:0/394003701 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f807406c3d0 0x7f8074110320 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:19:46.472 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:46.469+0000 7f807ab1b700 1 --2- 172.21.15.67:0/394003701 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f8074109500 0x7f80741098e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:19:46.472 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:46.469+0000 7f807ab1b700 1 --2- 172.21.15.67:0/394003701 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f807406ba10 0x7f807406be90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:19:46.472 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:46.469+0000 7f807ab1b700 1 -- 172.21.15.67:0/394003701 >> 172.21.15.67:0/394003701 conn(0x7f8074079310 msgr2=0x7f8074079720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T21:19:46.472 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:46.469+0000 7f807ab1b700 1 -- 172.21.15.67:0/394003701 shutdown_connections 2024-05-03T21:19:46.472 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:46.469+0000 7f807ab1b700 1 -- 172.21.15.67:0/394003701 wait complete. 2024-05-03T21:19:46.474 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:46.470+0000 7f807ab1b700 1 Processor -- start 2024-05-03T21:19:46.474 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:46.470+0000 7f807ab1b700 1 -- start start 2024-05-03T21:19:46.474 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:46.470+0000 7f807ab1b700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f807406ba10 0x7f807419f8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:19:46.474 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:46.470+0000 7f807ab1b700 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f807406c3d0 0x7f807419fde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:19:46.474 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:46.470+0000 7f807ab1b700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8074109500 0x7f8074199a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:19:46.474 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:46.470+0000 7f807ab1b700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f80741138a0 con 0x7f8074109500 2024-05-03T21:19:46.474 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:46.470+0000 7f807ab1b700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f8074113720 con 0x7f807406ba10 2024-05-03T21:19:46.474 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:46.470+0000 7f807ab1b700 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f8074113a20 con 0x7f807406c3d0 2024-05-03T21:19:46.474 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:46.470+0000 7f80790b8700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8074109500 0x7f8074199a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:19:46.474 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:46.470+0000 7f80788b7700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f807406ba10 0x7f807419f8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:19:46.475 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:46.470+0000 7f80788b7700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f807406ba10 0x7f807419f8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.67:42688/0 (socket says 172.21.15.67:42688) 2024-05-03T21:19:46.475 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:46.470+0000 7f80788b7700 1 -- 172.21.15.67:0/1470028183 learned_addr learned my addr 172.21.15.67:0/1470028183 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-03T21:19:46.475 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:46.470+0000 7f8073fff700 1 --2- 172.21.15.67:0/1470028183 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f807406c3d0 0x7f807419fde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:19:46.475 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:46.471+0000 7f80788b7700 1 -- 172.21.15.67:0/1470028183 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f807406c3d0 msgr2=0x7f807419fde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:19:46.475 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:46.471+0000 7f80788b7700 1 --2- 172.21.15.67:0/1470028183 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f807406c3d0 0x7f807419fde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:19:46.475 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:46.471+0000 7f80788b7700 1 -- 172.21.15.67:0/1470028183 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8074109500 msgr2=0x7f8074199a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:19:46.475 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:46.471+0000 7f80788b7700 1 --2- 172.21.15.67:0/1470028183 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8074109500 0x7f8074199a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:19:46.475 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:46.471+0000 7f80788b7700 1 -- 172.21.15.67:0/1470028183 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f807419a1c0 con 0x7f807406ba10 2024-05-03T21:19:46.475 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:46.471+0000 7f80788b7700 1 --2- 172.21.15.67:0/1470028183 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f807406ba10 0x7f807419f8a0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f806800d980 tx=0x7f806800dc90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T21:19:46.475 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:46.471+0000 7f8071ffb700 1 -- 172.21.15.67:0/1470028183 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8068016070 con 0x7f807406ba10 2024-05-03T21:19:46.475 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:46.471+0000 7f807ab1b700 1 -- 172.21.15.67:0/1470028183 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f807419a4b0 con 0x7f807406ba10 2024-05-03T21:19:46.475 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:46.471+0000 7f8071ffb700 1 -- 172.21.15.67:0/1470028183 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(28 keys) ==== 1170+0+0 (secure 0 0 0) 0x7f8068004d30 con 0x7f807406ba10 2024-05-03T21:19:46.475 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:46.471+0000 7f8071ffb700 1 -- 172.21.15.67:0/1470028183 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80680056c0 con 0x7f807406ba10 2024-05-03T21:19:46.475 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:46.471+0000 7f807ab1b700 1 -- 172.21.15.67:0/1470028183 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8074102390 con 0x7f807406ba10 2024-05-03T21:19:46.476 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:46.472+0000 7f807ab1b700 1 -- 172.21.15.67:0/1470028183 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f8074101930 con 0x7f807406ba10 2024-05-03T21:19:46.476 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:46.473+0000 7f8071ffb700 1 -- 172.21.15.67:0/1470028183 <== mon.2 v2:172.21.15.82:3300/0 4 ==== mgrmap(e 17) ==== 149703+0+0 (secure 0 0 0) 0x7f8068004750 con 0x7f807406ba10 2024-05-03T21:19:46.477 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:46.474+0000 7f8071ffb700 1 --2- 172.21.15.67:0/1470028183 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f80640b2990 0x7f80640b4e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:19:46.477 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:46.474+0000 7f8073fff700 1 --2- 172.21.15.67:0/1470028183 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f80640b2990 0x7f80640b4e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:19:46.477 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:46.474+0000 7f8071ffb700 1 -- 172.21.15.67:0/1470028183 <== mon.2 v2:172.21.15.82:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 8211+0+0 (secure 0 0 0) 0x7f80680e26d0 con 0x7f807406ba10 2024-05-03T21:19:46.477 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:46.475+0000 7f8071ffb700 1 --2- 172.21.15.67:0/1470028183 >> [v2:172.21.15.67:6826/3939417129,v1:172.21.15.67:6827/3939417129] conn(0x7f80640bc4d0 0x7f80640be970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:19:46.477 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:46.475+0000 7f8071ffb700 1 -- 172.21.15.67:0/1470028183 --> [v2:172.21.15.67:6826/3939417129,v1:172.21.15.67:6827/3939417129] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f806801ce50 con 0x7f80640bc4d0 2024-05-03T21:19:46.477 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:46.475+0000 7f8071ffb700 1 -- 172.21.15.67:0/1470028183 <== mon.2 v2:172.21.15.82:3300/0 6 ==== mon_get_version_reply(handle=1 version=84) ==== 24+0+0 (secure 0 0 0) 0x7f80680e2a80 con 0x7f807406ba10 2024-05-03T21:19:46.477 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:46.475+0000 7f80790b8700 1 --2- 172.21.15.67:0/1470028183 >> [v2:172.21.15.67:6826/3939417129,v1:172.21.15.67:6827/3939417129] conn(0x7f80640bc4d0 0x7f80640be970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:19:46.477 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:46.475+0000 7f8073fff700 1 --2- 172.21.15.67:0/1470028183 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f80640b2990 0x7f80640b4e50 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f8060009680 tx=0x7f8060009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T21:19:46.477 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:46.475+0000 7f80790b8700 1 --2- 172.21.15.67:0/1470028183 >> [v2:172.21.15.67:6826/3939417129,v1:172.21.15.67:6827/3939417129] conn(0x7f80640bc4d0 0x7f80640be970 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.9 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T21:19:46.478 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:46.476+0000 7f8071ffb700 1 -- 172.21.15.67:0/1470028183 <== osd.9 v2:172.21.15.67:6826/3939417129 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f806801ce50 con 0x7f80640bc4d0 2024-05-03T21:19:46.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:46.493+0000 7f807ab1b700 1 -- 172.21.15.67:0/1470028183 --> [v2:172.21.15.67:6826/3939417129,v1:172.21.15.67:6827/3939417129] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f8074104bc0 con 0x7f80640bc4d0 2024-05-03T21:19:46.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:46.494+0000 7f8071ffb700 1 -- 172.21.15.67:0/1470028183 <== osd.9 v2:172.21.15.67:6826/3939417129 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f8074104bc0 con 0x7f80640bc4d0 2024-05-03T21:19:46.497 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:46.494+0000 7f807ab1b700 1 -- 172.21.15.67:0/1470028183 >> [v2:172.21.15.67:6826/3939417129,v1:172.21.15.67:6827/3939417129] conn(0x7f80640bc4d0 msgr2=0x7f80640be970 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:19:46.497 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:46.494+0000 7f807ab1b700 1 --2- 172.21.15.67:0/1470028183 >> [v2:172.21.15.67:6826/3939417129,v1:172.21.15.67:6827/3939417129] conn(0x7f80640bc4d0 0x7f80640be970 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:19:46.497 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:46.494+0000 7f807ab1b700 1 -- 172.21.15.67:0/1470028183 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f80640b2990 msgr2=0x7f80640b4e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:19:46.497 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:46.494+0000 7f807ab1b700 1 --2- 172.21.15.67:0/1470028183 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f80640b2990 0x7f80640b4e50 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f8060009680 tx=0x7f8060009260 comp rx=0 tx=0).stop 2024-05-03T21:19:46.497 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:46.494+0000 7f807ab1b700 1 -- 172.21.15.67:0/1470028183 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f807406ba10 msgr2=0x7f807419f8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:19:46.497 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:46.494+0000 7f807ab1b700 1 --2- 172.21.15.67:0/1470028183 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f807406ba10 0x7f807419f8a0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f806800d980 tx=0x7f806800dc90 comp rx=0 tx=0).stop 2024-05-03T21:19:46.498 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:46.494+0000 7f80790b8700 1 -- 172.21.15.67:0/1470028183 reap_dead start 2024-05-03T21:19:46.498 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:46.494+0000 7f807ab1b700 1 -- 172.21.15.67:0/1470028183 shutdown_connections 2024-05-03T21:19:46.498 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:46.494+0000 7f807ab1b700 1 -- 172.21.15.67:0/1470028183 >> 172.21.15.67:0/1470028183 conn(0x7f8074079310 msgr2=0x7f8074100ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T21:19:46.498 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:46.494+0000 7f807ab1b700 1 -- 172.21.15.67:0/1470028183 shutdown_connections 2024-05-03T21:19:46.498 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:46.494+0000 7f807ab1b700 1 -- 172.21.15.67:0/1470028183 wait complete. 2024-05-03T21:19:46.996 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:19:46 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:19:46.996 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:19:46 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:19:46.996 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:19:46 smithi082 ceph-mon[32933]: pgmap v320: 97 pgs: 97 active+clean; 632 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 6 op/s; 10 B/s, 1 objects/s recovering 2024-05-03T21:19:46.996 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:19:46 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T21:19:46.996 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:19:46 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T21:19:46.996 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:19:46 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:19:47.003 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:19:46 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:19:47.003 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:19:46 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:19:47.003 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:19:46 smithi067 ceph-mon[29730]: pgmap v320: 97 pgs: 97 active+clean; 632 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 6 op/s; 10 B/s, 1 objects/s recovering 2024-05-03T21:19:47.003 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:19:46 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T21:19:47.003 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:19:46 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T21:19:47.003 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:19:46 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:19:47.074 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:19:46 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:19:47.075 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:19:46 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:19:47.075 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:19:46 smithi138 ceph-mon[33110]: pgmap v320: 97 pgs: 97 active+clean; 632 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 6 op/s; 10 B/s, 1 objects/s recovering 2024-05-03T21:19:47.075 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:19:46 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T21:19:47.075 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:19:46 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T21:19:47.075 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:19:46 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:19:47.431 INFO:teuthology.orchestra.run.smithi067.stdout:34359738451 2024-05-03T21:19:47.432 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9a047fe7599a549042d06a3fb5b6813a203eabea shell --fsid 713bc57a-0991-11ef-bc97-c7b262605968 -- ceph osd last-stat-seq osd.0 2024-05-03T21:19:48.950 INFO:teuthology.orchestra.run.smithi067.stdout:270582939687 2024-05-03T21:19:48.951 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9a047fe7599a549042d06a3fb5b6813a203eabea shell --fsid 713bc57a-0991-11ef-bc97-c7b262605968 -- ceph osd last-stat-seq osd.9 2024-05-03T21:19:49.221 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:19:48 smithi067 ceph-mon[29730]: pgmap v321: 97 pgs: 97 active+clean; 632 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 6 op/s; 9 B/s, 0 objects/s recovering 2024-05-03T21:19:49.246 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:19:48 smithi082 ceph-mon[32933]: pgmap v321: 97 pgs: 97 active+clean; 632 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 6 op/s; 9 B/s, 0 objects/s recovering 2024-05-03T21:19:49.306 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/713bc57a-0991-11ef-bc97-c7b262605968/mon.a/config 2024-05-03T21:19:49.310 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/713bc57a-0991-11ef-bc97-c7b262605968/mon.a/config 2024-05-03T21:19:49.311 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/713bc57a-0991-11ef-bc97-c7b262605968/mon.a/config 2024-05-03T21:19:49.311 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/713bc57a-0991-11ef-bc97-c7b262605968/mon.a/config 2024-05-03T21:19:49.312 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/713bc57a-0991-11ef-bc97-c7b262605968/mon.a/config 2024-05-03T21:19:49.313 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/713bc57a-0991-11ef-bc97-c7b262605968/mon.a/config 2024-05-03T21:19:49.319 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/713bc57a-0991-11ef-bc97-c7b262605968/mon.a/config 2024-05-03T21:19:49.324 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:19:48 smithi138 ceph-mon[33110]: pgmap v321: 97 pgs: 97 active+clean; 632 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 6 op/s; 9 B/s, 0 objects/s recovering 2024-05-03T21:19:50.439 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/713bc57a-0991-11ef-bc97-c7b262605968/mon.a/config 2024-05-03T21:19:50.462 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:50.459+0000 7fc3ca7da700 1 -- 172.21.15.67:0/1861206124 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc3c4102540 msgr2=0x7fc3c410e9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:19:50.462 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:50.459+0000 7fc3ca7da700 1 --2- 172.21.15.67:0/1861206124 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc3c4102540 0x7fc3c410e9b0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fc3bc00c240 tx=0x7fc3bc00c550 comp rx=0 tx=0).stop 2024-05-03T21:19:50.462 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:50.459+0000 7fc3ca7da700 1 -- 172.21.15.67:0/1861206124 shutdown_connections 2024-05-03T21:19:50.462 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:50.459+0000 7fc3ca7da700 1 --2- 172.21.15.67:0/1861206124 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc3c4102540 0x7fc3c410e9b0 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:19:50.462 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:50.459+0000 7fc3ca7da700 1 --2- 172.21.15.67:0/1861206124 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fc3c4107be0 0x7fc3c4107fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:19:50.462 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:50.459+0000 7fc3ca7da700 1 --2- 172.21.15.67:0/1861206124 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fc3c4101b80 0x7fc3c4102000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:19:50.462 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:50.459+0000 7fc3ca7da700 1 -- 172.21.15.67:0/1861206124 >> 172.21.15.67:0/1861206124 conn(0x7fc3c4078a10 msgr2=0x7fc3c40ffdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T21:19:50.462 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:50.459+0000 7fc3ca7da700 1 -- 172.21.15.67:0/1861206124 shutdown_connections 2024-05-03T21:19:50.463 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:50.459+0000 7fc3ca7da700 1 -- 172.21.15.67:0/1861206124 wait complete. 2024-05-03T21:19:50.464 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:50.460+0000 7fc3ca7da700 1 Processor -- start 2024-05-03T21:19:50.465 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:50.460+0000 7fc3ca7da700 1 -- start start 2024-05-03T21:19:50.465 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:50.461+0000 7fc3ca7da700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fc3c4101b80 0x7fc3c41037a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:19:50.465 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:50.461+0000 7fc3ca7da700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc3c4102540 0x7fc3c4103ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:19:50.465 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:50.461+0000 7fc3ca7da700 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fc3c4107be0 0x7fc3c4104220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:19:50.465 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:50.461+0000 7fc3ca7da700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fc3c4111730 con 0x7fc3c4102540 2024-05-03T21:19:50.465 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:50.461+0000 7fc3ca7da700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7fc3c41115b0 con 0x7fc3c4101b80 2024-05-03T21:19:50.465 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:50.461+0000 7fc3ca7da700 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7fc3c41118b0 con 0x7fc3c4107be0 2024-05-03T21:19:50.465 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:50.461+0000 7fc3c37fe700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc3c4102540 0x7fc3c4103ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:19:50.465 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:50.461+0000 7fc3c37fe700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc3c4102540 0x7fc3c4103ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:43060/0 (socket says 172.21.15.67:43060) 2024-05-03T21:19:50.465 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:50.461+0000 7fc3c37fe700 1 -- 172.21.15.67:0/3617931036 learned_addr learned my addr 172.21.15.67:0/3617931036 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-03T21:19:50.465 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:50.461+0000 7fc3c3fff700 1 --2- 172.21.15.67:0/3617931036 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fc3c4101b80 0x7fc3c41037a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:19:50.465 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:50.461+0000 7fc3c8d77700 1 --2- 172.21.15.67:0/3617931036 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fc3c4107be0 0x7fc3c4104220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:19:50.466 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:50.461+0000 7fc3c37fe700 1 -- 172.21.15.67:0/3617931036 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fc3c4107be0 msgr2=0x7fc3c4104220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:19:50.466 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:50.461+0000 7fc3c37fe700 1 --2- 172.21.15.67:0/3617931036 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fc3c4107be0 0x7fc3c4104220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:19:50.466 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:50.461+0000 7fc3c37fe700 1 -- 172.21.15.67:0/3617931036 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fc3c4101b80 msgr2=0x7fc3c41037a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:19:50.466 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:50.461+0000 7fc3c37fe700 1 --2- 172.21.15.67:0/3617931036 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fc3c4101b80 0x7fc3c41037a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:19:50.466 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:50.461+0000 7fc3c37fe700 1 -- 172.21.15.67:0/3617931036 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc3c41a8130 con 0x7fc3c4102540 2024-05-03T21:19:50.466 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:50.461+0000 7fc3c3fff700 1 --2- 172.21.15.67:0/3617931036 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fc3c4101b80 0x7fc3c41037a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-03T21:19:50.466 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:50.461+0000 7fc3c37fe700 1 --2- 172.21.15.67:0/3617931036 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc3c4102540 0x7fc3c4103ce0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fc3b800c9b0 tx=0x7fc3b800ccc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T21:19:50.466 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:50.461+0000 7fc3c17fa700 1 -- 172.21.15.67:0/3617931036 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3b8007b00 con 0x7fc3c4102540 2024-05-03T21:19:50.466 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:50.462+0000 7fc3ca7da700 1 -- 172.21.15.67:0/3617931036 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc3c41a8420 con 0x7fc3c4102540 2024-05-03T21:19:50.466 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:50.462+0000 7fc3ca7da700 1 -- 172.21.15.67:0/3617931036 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc3c41a89b0 con 0x7fc3c4102540 2024-05-03T21:19:50.466 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:50.462+0000 7fc3c17fa700 1 -- 172.21.15.67:0/3617931036 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(28 keys) ==== 1170+0+0 (secure 0 0 0) 0x7fc3b800ce50 con 0x7fc3c4102540 2024-05-03T21:19:50.466 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:50.462+0000 7fc3c17fa700 1 -- 172.21.15.67:0/3617931036 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3b80186f0 con 0x7fc3c4102540 2024-05-03T21:19:50.466 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:50.462+0000 7fc3ca7da700 1 -- 172.21.15.67:0/3617931036 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fc3b0001030 con 0x7fc3c4102540 2024-05-03T21:19:50.467 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:50.464+0000 7fc3c17fa700 1 -- 172.21.15.67:0/3617931036 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 17) ==== 149703+0+0 (secure 0 0 0) 0x7fc3b8007ca0 con 0x7fc3c4102540 2024-05-03T21:19:50.467 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:50.465+0000 7fc3c17fa700 1 --2- 172.21.15.67:0/3617931036 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7fc3ac0b2a10 0x7fc3ac0b4ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:19:50.468 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:50.465+0000 7fc3c17fa700 1 -- 172.21.15.67:0/3617931036 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 8211+0+0 (secure 0 0 0) 0x7fc3b80e2760 con 0x7fc3c4102540 2024-05-03T21:19:50.468 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:50.466+0000 7fc3c3fff700 1 --2- 172.21.15.67:0/3617931036 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7fc3ac0b2a10 0x7fc3ac0b4ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:19:50.468 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:50.466+0000 7fc3c17fa700 1 --2- 172.21.15.67:0/3617931036 >> [v2:172.21.15.67:6810/3915934652,v1:172.21.15.67:6811/3915934652] conn(0x7fc3ac0bc550 0x7fc3ac0be9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:19:50.468 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:50.466+0000 7fc3c17fa700 1 -- 172.21.15.67:0/3617931036 --> [v2:172.21.15.67:6810/3915934652,v1:172.21.15.67:6811/3915934652] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fc3ac0bf0c0 con 0x7fc3ac0bc550 2024-05-03T21:19:50.468 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:50.466+0000 7fc3c17fa700 1 -- 172.21.15.67:0/3617931036 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_get_version_reply(handle=1 version=84) ==== 24+0+0 (secure 0 0 0) 0x7fc3b80e2b50 con 0x7fc3c4102540 2024-05-03T21:19:50.468 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:50.466+0000 7fc3c8d77700 1 --2- 172.21.15.67:0/3617931036 >> [v2:172.21.15.67:6810/3915934652,v1:172.21.15.67:6811/3915934652] conn(0x7fc3ac0bc550 0x7fc3ac0be9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:19:50.469 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:50.466+0000 7fc3c3fff700 1 --2- 172.21.15.67:0/3617931036 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7fc3ac0b2a10 0x7fc3ac0b4ed0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fc3b4005950 tx=0x7fc3b400b410 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T21:19:50.469 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:50.467+0000 7fc3c8d77700 1 --2- 172.21.15.67:0/3617931036 >> [v2:172.21.15.67:6810/3915934652,v1:172.21.15.67:6811/3915934652] conn(0x7fc3ac0bc550 0x7fc3ac0be9f0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T21:19:50.469 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:50.468+0000 7fc3c17fa700 1 -- 172.21.15.67:0/3617931036 <== osd.3 v2:172.21.15.67:6810/3915934652 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fc3ac0bf0c0 con 0x7fc3ac0bc550 2024-05-03T21:19:50.482 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/713bc57a-0991-11ef-bc97-c7b262605968/mon.a/config 2024-05-03T21:19:50.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:50.487+0000 7fc3ca7da700 1 -- 172.21.15.67:0/3617931036 --> [v2:172.21.15.67:6810/3915934652,v1:172.21.15.67:6811/3915934652] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fc3b0002e60 con 0x7fc3ac0bc550 2024-05-03T21:19:50.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:50.488+0000 7fc3c17fa700 1 -- 172.21.15.67:0/3617931036 <== osd.3 v2:172.21.15.67:6810/3915934652 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7fc3b0002e60 con 0x7fc3ac0bc550 2024-05-03T21:19:50.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:50.488+0000 7fc3ca7da700 1 -- 172.21.15.67:0/3617931036 >> [v2:172.21.15.67:6810/3915934652,v1:172.21.15.67:6811/3915934652] conn(0x7fc3ac0bc550 msgr2=0x7fc3ac0be9f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:19:50.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:50.488+0000 7fc3ca7da700 1 --2- 172.21.15.67:0/3617931036 >> [v2:172.21.15.67:6810/3915934652,v1:172.21.15.67:6811/3915934652] conn(0x7fc3ac0bc550 0x7fc3ac0be9f0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:19:50.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:50.488+0000 7fc3ca7da700 1 -- 172.21.15.67:0/3617931036 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7fc3ac0b2a10 msgr2=0x7fc3ac0b4ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:19:50.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:50.488+0000 7fc3ca7da700 1 --2- 172.21.15.67:0/3617931036 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7fc3ac0b2a10 0x7fc3ac0b4ed0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fc3b4005950 tx=0x7fc3b400b410 comp rx=0 tx=0).stop 2024-05-03T21:19:50.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:50.488+0000 7fc3ca7da700 1 -- 172.21.15.67:0/3617931036 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc3c4102540 msgr2=0x7fc3c4103ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:19:50.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:50.488+0000 7fc3ca7da700 1 --2- 172.21.15.67:0/3617931036 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc3c4102540 0x7fc3c4103ce0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fc3b800c9b0 tx=0x7fc3b800ccc0 comp rx=0 tx=0).stop 2024-05-03T21:19:50.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:50.488+0000 7fc3c8d77700 1 -- 172.21.15.67:0/3617931036 reap_dead start 2024-05-03T21:19:50.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:50.488+0000 7fc3ca7da700 1 -- 172.21.15.67:0/3617931036 shutdown_connections 2024-05-03T21:19:50.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:50.488+0000 7fc3ca7da700 1 -- 172.21.15.67:0/3617931036 >> 172.21.15.67:0/3617931036 conn(0x7fc3c4078a10 msgr2=0x7fc3c40ffe60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T21:19:50.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:50.489+0000 7fc3ca7da700 1 -- 172.21.15.67:0/3617931036 shutdown_connections 2024-05-03T21:19:50.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:50.489+0000 7fc3ca7da700 1 -- 172.21.15.67:0/3617931036 wait complete. 2024-05-03T21:19:50.891 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/713bc57a-0991-11ef-bc97-c7b262605968/mon.a/config 2024-05-03T21:19:51.196 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:19:50 smithi067 ceph-mon[29730]: pgmap v322: 97 pgs: 97 active+clean; 632 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s; 0 B/s, 0 objects/s recovering 2024-05-03T21:19:51.245 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:19:50 smithi082 ceph-mon[32933]: pgmap v322: 97 pgs: 97 active+clean; 632 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s; 0 B/s, 0 objects/s recovering 2024-05-03T21:19:51.324 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:19:50 smithi138 ceph-mon[33110]: pgmap v322: 97 pgs: 97 active+clean; 632 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s; 0 B/s, 0 objects/s recovering 2024-05-03T21:19:52.344 INFO:teuthology.orchestra.run.smithi067.stdout:115964117061 2024-05-03T21:19:52.344 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9a047fe7599a549042d06a3fb5b6813a203eabea shell --fsid 713bc57a-0991-11ef-bc97-c7b262605968 -- ceph osd last-stat-seq osd.3 2024-05-03T21:19:52.935 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:19:52 smithi067 ceph-mon[29730]: pgmap v323: 97 pgs: 97 active+clean; 632 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s; 0 B/s, 0 objects/s recovering 2024-05-03T21:19:53.245 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:19:52 smithi082 ceph-mon[32933]: pgmap v323: 97 pgs: 97 active+clean; 632 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s; 0 B/s, 0 objects/s recovering 2024-05-03T21:19:53.324 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:19:52 smithi138 ceph-mon[33110]: pgmap v323: 97 pgs: 97 active+clean; 632 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s; 0 B/s, 0 objects/s recovering 2024-05-03T21:19:54.971 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:19:54 smithi067 ceph-mon[29730]: pgmap v324: 97 pgs: 97 active+clean; 632 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:19:55.245 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:19:54 smithi082 ceph-mon[32933]: pgmap v324: 97 pgs: 97 active+clean; 632 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:19:55.324 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:19:54 smithi138 ceph-mon[33110]: pgmap v324: 97 pgs: 97 active+clean; 632 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:19:56.945 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:19:56 smithi067 ceph-mon[29730]: pgmap v325: 97 pgs: 97 active+clean; 632 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:19:57.245 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:19:56 smithi082 ceph-mon[32933]: pgmap v325: 97 pgs: 97 active+clean; 632 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:19:57.324 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:19:56 smithi138 ceph-mon[33110]: pgmap v325: 97 pgs: 97 active+clean; 632 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:19:59.163 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:19:58 smithi067 ceph-mon[29730]: pgmap v326: 97 pgs: 97 active+clean; 632 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:19:59.247 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:19:58 smithi082 ceph-mon[32933]: pgmap v326: 97 pgs: 97 active+clean; 632 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:19:59.324 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:19:58 smithi138 ceph-mon[33110]: pgmap v326: 97 pgs: 97 active+clean; 632 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:19:59.802 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:59.800+0000 7f2d20b2e700 1 -- 172.21.15.67:0/3782344921 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2d1c102fe0 msgr2=0x7f2d1c106fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:19:59.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:59.800+0000 7f2d20b2e700 1 --2- 172.21.15.67:0/3782344921 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2d1c102fe0 0x7f2d1c106fe0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f2d100099c0 tx=0x7f2d10009cd0 comp rx=0 tx=0).stop 2024-05-03T21:19:59.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:59.800+0000 7f2d20b2e700 1 -- 172.21.15.67:0/3782344921 shutdown_connections 2024-05-03T21:19:59.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:59.800+0000 7f2d20b2e700 1 --2- 172.21.15.67:0/3782344921 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f2d1c1076c0 0x7f2d1c10f290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:19:59.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:59.800+0000 7f2d20b2e700 1 --2- 172.21.15.67:0/3782344921 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f2d1c102630 0x7f2d1c102a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:19:59.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:59.800+0000 7f2d20b2e700 1 --2- 172.21.15.67:0/3782344921 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2d1c102fe0 0x7f2d1c106fe0 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:19:59.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:59.800+0000 7f2d20b2e700 1 -- 172.21.15.67:0/3782344921 >> 172.21.15.67:0/3782344921 conn(0x7f2d1c079350 msgr2=0x7f2d1c079760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T21:19:59.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:59.800+0000 7f2d20b2e700 1 -- 172.21.15.67:0/3782344921 shutdown_connections 2024-05-03T21:19:59.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:59.800+0000 7f2d20b2e700 1 -- 172.21.15.67:0/3782344921 wait complete. 2024-05-03T21:19:59.805 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:59.801+0000 7f2d20b2e700 1 Processor -- start 2024-05-03T21:19:59.805 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:59.801+0000 7f2d20b2e700 1 -- start start 2024-05-03T21:19:59.805 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:59.801+0000 7f2d20b2e700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f2d1c102630 0x7f2d1c104080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:19:59.805 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:59.801+0000 7f2d20b2e700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2d1c102fe0 0x7f2d1c1045c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:19:59.805 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:59.801+0000 7f2d20b2e700 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f2d1c1076c0 0x7f2d1c104b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:19:59.805 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:59.801+0000 7f2d20b2e700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f2d1c1127e0 con 0x7f2d1c102fe0 2024-05-03T21:19:59.806 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:59.801+0000 7f2d20b2e700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f2d1c112660 con 0x7f2d1c102630 2024-05-03T21:19:59.806 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:59.801+0000 7f2d20b2e700 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f2d1c112960 con 0x7f2d1c1076c0 2024-05-03T21:19:59.806 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:59.801+0000 7f2d19d9b700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2d1c102fe0 0x7f2d1c1045c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:19:59.806 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:59.801+0000 7f2d19d9b700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2d1c102fe0 0x7f2d1c1045c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:36358/0 (socket says 172.21.15.67:36358) 2024-05-03T21:19:59.806 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:59.801+0000 7f2d19d9b700 1 -- 172.21.15.67:0/1059969170 learned_addr learned my addr 172.21.15.67:0/1059969170 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-03T21:19:59.806 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:59.801+0000 7f2d1a59c700 1 --2- 172.21.15.67:0/1059969170 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f2d1c102630 0x7f2d1c104080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:19:59.806 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:59.802+0000 7f2d1ad9d700 1 --2- 172.21.15.67:0/1059969170 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f2d1c1076c0 0x7f2d1c104b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:19:59.806 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:59.802+0000 7f2d19d9b700 1 -- 172.21.15.67:0/1059969170 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f2d1c1076c0 msgr2=0x7f2d1c104b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:19:59.806 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:59.802+0000 7f2d19d9b700 1 --2- 172.21.15.67:0/1059969170 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f2d1c1076c0 0x7f2d1c104b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:19:59.806 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:59.802+0000 7f2d19d9b700 1 -- 172.21.15.67:0/1059969170 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f2d1c102630 msgr2=0x7f2d1c104080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:19:59.806 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:59.802+0000 7f2d19d9b700 1 --2- 172.21.15.67:0/1059969170 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f2d1c102630 0x7f2d1c104080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:19:59.806 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:59.802+0000 7f2d19d9b700 1 -- 172.21.15.67:0/1059969170 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2d1c1a8290 con 0x7f2d1c102fe0 2024-05-03T21:19:59.806 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:59.802+0000 7f2d1a59c700 1 --2- 172.21.15.67:0/1059969170 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f2d1c102630 0x7f2d1c104080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-03T21:19:59.807 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:59.802+0000 7f2d19d9b700 1 --2- 172.21.15.67:0/1059969170 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2d1c102fe0 0x7f2d1c1045c0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f2d1000b580 tx=0x7f2d1000f690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T21:19:59.807 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:59.802+0000 7f2d0b7fe700 1 -- 172.21.15.67:0/1059969170 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d10009e60 con 0x7f2d1c102fe0 2024-05-03T21:19:59.807 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:59.802+0000 7f2d20b2e700 1 -- 172.21.15.67:0/1059969170 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2d1c1a8520 con 0x7f2d1c102fe0 2024-05-03T21:19:59.807 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:59.802+0000 7f2d0b7fe700 1 -- 172.21.15.67:0/1059969170 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(28 keys) ==== 1170+0+0 (secure 0 0 0) 0x7f2d1000be40 con 0x7f2d1c102fe0 2024-05-03T21:19:59.807 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:59.802+0000 7f2d0b7fe700 1 -- 172.21.15.67:0/1059969170 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d10005610 con 0x7f2d1c102fe0 2024-05-03T21:19:59.807 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:59.802+0000 7f2d20b2e700 1 -- 172.21.15.67:0/1059969170 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2d1c1a8970 con 0x7f2d1c102fe0 2024-05-03T21:19:59.807 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:59.803+0000 7f2d20b2e700 1 -- 172.21.15.67:0/1059969170 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f2d1c063bb0 con 0x7f2d1c102fe0 2024-05-03T21:19:59.808 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:59.805+0000 7f2d0b7fe700 1 -- 172.21.15.67:0/1059969170 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 17) ==== 149703+0+0 (secure 0 0 0) 0x7f2d10003b60 con 0x7f2d1c102fe0 2024-05-03T21:19:59.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:59.805+0000 7f2d0b7fe700 1 --2- 172.21.15.67:0/1059969170 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f2d040b2910 0x7f2d040b4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:19:59.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:59.806+0000 7f2d1a59c700 1 --2- 172.21.15.67:0/1059969170 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f2d040b2910 0x7f2d040b4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:19:59.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:59.806+0000 7f2d0b7fe700 1 -- 172.21.15.67:0/1059969170 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 8211+0+0 (secure 0 0 0) 0x7f2d100e30f0 con 0x7f2d1c102fe0 2024-05-03T21:19:59.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:59.806+0000 7f2d0b7fe700 1 --2- 172.21.15.67:0/1059969170 >> [v2:172.21.15.138:6824/2528685018,v1:172.21.15.138:6825/2528685018] conn(0x7f2d040bc450 0x7f2d040be8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:19:59.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:59.806+0000 7f2d0b7fe700 1 -- 172.21.15.67:0/1059969170 --> [v2:172.21.15.138:6824/2528685018,v1:172.21.15.138:6825/2528685018] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f2d040befc0 con 0x7f2d040bc450 2024-05-03T21:19:59.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:59.806+0000 7f2d0b7fe700 1 -- 172.21.15.67:0/1059969170 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_get_version_reply(handle=1 version=84) ==== 24+0+0 (secure 0 0 0) 0x7f2d100e3540 con 0x7f2d1c102fe0 2024-05-03T21:19:59.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:59.806+0000 7f2d1a59c700 1 --2- 172.21.15.67:0/1059969170 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f2d040b2910 0x7f2d040b4dd0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f2d0c011490 tx=0x7f2d0c009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T21:19:59.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:59.806+0000 7f2d1ad9d700 1 --2- 172.21.15.67:0/1059969170 >> [v2:172.21.15.138:6824/2528685018,v1:172.21.15.138:6825/2528685018] conn(0x7f2d040bc450 0x7f2d040be8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:19:59.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:59.807+0000 7f2d1ad9d700 1 --2- 172.21.15.67:0/1059969170 >> [v2:172.21.15.138:6824/2528685018,v1:172.21.15.138:6825/2528685018] conn(0x7f2d040bc450 0x7f2d040be8f0 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.11 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T21:19:59.810 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:59.807+0000 7f2d0b7fe700 1 -- 172.21.15.67:0/1059969170 <== osd.11 v2:172.21.15.138:6824/2528685018 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f2d040befc0 con 0x7f2d040bc450 2024-05-03T21:19:59.827 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:59.825+0000 7f2d20b2e700 1 -- 172.21.15.67:0/1059969170 --> [v2:172.21.15.138:6824/2528685018,v1:172.21.15.138:6825/2528685018] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f2d1c064030 con 0x7f2d040bc450 2024-05-03T21:19:59.828 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:59.826+0000 7f2d0b7fe700 1 -- 172.21.15.67:0/1059969170 <== osd.11 v2:172.21.15.138:6824/2528685018 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f2d1c064030 con 0x7f2d040bc450 2024-05-03T21:19:59.829 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:59.826+0000 7f2d20b2e700 1 -- 172.21.15.67:0/1059969170 >> [v2:172.21.15.138:6824/2528685018,v1:172.21.15.138:6825/2528685018] conn(0x7f2d040bc450 msgr2=0x7f2d040be8f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:19:59.829 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:59.826+0000 7f2d20b2e700 1 --2- 172.21.15.67:0/1059969170 >> [v2:172.21.15.138:6824/2528685018,v1:172.21.15.138:6825/2528685018] conn(0x7f2d040bc450 0x7f2d040be8f0 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:19:59.829 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:59.826+0000 7f2d20b2e700 1 -- 172.21.15.67:0/1059969170 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f2d040b2910 msgr2=0x7f2d040b4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:19:59.829 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:59.826+0000 7f2d20b2e700 1 --2- 172.21.15.67:0/1059969170 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f2d040b2910 0x7f2d040b4dd0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f2d0c011490 tx=0x7f2d0c009260 comp rx=0 tx=0).stop 2024-05-03T21:19:59.829 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:59.826+0000 7f2d20b2e700 1 -- 172.21.15.67:0/1059969170 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2d1c102fe0 msgr2=0x7f2d1c1045c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:19:59.829 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:59.826+0000 7f2d20b2e700 1 --2- 172.21.15.67:0/1059969170 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2d1c102fe0 0x7f2d1c1045c0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f2d1000b580 tx=0x7f2d1000f690 comp rx=0 tx=0).stop 2024-05-03T21:19:59.829 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:59.826+0000 7f2d1ad9d700 1 -- 172.21.15.67:0/1059969170 reap_dead start 2024-05-03T21:19:59.829 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:59.826+0000 7f2d20b2e700 1 -- 172.21.15.67:0/1059969170 shutdown_connections 2024-05-03T21:19:59.829 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:59.826+0000 7f2d20b2e700 1 -- 172.21.15.67:0/1059969170 >> 172.21.15.67:0/1059969170 conn(0x7f2d1c079350 msgr2=0x7f2d1c06be50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T21:19:59.829 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:59.826+0000 7f2d20b2e700 1 -- 172.21.15.67:0/1059969170 shutdown_connections 2024-05-03T21:19:59.829 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:19:59.826+0000 7f2d20b2e700 1 -- 172.21.15.67:0/1059969170 wait complete. 2024-05-03T21:20:00.924 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:00.919+0000 7f98c403b700 1 -- 172.21.15.67:0/393523888 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f98bc104100 msgr2=0x7f98bc108120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:20:00.924 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:00.919+0000 7f98c403b700 1 --2- 172.21.15.67:0/393523888 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f98bc104100 0x7f98bc108120 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f98ac0099c0 tx=0x7f98ac009cd0 comp rx=0 tx=0).stop 2024-05-03T21:20:00.924 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:00.919+0000 7f98c403b700 1 -- 172.21.15.67:0/393523888 shutdown_connections 2024-05-03T21:20:00.924 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:00.919+0000 7f98c403b700 1 --2- 172.21.15.67:0/393523888 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f98bc108850 0x7f98bc1103d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:20:00.924 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:00.919+0000 7f98c403b700 1 --2- 172.21.15.67:0/393523888 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f98bc103750 0x7f98bc103b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:20:00.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:00.919+0000 7f98c403b700 1 --2- 172.21.15.67:0/393523888 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f98bc104100 0x7f98bc108120 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:20:00.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:00.919+0000 7f98c403b700 1 -- 172.21.15.67:0/393523888 >> 172.21.15.67:0/393523888 conn(0x7f98bc07a600 msgr2=0x7f98bc07aa10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T21:20:00.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:00.920+0000 7f98c403b700 1 -- 172.21.15.67:0/393523888 shutdown_connections 2024-05-03T21:20:00.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:00.920+0000 7f98c403b700 1 -- 172.21.15.67:0/393523888 wait complete. 2024-05-03T21:20:00.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:00.920+0000 7f98c403b700 1 Processor -- start 2024-05-03T21:20:00.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:00.920+0000 7f98c403b700 1 -- start start 2024-05-03T21:20:00.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:00.921+0000 7f98c403b700 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f98bc103750 0x7f98bc19f8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:20:00.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:00.921+0000 7f98c403b700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f98bc104100 0x7f98bc19fe00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:20:00.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:00.921+0000 7f98c403b700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f98bc108850 0x7f98bc199a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:20:00.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:00.921+0000 7f98c403b700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f98bc113920 con 0x7f98bc108850 2024-05-03T21:20:00.926 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:00.921+0000 7f98c403b700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f98bc1137a0 con 0x7f98bc104100 2024-05-03T21:20:00.926 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:00.921+0000 7f98c403b700 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f98bc113aa0 con 0x7f98bc103750 2024-05-03T21:20:00.926 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:00.921+0000 7f98c25d8700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f98bc108850 0x7f98bc199a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:20:00.926 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:00.921+0000 7f98c25d8700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f98bc108850 0x7f98bc199a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:36366/0 (socket says 172.21.15.67:36366) 2024-05-03T21:20:00.926 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:00.921+0000 7f98c25d8700 1 -- 172.21.15.67:0/1444121607 learned_addr learned my addr 172.21.15.67:0/1444121607 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-03T21:20:00.926 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:00.921+0000 7f98c15d6700 1 --2- 172.21.15.67:0/1444121607 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f98bc104100 0x7f98bc19fe00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:20:00.926 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:00.921+0000 7f98c1dd7700 1 --2- 172.21.15.67:0/1444121607 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f98bc103750 0x7f98bc19f8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:20:00.926 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:00.921+0000 7f98c25d8700 1 -- 172.21.15.67:0/1444121607 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f98bc103750 msgr2=0x7f98bc19f8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:20:00.926 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:00.921+0000 7f98c25d8700 1 --2- 172.21.15.67:0/1444121607 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f98bc103750 0x7f98bc19f8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:20:00.926 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:00.921+0000 7f98c25d8700 1 -- 172.21.15.67:0/1444121607 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f98bc104100 msgr2=0x7f98bc19fe00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:20:00.926 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:00.921+0000 7f98c25d8700 1 --2- 172.21.15.67:0/1444121607 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f98bc104100 0x7f98bc19fe00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:20:00.927 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:00.921+0000 7f98c25d8700 1 -- 172.21.15.67:0/1444121607 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f98bc19a1e0 con 0x7f98bc108850 2024-05-03T21:20:00.927 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:00.921+0000 7f98c25d8700 1 --2- 172.21.15.67:0/1444121607 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f98bc108850 0x7f98bc199a80 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f98b800b550 tx=0x7f98b800b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T21:20:00.927 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:00.921+0000 7f98b2ffd700 1 -- 172.21.15.67:0/1444121607 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98b8015020 con 0x7f98bc108850 2024-05-03T21:20:00.927 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:00.921+0000 7f98c403b700 1 -- 172.21.15.67:0/1444121607 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f98bc19a4d0 con 0x7f98bc108850 2024-05-03T21:20:00.927 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:00.921+0000 7f98b2ffd700 1 -- 172.21.15.67:0/1444121607 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(28 keys) ==== 1170+0+0 (secure 0 0 0) 0x7f98b8011d30 con 0x7f98bc108850 2024-05-03T21:20:00.927 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:00.921+0000 7f98b2ffd700 1 -- 172.21.15.67:0/1444121607 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98b800f2e0 con 0x7f98bc108850 2024-05-03T21:20:00.927 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:00.921+0000 7f98c403b700 1 -- 172.21.15.67:0/1444121607 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f98bc1a6790 con 0x7f98bc108850 2024-05-03T21:20:00.929 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:00.922+0000 7f98c403b700 1 -- 172.21.15.67:0/1444121607 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f98bc101ec0 con 0x7f98bc108850 2024-05-03T21:20:00.929 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:00.924+0000 7f98b2ffd700 1 -- 172.21.15.67:0/1444121607 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 17) ==== 149703+0+0 (secure 0 0 0) 0x7f98b8018020 con 0x7f98bc108850 2024-05-03T21:20:00.929 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:00.924+0000 7f98b2ffd700 1 --2- 172.21.15.67:0/1444121607 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f98a80b2910 0x7f98a80b4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:20:00.929 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:00.925+0000 7f98c1dd7700 1 --2- 172.21.15.67:0/1444121607 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f98a80b2910 0x7f98a80b4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:20:00.929 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:00.925+0000 7f98b2ffd700 1 -- 172.21.15.67:0/1444121607 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 8211+0+0 (secure 0 0 0) 0x7f98b80e1f70 con 0x7f98bc108850 2024-05-03T21:20:00.929 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:00.925+0000 7f98b2ffd700 1 --2- 172.21.15.67:0/1444121607 >> [v2:172.21.15.82:6800/1139376397,v1:172.21.15.82:6801/1139376397] conn(0x7f98a80bc450 0x7f98a80be8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:20:00.929 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:00.925+0000 7f98b2ffd700 1 -- 172.21.15.67:0/1444121607 --> [v2:172.21.15.82:6800/1139376397,v1:172.21.15.82:6801/1139376397] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f98b801be50 con 0x7f98a80bc450 2024-05-03T21:20:00.929 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:00.925+0000 7f98b2ffd700 1 -- 172.21.15.67:0/1444121607 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_get_version_reply(handle=1 version=84) ==== 24+0+0 (secure 0 0 0) 0x7f98b80e23c0 con 0x7f98bc108850 2024-05-03T21:20:00.930 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:00.925+0000 7f98c1dd7700 1 --2- 172.21.15.67:0/1444121607 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f98a80b2910 0x7f98a80b4dd0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f98bc19abc0 tx=0x7f98b4009040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T21:20:00.930 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:00.925+0000 7f98c15d6700 1 --2- 172.21.15.67:0/1444121607 >> [v2:172.21.15.82:6800/1139376397,v1:172.21.15.82:6801/1139376397] conn(0x7f98a80bc450 0x7f98a80be8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:20:00.930 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:00.925+0000 7f98c15d6700 1 --2- 172.21.15.67:0/1444121607 >> [v2:172.21.15.82:6800/1139376397,v1:172.21.15.82:6801/1139376397] conn(0x7f98a80bc450 0x7f98a80be8f0 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T21:20:00.930 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:00.926+0000 7f98b2ffd700 1 -- 172.21.15.67:0/1444121607 <== osd.1 v2:172.21.15.82:6800/1139376397 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f98b801be50 con 0x7f98a80bc450 2024-05-03T21:20:00.945 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:00.943+0000 7f98c403b700 1 -- 172.21.15.67:0/1444121607 --> [v2:172.21.15.82:6800/1139376397,v1:172.21.15.82:6801/1139376397] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f98bc104be0 con 0x7f98a80bc450 2024-05-03T21:20:00.946 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:00.944+0000 7f98b2ffd700 1 -- 172.21.15.67:0/1444121607 <== osd.1 v2:172.21.15.82:6800/1139376397 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f98bc104be0 con 0x7f98a80bc450 2024-05-03T21:20:00.947 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:00.944+0000 7f98c403b700 1 -- 172.21.15.67:0/1444121607 >> [v2:172.21.15.82:6800/1139376397,v1:172.21.15.82:6801/1139376397] conn(0x7f98a80bc450 msgr2=0x7f98a80be8f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:20:00.947 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:00.944+0000 7f98c403b700 1 --2- 172.21.15.67:0/1444121607 >> [v2:172.21.15.82:6800/1139376397,v1:172.21.15.82:6801/1139376397] conn(0x7f98a80bc450 0x7f98a80be8f0 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:20:00.947 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:00.944+0000 7f98c403b700 1 -- 172.21.15.67:0/1444121607 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f98a80b2910 msgr2=0x7f98a80b4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:20:00.947 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:00.944+0000 7f98c403b700 1 --2- 172.21.15.67:0/1444121607 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f98a80b2910 0x7f98a80b4dd0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f98bc19abc0 tx=0x7f98b4009040 comp rx=0 tx=0).stop 2024-05-03T21:20:00.948 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:00.944+0000 7f98c403b700 1 -- 172.21.15.67:0/1444121607 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f98bc108850 msgr2=0x7f98bc199a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:20:00.948 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:00.944+0000 7f98c403b700 1 --2- 172.21.15.67:0/1444121607 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f98bc108850 0x7f98bc199a80 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f98b800b550 tx=0x7f98b800b860 comp rx=0 tx=0).stop 2024-05-03T21:20:00.948 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:00.944+0000 7f98c25d8700 1 -- 172.21.15.67:0/1444121607 reap_dead start 2024-05-03T21:20:00.948 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:00.944+0000 7f98c403b700 1 -- 172.21.15.67:0/1444121607 shutdown_connections 2024-05-03T21:20:00.948 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:00.944+0000 7f98c403b700 1 -- 172.21.15.67:0/1444121607 >> 172.21.15.67:0/1444121607 conn(0x7f98bc07a600 msgr2=0x7f98bc101460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T21:20:00.948 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:00.944+0000 7f98c403b700 1 -- 172.21.15.67:0/1444121607 shutdown_connections 2024-05-03T21:20:00.948 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:00.944+0000 7f98c403b700 1 -- 172.21.15.67:0/1444121607 wait complete. 2024-05-03T21:20:01.221 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:20:00 smithi067 ceph-mon[29730]: pgmap v327: 97 pgs: 97 active+clean; 632 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T21:20:01.221 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:20:00 smithi067 ceph-mon[29730]: overall HEALTH_OK 2024-05-03T21:20:01.246 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:20:00 smithi082 ceph-mon[32933]: pgmap v327: 97 pgs: 97 active+clean; 632 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T21:20:01.246 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:20:00 smithi082 ceph-mon[32933]: overall HEALTH_OK 2024-05-03T21:20:01.324 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:20:00 smithi138 ceph-mon[33110]: pgmap v327: 97 pgs: 97 active+clean; 632 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T21:20:01.324 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:20:00 smithi138 ceph-mon[33110]: overall HEALTH_OK 2024-05-03T21:20:01.533 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:01.523+0000 7f1c51a88700 1 -- 172.21.15.67:0/3011608992 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1c4c102530 msgr2=0x7f1c4c112220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:20:01.533 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:01.523+0000 7f1c51a88700 1 --2- 172.21.15.67:0/3011608992 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1c4c102530 0x7f1c4c112220 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f1c4400b0a0 tx=0x7f1c4400b3b0 comp rx=0 tx=0).stop 2024-05-03T21:20:01.533 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:01.523+0000 7f1c51a88700 1 -- 172.21.15.67:0/3011608992 shutdown_connections 2024-05-03T21:20:01.533 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:01.523+0000 7f1c51a88700 1 --2- 172.21.15.67:0/3011608992 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1c4c102530 0x7f1c4c112220 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:20:01.533 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:01.523+0000 7f1c51a88700 1 --2- 172.21.15.67:0/3011608992 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f1c4c1043b0 0x7f1c4c104790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:20:01.533 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:01.523+0000 7f1c51a88700 1 --2- 172.21.15.67:0/3011608992 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f1c4c101a90 0x7f1c4c101f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:20:01.533 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:01.523+0000 7f1c51a88700 1 -- 172.21.15.67:0/3011608992 >> 172.21.15.67:0/3011608992 conn(0x7f1c4c078a10 msgr2=0x7f1c4c0ffdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T21:20:01.533 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:01.523+0000 7f1c51a88700 1 -- 172.21.15.67:0/3011608992 shutdown_connections 2024-05-03T21:20:01.533 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:01.523+0000 7f1c51a88700 1 -- 172.21.15.67:0/3011608992 wait complete. 2024-05-03T21:20:01.534 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:01.523+0000 7f1c51a88700 1 Processor -- start 2024-05-03T21:20:01.534 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:01.524+0000 7f1c51a88700 1 -- start start 2024-05-03T21:20:01.534 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:01.524+0000 7f1c51a88700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1c4c101a90 0x7f1c4c19bed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:20:01.534 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:01.524+0000 7f1c51a88700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f1c4c102530 0x7f1c4c19c410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:20:01.534 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:01.524+0000 7f1c51a88700 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f1c4c1043b0 0x7f1c4c1a07a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:20:01.534 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:01.524+0000 7f1c51a88700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f1c4c115b80 con 0x7f1c4c101a90 2024-05-03T21:20:01.534 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:01.524+0000 7f1c51a88700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f1c4c115a00 con 0x7f1c4c102530 2024-05-03T21:20:01.534 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:01.524+0000 7f1c51a88700 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f1c4c115d00 con 0x7f1c4c1043b0 2024-05-03T21:20:01.534 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:01.524+0000 7f1c4b7fe700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1c4c101a90 0x7f1c4c19bed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:20:01.534 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:01.524+0000 7f1c4b7fe700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1c4c101a90 0x7f1c4c19bed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:36380/0 (socket says 172.21.15.67:36380) 2024-05-03T21:20:01.534 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:01.524+0000 7f1c4b7fe700 1 -- 172.21.15.67:0/985272539 learned_addr learned my addr 172.21.15.67:0/985272539 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-03T21:20:01.534 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:01.524+0000 7f1c4bfff700 1 --2- 172.21.15.67:0/985272539 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f1c4c1043b0 0x7f1c4c1a07a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:20:01.535 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:01.524+0000 7f1c4affd700 1 --2- 172.21.15.67:0/985272539 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f1c4c102530 0x7f1c4c19c410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:20:01.535 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:01.524+0000 7f1c4b7fe700 1 -- 172.21.15.67:0/985272539 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f1c4c1043b0 msgr2=0x7f1c4c1a07a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:20:01.535 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:01.524+0000 7f1c4b7fe700 1 --2- 172.21.15.67:0/985272539 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f1c4c1043b0 0x7f1c4c1a07a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:20:01.535 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:01.524+0000 7f1c4b7fe700 1 -- 172.21.15.67:0/985272539 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f1c4c102530 msgr2=0x7f1c4c19c410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:20:01.535 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:01.524+0000 7f1c4b7fe700 1 --2- 172.21.15.67:0/985272539 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f1c4c102530 0x7f1c4c19c410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:20:01.535 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:01.524+0000 7f1c4b7fe700 1 -- 172.21.15.67:0/985272539 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1c4c1a0ee0 con 0x7f1c4c101a90 2024-05-03T21:20:01.535 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:01.525+0000 7f1c4b7fe700 1 --2- 172.21.15.67:0/985272539 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1c4c101a90 0x7f1c4c19bed0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f1c3400d980 tx=0x7f1c3400dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T21:20:01.535 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:01.525+0000 7f1c48ff9700 1 -- 172.21.15.67:0/985272539 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c34017070 con 0x7f1c4c101a90 2024-05-03T21:20:01.535 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:01.525+0000 7f1c51a88700 1 -- 172.21.15.67:0/985272539 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1c4c1a11d0 con 0x7f1c4c101a90 2024-05-03T21:20:01.535 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:01.525+0000 7f1c48ff9700 1 -- 172.21.15.67:0/985272539 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(28 keys) ==== 1170+0+0 (secure 0 0 0) 0x7f1c34005020 con 0x7f1c4c101a90 2024-05-03T21:20:01.537 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:01.525+0000 7f1c48ff9700 1 -- 172.21.15.67:0/985272539 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c3400fc60 con 0x7f1c4c101a90 2024-05-03T21:20:01.537 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:01.525+0000 7f1c51a88700 1 -- 172.21.15.67:0/985272539 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1c4c1a8a50 con 0x7f1c4c101a90 2024-05-03T21:20:01.537 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:01.526+0000 7f1c51a88700 1 -- 172.21.15.67:0/985272539 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f1c4c063c30 con 0x7f1c4c101a90 2024-05-03T21:20:01.537 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:01.527+0000 7f1c48ff9700 1 -- 172.21.15.67:0/985272539 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 17) ==== 149703+0+0 (secure 0 0 0) 0x7f1c340051d0 con 0x7f1c4c101a90 2024-05-03T21:20:01.537 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:01.528+0000 7f1c48ff9700 1 --2- 172.21.15.67:0/985272539 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f1c380b2910 0x7f1c380b4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:20:01.537 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:01.528+0000 7f1c4affd700 1 --2- 172.21.15.67:0/985272539 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f1c380b2910 0x7f1c380b4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:20:01.537 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:01.528+0000 7f1c48ff9700 1 -- 172.21.15.67:0/985272539 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 8211+0+0 (secure 0 0 0) 0x7f1c340e26d0 con 0x7f1c4c101a90 2024-05-03T21:20:01.537 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:01.528+0000 7f1c48ff9700 1 --2- 172.21.15.67:0/985272539 >> [v2:172.21.15.138:6816/2223920070,v1:172.21.15.138:6817/2223920070] conn(0x7f1c380bc450 0x7f1c380be8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:20:01.538 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:01.528+0000 7f1c48ff9700 1 -- 172.21.15.67:0/985272539 --> [v2:172.21.15.138:6816/2223920070,v1:172.21.15.138:6817/2223920070] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f1c380befc0 con 0x7f1c380bc450 2024-05-03T21:20:01.538 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:01.528+0000 7f1c48ff9700 1 -- 172.21.15.67:0/985272539 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_get_version_reply(handle=1 version=84) ==== 24+0+0 (secure 0 0 0) 0x7f1c340e2b20 con 0x7f1c4c101a90 2024-05-03T21:20:01.538 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:01.529+0000 7f1c4affd700 1 --2- 172.21.15.67:0/985272539 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f1c380b2910 0x7f1c380b4dd0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f1c4c100fb0 tx=0x7f1c3c00b410 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T21:20:01.538 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:01.529+0000 7f1c4bfff700 1 --2- 172.21.15.67:0/985272539 >> [v2:172.21.15.138:6816/2223920070,v1:172.21.15.138:6817/2223920070] conn(0x7f1c380bc450 0x7f1c380be8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:20:01.538 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:01.530+0000 7f1c4bfff700 1 --2- 172.21.15.67:0/985272539 >> [v2:172.21.15.138:6816/2223920070,v1:172.21.15.138:6817/2223920070] conn(0x7f1c380bc450 0x7f1c380be8f0 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T21:20:01.538 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:01.530+0000 7f1c48ff9700 1 -- 172.21.15.67:0/985272539 <== osd.8 v2:172.21.15.138:6816/2223920070 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f1c380befc0 con 0x7f1c380bc450 2024-05-03T21:20:01.551 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:01.549+0000 7f1c51a88700 1 -- 172.21.15.67:0/985272539 --> [v2:172.21.15.138:6816/2223920070,v1:172.21.15.138:6817/2223920070] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f1c4c0640b0 con 0x7f1c380bc450 2024-05-03T21:20:01.561 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:01.550+0000 7f1c48ff9700 1 -- 172.21.15.67:0/985272539 <== osd.8 v2:172.21.15.138:6816/2223920070 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f1c4c0640b0 con 0x7f1c380bc450 2024-05-03T21:20:01.563 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:01.550+0000 7f1c51a88700 1 -- 172.21.15.67:0/985272539 >> [v2:172.21.15.138:6816/2223920070,v1:172.21.15.138:6817/2223920070] conn(0x7f1c380bc450 msgr2=0x7f1c380be8f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:20:01.563 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:01.550+0000 7f1c51a88700 1 --2- 172.21.15.67:0/985272539 >> [v2:172.21.15.138:6816/2223920070,v1:172.21.15.138:6817/2223920070] conn(0x7f1c380bc450 0x7f1c380be8f0 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:20:01.563 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:01.550+0000 7f1c51a88700 1 -- 172.21.15.67:0/985272539 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f1c380b2910 msgr2=0x7f1c380b4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:20:01.563 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:01.550+0000 7f1c51a88700 1 --2- 172.21.15.67:0/985272539 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f1c380b2910 0x7f1c380b4dd0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f1c4c100fb0 tx=0x7f1c3c00b410 comp rx=0 tx=0).stop 2024-05-03T21:20:01.563 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:01.550+0000 7f1c51a88700 1 -- 172.21.15.67:0/985272539 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1c4c101a90 msgr2=0x7f1c4c19bed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:20:01.563 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:01.550+0000 7f1c51a88700 1 --2- 172.21.15.67:0/985272539 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1c4c101a90 0x7f1c4c19bed0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f1c3400d980 tx=0x7f1c3400dc90 comp rx=0 tx=0).stop 2024-05-03T21:20:01.563 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:01.550+0000 7f1c4bfff700 1 -- 172.21.15.67:0/985272539 reap_dead start 2024-05-03T21:20:01.563 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:01.550+0000 7f1c51a88700 1 -- 172.21.15.67:0/985272539 shutdown_connections 2024-05-03T21:20:01.563 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:01.550+0000 7f1c51a88700 1 -- 172.21.15.67:0/985272539 >> 172.21.15.67:0/985272539 conn(0x7f1c4c078a10 msgr2=0x7f1c4c0ff040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T21:20:01.564 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:01.550+0000 7f1c51a88700 1 -- 172.21.15.67:0/985272539 shutdown_connections 2024-05-03T21:20:01.564 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:01.550+0000 7f1c51a88700 1 -- 172.21.15.67:0/985272539 wait complete. 2024-05-03T21:20:02.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:02.361+0000 7f11852aa700 1 -- 172.21.15.67:0/936344230 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1180109510 msgr2=0x7f11801098f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:20:02.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:02.361+0000 7f11852aa700 1 --2- 172.21.15.67:0/936344230 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1180109510 0x7f11801098f0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f117000b0a0 tx=0x7f117000b3b0 comp rx=0 tx=0).stop 2024-05-03T21:20:02.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:02.361+0000 7f11852aa700 1 -- 172.21.15.67:0/936344230 shutdown_connections 2024-05-03T21:20:02.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:02.361+0000 7f11852aa700 1 --2- 172.21.15.67:0/936344230 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1180109510 0x7f11801098f0 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:20:02.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:02.361+0000 7f11852aa700 1 --2- 172.21.15.67:0/936344230 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f118006bb40 0x7f1180110330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:20:02.364 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:02.361+0000 7f11852aa700 1 --2- 172.21.15.67:0/936344230 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f118006b180 0x7f118006b600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:20:02.364 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:02.361+0000 7f11852aa700 1 -- 172.21.15.67:0/936344230 >> 172.21.15.67:0/936344230 conn(0x7f1180079350 msgr2=0x7f1180079760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T21:20:02.364 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:02.361+0000 7f11852aa700 1 -- 172.21.15.67:0/936344230 shutdown_connections 2024-05-03T21:20:02.364 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:02.361+0000 7f11852aa700 1 -- 172.21.15.67:0/936344230 wait complete. 2024-05-03T21:20:02.365 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:02.361+0000 7f11852aa700 1 Processor -- start 2024-05-03T21:20:02.365 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:02.362+0000 7f11852aa700 1 -- start start 2024-05-03T21:20:02.366 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:02.362+0000 7f11852aa700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f118006b180 0x7f118019fa40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:20:02.366 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:02.362+0000 7f11852aa700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f118006bb40 0x7f118019ff80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:20:02.366 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:02.362+0000 7f11852aa700 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f1180109510 0x7f1180199b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:20:02.366 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:02.362+0000 7f11852aa700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f11801138d0 con 0x7f118006b180 2024-05-03T21:20:02.366 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:02.362+0000 7f11852aa700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f1180113750 con 0x7f118006bb40 2024-05-03T21:20:02.366 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:02.362+0000 7f11852aa700 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f1180113a50 con 0x7f1180109510 2024-05-03T21:20:02.366 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:02.362+0000 7f117effd700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f118006b180 0x7f118019fa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:20:02.366 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:02.362+0000 7f117effd700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f118006b180 0x7f118019fa40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:36396/0 (socket says 172.21.15.67:36396) 2024-05-03T21:20:02.366 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:02.362+0000 7f117effd700 1 -- 172.21.15.67:0/179099405 learned_addr learned my addr 172.21.15.67:0/179099405 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-03T21:20:02.366 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:02.362+0000 7f117e7fc700 1 --2- 172.21.15.67:0/179099405 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f118006bb40 0x7f118019ff80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:20:02.366 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:02.362+0000 7f117f7fe700 1 --2- 172.21.15.67:0/179099405 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f1180109510 0x7f1180199b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:20:02.366 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:02.362+0000 7f117effd700 1 -- 172.21.15.67:0/179099405 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f1180109510 msgr2=0x7f1180199b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:20:02.366 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:02.362+0000 7f117effd700 1 --2- 172.21.15.67:0/179099405 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f1180109510 0x7f1180199b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:20:02.367 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:02.362+0000 7f117effd700 1 -- 172.21.15.67:0/179099405 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f118006bb40 msgr2=0x7f118019ff80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:20:02.367 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:02.362+0000 7f117effd700 1 --2- 172.21.15.67:0/179099405 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f118006bb40 0x7f118019ff80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:20:02.367 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:02.362+0000 7f117effd700 1 -- 172.21.15.67:0/179099405 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f118019a3f0 con 0x7f118006b180 2024-05-03T21:20:02.367 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:02.362+0000 7f117f7fe700 1 --2- 172.21.15.67:0/179099405 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f1180109510 0x7f1180199b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-03T21:20:02.367 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:02.362+0000 7f117effd700 1 --2- 172.21.15.67:0/179099405 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f118006b180 0x7f118019fa40 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f117000b070 tx=0x7f1170004dd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T21:20:02.367 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:02.363+0000 7f1177fff700 1 -- 172.21.15.67:0/179099405 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1170021070 con 0x7f118006b180 2024-05-03T21:20:02.367 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:02.363+0000 7f1177fff700 1 -- 172.21.15.67:0/179099405 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(28 keys) ==== 1170+0+0 (secure 0 0 0) 0x7f1170007c00 con 0x7f118006b180 2024-05-03T21:20:02.367 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:02.363+0000 7f11852aa700 1 -- 172.21.15.67:0/179099405 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f118019a680 con 0x7f118006b180 2024-05-03T21:20:02.367 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:02.363+0000 7f1177fff700 1 -- 172.21.15.67:0/179099405 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11700053b0 con 0x7f118006b180 2024-05-03T21:20:02.367 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:02.363+0000 7f11852aa700 1 -- 172.21.15.67:0/179099405 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f11801a69c0 con 0x7f118006b180 2024-05-03T21:20:02.368 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:02.364+0000 7f11852aa700 1 -- 172.21.15.67:0/179099405 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f118010b9f0 con 0x7f118006b180 2024-05-03T21:20:02.368 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:02.366+0000 7f1177fff700 1 -- 172.21.15.67:0/179099405 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 17) ==== 149703+0+0 (secure 0 0 0) 0x7f1170024070 con 0x7f118006b180 2024-05-03T21:20:02.368 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:02.366+0000 7f1177fff700 1 --2- 172.21.15.67:0/179099405 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f116c0b2910 0x7f116c0b4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:20:02.368 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:02.366+0000 7f117e7fc700 1 --2- 172.21.15.67:0/179099405 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f116c0b2910 0x7f116c0b4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:20:02.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:02.366+0000 7f1177fff700 1 -- 172.21.15.67:0/179099405 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 8211+0+0 (secure 0 0 0) 0x7f117000f020 con 0x7f118006b180 2024-05-03T21:20:02.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:02.367+0000 7f117e7fc700 1 --2- 172.21.15.67:0/179099405 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f116c0b2910 0x7f116c0b4dd0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f118019b260 tx=0x7f116800b3c0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T21:20:02.372 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:02.370+0000 7f1177fff700 1 -- 172.21.15.67:0/179099405 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191839 (secure 0 0 0) 0x7f117002b020 con 0x7f118006b180 2024-05-03T21:20:02.531 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:02.529+0000 7f413f3a5700 1 -- 172.21.15.67:0/721357337 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4138103eb0 msgr2=0x7f4138110390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:20:02.531 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:02.529+0000 7f413f3a5700 1 --2- 172.21.15.67:0/721357337 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4138103eb0 0x7f4138110390 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f413000b0a0 tx=0x7f413000b3b0 comp rx=0 tx=0).stop 2024-05-03T21:20:02.532 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:02.529+0000 7f413f3a5700 1 -- 172.21.15.67:0/721357337 shutdown_connections 2024-05-03T21:20:02.532 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:02.529+0000 7f413f3a5700 1 --2- 172.21.15.67:0/721357337 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4138103eb0 0x7f4138110390 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:20:02.532 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:02.529+0000 7f413f3a5700 1 --2- 172.21.15.67:0/721357337 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f4138109600 0x7f41381099e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:20:02.532 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:02.529+0000 7f413f3a5700 1 --2- 172.21.15.67:0/721357337 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f41381034f0 0x7f4138103970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:20:02.532 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:02.529+0000 7f413f3a5700 1 -- 172.21.15.67:0/721357337 >> 172.21.15.67:0/721357337 conn(0x7f413807a650 msgr2=0x7f413807aa60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T21:20:02.532 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:02.529+0000 7f413f3a5700 1 -- 172.21.15.67:0/721357337 shutdown_connections 2024-05-03T21:20:02.532 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:02.529+0000 7f413f3a5700 1 -- 172.21.15.67:0/721357337 wait complete. 2024-05-03T21:20:02.534 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:02.530+0000 7f413f3a5700 1 Processor -- start 2024-05-03T21:20:02.534 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:02.530+0000 7f413f3a5700 1 -- start start 2024-05-03T21:20:02.534 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:02.530+0000 7f413f3a5700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f41381034f0 0x7f413819bf40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:20:02.534 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:02.530+0000 7f413f3a5700 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f4138103eb0 0x7f413819c480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:20:02.534 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:02.530+0000 7f413f3a5700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f4138109600 0x7f41381a0810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:20:02.534 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:02.530+0000 7f413f3a5700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f4138113900 con 0x7f41381034f0 2024-05-03T21:20:02.534 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:02.530+0000 7f413f3a5700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f4138113780 con 0x7f4138109600 2024-05-03T21:20:02.534 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:02.530+0000 7f413f3a5700 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f4138113a80 con 0x7f4138103eb0 2024-05-03T21:20:02.534 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:02.530+0000 7f413d141700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f41381034f0 0x7f413819bf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:20:02.534 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:02.530+0000 7f413d141700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f41381034f0 0x7f413819bf40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:36408/0 (socket says 172.21.15.67:36408) 2024-05-03T21:20:02.535 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:02.530+0000 7f413d141700 1 -- 172.21.15.67:0/2617282159 learned_addr learned my addr 172.21.15.67:0/2617282159 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-03T21:20:02.535 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:02.530+0000 7f413c940700 1 --2- 172.21.15.67:0/2617282159 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f4138103eb0 0x7f413819c480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:20:02.535 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:02.530+0000 7f413d141700 1 -- 172.21.15.67:0/2617282159 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f4138103eb0 msgr2=0x7f413819c480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:20:02.535 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:02.530+0000 7f413d141700 1 --2- 172.21.15.67:0/2617282159 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f4138103eb0 0x7f413819c480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:20:02.535 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:02.530+0000 7f413d141700 1 -- 172.21.15.67:0/2617282159 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f4138109600 msgr2=0x7f41381a0810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:20:02.535 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:02.530+0000 7f413d141700 1 --2- 172.21.15.67:0/2617282159 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f4138109600 0x7f41381a0810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:20:02.535 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:02.530+0000 7f413d141700 1 -- 172.21.15.67:0/2617282159 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f41381a0f50 con 0x7f41381034f0 2024-05-03T21:20:02.535 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:02.531+0000 7f413d141700 1 --2- 172.21.15.67:0/2617282159 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f41381034f0 0x7f413819bf40 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f412800d980 tx=0x7f412800dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T21:20:02.535 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:02.531+0000 7f412e7fc700 1 -- 172.21.15.67:0/2617282159 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4128016070 con 0x7f41381034f0 2024-05-03T21:20:02.535 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:02.531+0000 7f412e7fc700 1 -- 172.21.15.67:0/2617282159 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(28 keys) ==== 1170+0+0 (secure 0 0 0) 0x7f4128004d30 con 0x7f41381034f0 2024-05-03T21:20:02.536 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:02.531+0000 7f412e7fc700 1 -- 172.21.15.67:0/2617282159 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4128005680 con 0x7f41381034f0 2024-05-03T21:20:02.536 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:02.531+0000 7f413f3a5700 1 -- 172.21.15.67:0/2617282159 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f41381a1240 con 0x7f41381034f0 2024-05-03T21:20:02.536 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:02.531+0000 7f413f3a5700 1 -- 172.21.15.67:0/2617282159 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f41381a8ac0 con 0x7f41381034f0 2024-05-03T21:20:02.536 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:02.532+0000 7f413f3a5700 1 -- 172.21.15.67:0/2617282159 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f41380770c0 con 0x7f41381034f0 2024-05-03T21:20:02.537 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:02.533+0000 7f412e7fc700 1 -- 172.21.15.67:0/2617282159 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 17) ==== 149703+0+0 (secure 0 0 0) 0x7f4128004750 con 0x7f41381034f0 2024-05-03T21:20:02.537 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:02.534+0000 7f412e7fc700 1 --2- 172.21.15.67:0/2617282159 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f41240b2990 0x7f41240b4e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:20:02.537 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:02.534+0000 7f413c940700 1 --2- 172.21.15.67:0/2617282159 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f41240b2990 0x7f41240b4e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:20:02.538 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:02.534+0000 7f412e7fc700 1 -- 172.21.15.67:0/2617282159 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 8211+0+0 (secure 0 0 0) 0x7f41280e2720 con 0x7f41381034f0 2024-05-03T21:20:02.538 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:02.534+0000 7f412e7fc700 1 --2- 172.21.15.67:0/2617282159 >> [v2:172.21.15.82:6824/2505507029,v1:172.21.15.82:6825/2505507029] conn(0x7f41240bc530 0x7f41240be9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:20:02.538 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:02.534+0000 7f412e7fc700 1 -- 172.21.15.67:0/2617282159 --> [v2:172.21.15.82:6824/2505507029,v1:172.21.15.82:6825/2505507029] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f412801be50 con 0x7f41240bc530 2024-05-03T21:20:02.538 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:02.534+0000 7f412e7fc700 1 -- 172.21.15.67:0/2617282159 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_get_version_reply(handle=1 version=84) ==== 24+0+0 (secure 0 0 0) 0x7f41280e2b70 con 0x7f41381034f0 2024-05-03T21:20:02.538 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:02.535+0000 7f413c940700 1 --2- 172.21.15.67:0/2617282159 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f41240b2990 0x7f41240b4e50 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f413807af60 tx=0x7f4134009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T21:20:02.538 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:02.535+0000 7f413d942700 1 --2- 172.21.15.67:0/2617282159 >> [v2:172.21.15.82:6824/2505507029,v1:172.21.15.82:6825/2505507029] conn(0x7f41240bc530 0x7f41240be9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:20:02.538 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:02.535+0000 7f413d942700 1 --2- 172.21.15.67:0/2617282159 >> [v2:172.21.15.82:6824/2505507029,v1:172.21.15.82:6825/2505507029] conn(0x7f41240bc530 0x7f41240be9d0 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.10 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T21:20:02.538 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:02.536+0000 7f412e7fc700 1 -- 172.21.15.67:0/2617282159 <== osd.10 v2:172.21.15.82:6824/2505507029 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f412801be50 con 0x7f41240bc530 2024-05-03T21:20:02.555 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:02.553+0000 7f413f3a5700 1 -- 172.21.15.67:0/2617282159 --> [v2:172.21.15.82:6824/2505507029,v1:172.21.15.82:6825/2505507029] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f4138104c10 con 0x7f41240bc530 2024-05-03T21:20:02.555 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:02.553+0000 7f412e7fc700 1 -- 172.21.15.67:0/2617282159 <== osd.10 v2:172.21.15.82:6824/2505507029 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f4138104c10 con 0x7f41240bc530 2024-05-03T21:20:02.556 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:02.553+0000 7f413f3a5700 1 -- 172.21.15.67:0/2617282159 >> [v2:172.21.15.82:6824/2505507029,v1:172.21.15.82:6825/2505507029] conn(0x7f41240bc530 msgr2=0x7f41240be9d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:20:02.556 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:02.553+0000 7f413f3a5700 1 --2- 172.21.15.67:0/2617282159 >> [v2:172.21.15.82:6824/2505507029,v1:172.21.15.82:6825/2505507029] conn(0x7f41240bc530 0x7f41240be9d0 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:20:02.556 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:02.554+0000 7f413f3a5700 1 -- 172.21.15.67:0/2617282159 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f41240b2990 msgr2=0x7f41240b4e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:20:02.556 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:02.554+0000 7f413f3a5700 1 --2- 172.21.15.67:0/2617282159 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f41240b2990 0x7f41240b4e50 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f413807af60 tx=0x7f4134009260 comp rx=0 tx=0).stop 2024-05-03T21:20:02.556 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:02.554+0000 7f413f3a5700 1 -- 172.21.15.67:0/2617282159 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f41381034f0 msgr2=0x7f413819bf40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:20:02.557 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:02.554+0000 7f413f3a5700 1 --2- 172.21.15.67:0/2617282159 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f41381034f0 0x7f413819bf40 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f412800d980 tx=0x7f412800dc90 comp rx=0 tx=0).stop 2024-05-03T21:20:02.557 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:02.554+0000 7f413d942700 1 -- 172.21.15.67:0/2617282159 reap_dead start 2024-05-03T21:20:02.557 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:02.554+0000 7f413f3a5700 1 -- 172.21.15.67:0/2617282159 shutdown_connections 2024-05-03T21:20:02.557 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:02.554+0000 7f413f3a5700 1 -- 172.21.15.67:0/2617282159 >> 172.21.15.67:0/2617282159 conn(0x7f413807a650 msgr2=0x7f41380756d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T21:20:02.557 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:02.554+0000 7f413f3a5700 1 -- 172.21.15.67:0/2617282159 shutdown_connections 2024-05-03T21:20:02.557 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:02.554+0000 7f413f3a5700 1 -- 172.21.15.67:0/2617282159 wait complete. 2024-05-03T21:20:02.603 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:02.600+0000 7f11852aa700 1 -- 172.21.15.67:0/179099405 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) -- 0x7f118006a190 con 0x7f118006b180 2024-05-03T21:20:02.603 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:02.600+0000 7f1177fff700 1 -- 172.21.15.67:0/179099405 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f11700ab9c0 con 0x7f118006b180 2024-05-03T21:20:02.603 INFO:teuthology.orchestra.run.smithi067.stdout:34359738455 2024-05-03T21:20:02.605 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:02.603+0000 7f11852aa700 1 -- 172.21.15.67:0/179099405 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f116c0b2910 msgr2=0x7f116c0b4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:20:02.605 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:02.603+0000 7f11852aa700 1 --2- 172.21.15.67:0/179099405 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f116c0b2910 0x7f116c0b4dd0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f118019b260 tx=0x7f116800b3c0 comp rx=0 tx=0).stop 2024-05-03T21:20:02.605 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:02.603+0000 7f11852aa700 1 -- 172.21.15.67:0/179099405 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f118006b180 msgr2=0x7f118019fa40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:20:02.605 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:02.603+0000 7f11852aa700 1 --2- 172.21.15.67:0/179099405 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f118006b180 0x7f118019fa40 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f117000b070 tx=0x7f1170004dd0 comp rx=0 tx=0).stop 2024-05-03T21:20:02.605 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:02.603+0000 7f11852aa700 1 -- 172.21.15.67:0/179099405 shutdown_connections 2024-05-03T21:20:02.606 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:02.603+0000 7f11852aa700 1 --2- 172.21.15.67:0/179099405 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f116c0b2910 0x7f116c0b4dd0 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:20:02.606 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:02.603+0000 7f11852aa700 1 --2- 172.21.15.67:0/179099405 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f118006b180 0x7f118019fa40 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:20:02.606 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:02.603+0000 7f11852aa700 1 --2- 172.21.15.67:0/179099405 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f1180109510 0x7f1180199b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:20:02.606 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:02.603+0000 7f11852aa700 1 --2- 172.21.15.67:0/179099405 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f118006bb40 0x7f118019ff80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:20:02.606 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:02.603+0000 7f11852aa700 1 -- 172.21.15.67:0/179099405 >> 172.21.15.67:0/179099405 conn(0x7f1180079350 msgr2=0x7f1180100e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T21:20:02.606 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:02.603+0000 7f11852aa700 1 -- 172.21.15.67:0/179099405 shutdown_connections 2024-05-03T21:20:02.606 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:02.603+0000 7f11852aa700 1 -- 172.21.15.67:0/179099405 wait complete. 2024-05-03T21:20:02.967 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:02.964+0000 7f4da6ef0700 1 -- 172.21.15.67:0/2484982406 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4da01076c0 msgr2=0x7f4da010f290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:20:02.967 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:02.964+0000 7f4da6ef0700 1 --2- 172.21.15.67:0/2484982406 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4da01076c0 0x7f4da010f290 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f4d9800b0d0 tx=0x7f4d9800b3e0 comp rx=0 tx=0).stop 2024-05-03T21:20:02.967 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:02.964+0000 7f4da6ef0700 1 -- 172.21.15.67:0/2484982406 shutdown_connections 2024-05-03T21:20:02.967 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:02.964+0000 7f4da6ef0700 1 --2- 172.21.15.67:0/2484982406 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4da01076c0 0x7f4da010f290 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:20:02.967 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:02.964+0000 7f4da6ef0700 1 --2- 172.21.15.67:0/2484982406 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f4da0102630 0x7f4da0102a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:20:02.967 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:02.964+0000 7f4da6ef0700 1 --2- 172.21.15.67:0/2484982406 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f4da0102fe0 0x7f4da0106fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:20:02.967 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:02.964+0000 7f4da6ef0700 1 -- 172.21.15.67:0/2484982406 >> 172.21.15.67:0/2484982406 conn(0x7f4da0079350 msgr2=0x7f4da0079760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T21:20:02.967 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:02.964+0000 7f4da6ef0700 1 -- 172.21.15.67:0/2484982406 shutdown_connections 2024-05-03T21:20:02.967 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:02.964+0000 7f4da6ef0700 1 -- 172.21.15.67:0/2484982406 wait complete. 2024-05-03T21:20:02.967 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:02.965+0000 7f4da6ef0700 1 Processor -- start 2024-05-03T21:20:02.967 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:02.965+0000 7f4da6ef0700 1 -- start start 2024-05-03T21:20:02.969 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:02.965+0000 7f4da6ef0700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4da0102630 0x7f4da019ab70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:20:02.969 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:02.965+0000 7f4da6ef0700 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f4da0102fe0 0x7f4da019b0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:20:02.969 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:02.965+0000 7f4da6ef0700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f4da01076c0 0x7f4da01a5bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:20:02.970 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:02.965+0000 7f4da6ef0700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f4da01128b0 con 0x7f4da0102630 2024-05-03T21:20:02.970 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:02.965+0000 7f4da6ef0700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f4da0112730 con 0x7f4da01076c0 2024-05-03T21:20:02.970 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:02.965+0000 7f4da6ef0700 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f4da0112a30 con 0x7f4da0102fe0 2024-05-03T21:20:02.970 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:02.965+0000 7f4da4c8c700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4da0102630 0x7f4da019ab70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:20:02.970 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:02.965+0000 7f4da4c8c700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4da0102630 0x7f4da019ab70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:36432/0 (socket says 172.21.15.67:36432) 2024-05-03T21:20:02.970 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:02.965+0000 7f4da4c8c700 1 -- 172.21.15.67:0/3212487605 learned_addr learned my addr 172.21.15.67:0/3212487605 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-03T21:20:02.970 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:02.965+0000 7f4da548d700 1 --2- 172.21.15.67:0/3212487605 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f4da01076c0 0x7f4da01a5bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:20:02.970 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:02.965+0000 7f4d9ffff700 1 --2- 172.21.15.67:0/3212487605 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f4da0102fe0 0x7f4da019b0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:20:02.970 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:02.965+0000 7f4da4c8c700 1 -- 172.21.15.67:0/3212487605 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f4da0102fe0 msgr2=0x7f4da019b0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:20:02.970 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:02.965+0000 7f4da4c8c700 1 --2- 172.21.15.67:0/3212487605 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f4da0102fe0 0x7f4da019b0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:20:02.970 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:02.965+0000 7f4da4c8c700 1 -- 172.21.15.67:0/3212487605 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f4da01076c0 msgr2=0x7f4da01a5bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:20:02.970 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:02.965+0000 7f4da4c8c700 1 --2- 172.21.15.67:0/3212487605 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f4da01076c0 0x7f4da01a5bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:20:02.971 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:02.965+0000 7f4da4c8c700 1 -- 172.21.15.67:0/3212487605 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4da01a6130 con 0x7f4da0102630 2024-05-03T21:20:02.971 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:02.965+0000 7f4d9ffff700 1 --2- 172.21.15.67:0/3212487605 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f4da0102fe0 0x7f4da019b0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-03T21:20:02.971 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:02.966+0000 7f4da4c8c700 1 --2- 172.21.15.67:0/3212487605 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4da0102630 0x7f4da019ab70 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f4d9000c9b0 tx=0x7f4d9000ccc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T21:20:02.971 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:02.966+0000 7f4d9dffb700 1 -- 172.21.15.67:0/3212487605 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d90007b00 con 0x7f4da0102630 2024-05-03T21:20:02.971 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:02.966+0000 7f4d9dffb700 1 -- 172.21.15.67:0/3212487605 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(28 keys) ==== 1170+0+0 (secure 0 0 0) 0x7f4d90007ca0 con 0x7f4da0102630 2024-05-03T21:20:02.971 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:02.966+0000 7f4d9dffb700 1 -- 172.21.15.67:0/3212487605 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d90018780 con 0x7f4da0102630 2024-05-03T21:20:02.971 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:02.966+0000 7f4da6ef0700 1 -- 172.21.15.67:0/3212487605 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4da01a6360 con 0x7f4da0102630 2024-05-03T21:20:02.971 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:02.966+0000 7f4da6ef0700 1 -- 172.21.15.67:0/3212487605 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4da01a6820 con 0x7f4da0102630 2024-05-03T21:20:02.971 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:02.967+0000 7f4da6ef0700 1 -- 172.21.15.67:0/3212487605 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f4da0100c20 con 0x7f4da0102630 2024-05-03T21:20:02.972 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:02.968+0000 7f4d9dffb700 1 -- 172.21.15.67:0/3212487605 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 17) ==== 149703+0+0 (secure 0 0 0) 0x7f4d9000f450 con 0x7f4da0102630 2024-05-03T21:20:02.972 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:02.969+0000 7f4d9dffb700 1 --2- 172.21.15.67:0/3212487605 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f4d880b2990 0x7f4d880b4e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:20:02.972 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:02.969+0000 7f4d9ffff700 1 --2- 172.21.15.67:0/3212487605 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f4d880b2990 0x7f4d880b4e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:20:02.972 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:02.969+0000 7f4d9dffb700 1 -- 172.21.15.67:0/3212487605 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 8211+0+0 (secure 0 0 0) 0x7f4d900e2730 con 0x7f4da0102630 2024-05-03T21:20:02.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:02.969+0000 7f4d9dffb700 1 --2- 172.21.15.67:0/3212487605 >> [v2:172.21.15.82:6808/4031818951,v1:172.21.15.82:6809/4031818951] conn(0x7f4d880bc4d0 0x7f4d880be970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:20:02.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:02.969+0000 7f4d9dffb700 1 -- 172.21.15.67:0/3212487605 --> [v2:172.21.15.82:6808/4031818951,v1:172.21.15.82:6809/4031818951] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f4d90015e70 con 0x7f4d880bc4d0 2024-05-03T21:20:02.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:02.969+0000 7f4d9dffb700 1 -- 172.21.15.67:0/3212487605 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_get_version_reply(handle=1 version=84) ==== 24+0+0 (secure 0 0 0) 0x7f4d900e2b80 con 0x7f4da0102630 2024-05-03T21:20:02.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:02.969+0000 7f4d9ffff700 1 --2- 172.21.15.67:0/3212487605 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f4d880b2990 0x7f4d880b4e50 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f4da01088d0 tx=0x7f4d94008040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T21:20:02.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:02.969+0000 7f4da548d700 1 --2- 172.21.15.67:0/3212487605 >> [v2:172.21.15.82:6808/4031818951,v1:172.21.15.82:6809/4031818951] conn(0x7f4d880bc4d0 0x7f4d880be970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:20:02.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:02.970+0000 7f4da548d700 1 --2- 172.21.15.67:0/3212487605 >> [v2:172.21.15.82:6808/4031818951,v1:172.21.15.82:6809/4031818951] conn(0x7f4d880bc4d0 0x7f4d880be970 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T21:20:02.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:02.971+0000 7f4d9dffb700 1 -- 172.21.15.67:0/3212487605 <== osd.4 v2:172.21.15.82:6808/4031818951 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f4d90015e70 con 0x7f4d880bc4d0 2024-05-03T21:20:02.989 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:02.987+0000 7f4da6ef0700 1 -- 172.21.15.67:0/3212487605 --> [v2:172.21.15.82:6808/4031818951,v1:172.21.15.82:6809/4031818951] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f4da0103ac0 con 0x7f4d880bc4d0 2024-05-03T21:20:02.990 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:02.988+0000 7f4d9dffb700 1 -- 172.21.15.67:0/3212487605 <== osd.4 v2:172.21.15.82:6808/4031818951 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f4da0103ac0 con 0x7f4d880bc4d0 2024-05-03T21:20:02.991 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:02.988+0000 7f4da6ef0700 1 -- 172.21.15.67:0/3212487605 >> [v2:172.21.15.82:6808/4031818951,v1:172.21.15.82:6809/4031818951] conn(0x7f4d880bc4d0 msgr2=0x7f4d880be970 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:20:02.991 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:02.988+0000 7f4da6ef0700 1 --2- 172.21.15.67:0/3212487605 >> [v2:172.21.15.82:6808/4031818951,v1:172.21.15.82:6809/4031818951] conn(0x7f4d880bc4d0 0x7f4d880be970 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:20:02.991 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:02.988+0000 7f4da6ef0700 1 -- 172.21.15.67:0/3212487605 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f4d880b2990 msgr2=0x7f4d880b4e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:20:02.991 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:02.988+0000 7f4da6ef0700 1 --2- 172.21.15.67:0/3212487605 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f4d880b2990 0x7f4d880b4e50 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f4da01088d0 tx=0x7f4d94008040 comp rx=0 tx=0).stop 2024-05-03T21:20:02.991 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:02.988+0000 7f4da6ef0700 1 -- 172.21.15.67:0/3212487605 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4da0102630 msgr2=0x7f4da019ab70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:20:02.991 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:02.988+0000 7f4da6ef0700 1 --2- 172.21.15.67:0/3212487605 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4da0102630 0x7f4da019ab70 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f4d9000c9b0 tx=0x7f4d9000ccc0 comp rx=0 tx=0).stop 2024-05-03T21:20:02.991 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:02.988+0000 7f4da548d700 1 -- 172.21.15.67:0/3212487605 reap_dead start 2024-05-03T21:20:02.991 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:02.988+0000 7f4da6ef0700 1 -- 172.21.15.67:0/3212487605 shutdown_connections 2024-05-03T21:20:02.991 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:02.988+0000 7f4da6ef0700 1 -- 172.21.15.67:0/3212487605 >> 172.21.15.67:0/3212487605 conn(0x7f4da0079350 msgr2=0x7f4da0100240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T21:20:02.991 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:02.988+0000 7f4da6ef0700 1 -- 172.21.15.67:0/3212487605 shutdown_connections 2024-05-03T21:20:02.991 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:02.988+0000 7f4da6ef0700 1 -- 172.21.15.67:0/3212487605 wait complete. 2024-05-03T21:20:03.221 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:20:02 smithi067 ceph-mon[29730]: pgmap v328: 97 pgs: 97 active+clean; 632 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:20:03.221 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:20:02 smithi067 ceph-mon[29730]: from='client.? 172.21.15.67:0/179099405' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-05-03T21:20:03.245 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:20:02 smithi082 ceph-mon[32933]: pgmap v328: 97 pgs: 97 active+clean; 632 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:20:03.245 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:20:02 smithi082 ceph-mon[32933]: from='client.? 172.21.15.67:0/179099405' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-05-03T21:20:03.324 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:20:02 smithi138 ceph-mon[33110]: pgmap v328: 97 pgs: 97 active+clean; 632 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:20:03.324 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:20:02 smithi138 ceph-mon[33110]: from='client.? 172.21.15.67:0/179099405' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-05-03T21:20:03.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:03.973+0000 7f75f8b07700 1 -- 172.21.15.67:0/70052171 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f75f406bb40 msgr2=0x7f75f4110330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:20:03.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:03.973+0000 7f75f8b07700 1 --2- 172.21.15.67:0/70052171 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f75f406bb40 0x7f75f4110330 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f75e4009990 tx=0x7f75e4009ca0 comp rx=0 tx=0).stop 2024-05-03T21:20:03.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:03.973+0000 7f75f8b07700 1 -- 172.21.15.67:0/70052171 shutdown_connections 2024-05-03T21:20:03.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:03.973+0000 7f75f8b07700 1 --2- 172.21.15.67:0/70052171 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f75f406bb40 0x7f75f4110330 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:20:03.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:03.973+0000 7f75f8b07700 1 --2- 172.21.15.67:0/70052171 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f75f4109510 0x7f75f41098f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:20:03.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:03.973+0000 7f75f8b07700 1 --2- 172.21.15.67:0/70052171 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f75f406b180 0x7f75f406b600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:20:03.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:03.973+0000 7f75f8b07700 1 -- 172.21.15.67:0/70052171 >> 172.21.15.67:0/70052171 conn(0x7f75f4079350 msgr2=0x7f75f4079760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T21:20:03.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:03.973+0000 7f75f8b07700 1 -- 172.21.15.67:0/70052171 shutdown_connections 2024-05-03T21:20:03.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:03.973+0000 7f75f8b07700 1 -- 172.21.15.67:0/70052171 wait complete. 2024-05-03T21:20:03.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:03.974+0000 7f75f8b07700 1 Processor -- start 2024-05-03T21:20:03.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:03.974+0000 7f75f8b07700 1 -- start start 2024-05-03T21:20:03.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:03.974+0000 7f75f8b07700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f75f406b180 0x7f75f419f880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:20:03.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:03.974+0000 7f75f8b07700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f75f406bb40 0x7f75f419fdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:20:03.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:03.974+0000 7f75f8b07700 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f75f4109510 0x7f75f4199a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:20:03.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:03.974+0000 7f75f8b07700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f75f41138d0 con 0x7f75f406bb40 2024-05-03T21:20:03.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:03.974+0000 7f75f8b07700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f75f4113750 con 0x7f75f406b180 2024-05-03T21:20:03.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:03.974+0000 7f75f8b07700 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f75f4113a50 con 0x7f75f4109510 2024-05-03T21:20:03.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:03.974+0000 7f75f1d9b700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f75f406bb40 0x7f75f419fdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:20:03.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:03.974+0000 7f75f1d9b700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f75f406bb40 0x7f75f419fdc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:36444/0 (socket says 172.21.15.67:36444) 2024-05-03T21:20:03.978 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:03.974+0000 7f75f1d9b700 1 -- 172.21.15.67:0/2331367607 learned_addr learned my addr 172.21.15.67:0/2331367607 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-03T21:20:03.978 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:03.974+0000 7f75f259c700 1 --2- 172.21.15.67:0/2331367607 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f75f406b180 0x7f75f419f880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:20:03.978 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:03.974+0000 7f75f2d9d700 1 --2- 172.21.15.67:0/2331367607 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f75f4109510 0x7f75f4199a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:20:03.978 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:03.975+0000 7f75f1d9b700 1 -- 172.21.15.67:0/2331367607 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f75f4109510 msgr2=0x7f75f4199a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:20:03.978 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:03.975+0000 7f75f1d9b700 1 --2- 172.21.15.67:0/2331367607 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f75f4109510 0x7f75f4199a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:20:03.978 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:03.975+0000 7f75f1d9b700 1 -- 172.21.15.67:0/2331367607 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f75f406b180 msgr2=0x7f75f419f880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:20:03.978 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:03.975+0000 7f75f1d9b700 1 --2- 172.21.15.67:0/2331367607 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f75f406b180 0x7f75f419f880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:20:03.978 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:03.975+0000 7f75f1d9b700 1 -- 172.21.15.67:0/2331367607 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f75f419a1f0 con 0x7f75f406bb40 2024-05-03T21:20:03.978 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:03.975+0000 7f75f2d9d700 1 --2- 172.21.15.67:0/2331367607 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f75f4109510 0x7f75f4199a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-03T21:20:03.978 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:03.975+0000 7f75f1d9b700 1 --2- 172.21.15.67:0/2331367607 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f75f406bb40 0x7f75f419fdc0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f75dc00d980 tx=0x7f75dc00dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T21:20:03.978 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:03.975+0000 7f75eb7fe700 1 -- 172.21.15.67:0/2331367607 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75dc017070 con 0x7f75f406bb40 2024-05-03T21:20:03.978 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:03.975+0000 7f75f8b07700 1 -- 172.21.15.67:0/2331367607 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f75f419a4e0 con 0x7f75f406bb40 2024-05-03T21:20:03.978 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:03.975+0000 7f75eb7fe700 1 -- 172.21.15.67:0/2331367607 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(28 keys) ==== 1170+0+0 (secure 0 0 0) 0x7f75dc005020 con 0x7f75f406bb40 2024-05-03T21:20:03.979 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:03.975+0000 7f75eb7fe700 1 -- 172.21.15.67:0/2331367607 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75dc009d20 con 0x7f75f406bb40 2024-05-03T21:20:03.979 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:03.975+0000 7f75f8b07700 1 -- 172.21.15.67:0/2331367607 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f75f41a6750 con 0x7f75f406bb40 2024-05-03T21:20:03.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:03.976+0000 7f75f8b07700 1 -- 172.21.15.67:0/2331367607 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f75f4063c30 con 0x7f75f406bb40 2024-05-03T21:20:03.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:03.977+0000 7f75eb7fe700 1 -- 172.21.15.67:0/2331367607 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 17) ==== 149703+0+0 (secure 0 0 0) 0x7f75dc0051d0 con 0x7f75f406bb40 2024-05-03T21:20:03.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:03.978+0000 7f75eb7fe700 1 --2- 172.21.15.67:0/2331367607 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f75e00b2910 0x7f75e00b4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:20:03.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:03.978+0000 7f75f259c700 1 --2- 172.21.15.67:0/2331367607 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f75e00b2910 0x7f75e00b4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:20:03.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:03.978+0000 7f75eb7fe700 1 -- 172.21.15.67:0/2331367607 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 8211+0+0 (secure 0 0 0) 0x7f75dc0e2820 con 0x7f75f406bb40 2024-05-03T21:20:03.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:03.978+0000 7f75eb7fe700 1 --2- 172.21.15.67:0/2331367607 >> [v2:172.21.15.138:6808/2521889499,v1:172.21.15.138:6809/2521889499] conn(0x7f75e00bc450 0x7f75e00be8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:20:03.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:03.978+0000 7f75eb7fe700 1 -- 172.21.15.67:0/2331367607 --> [v2:172.21.15.138:6808/2521889499,v1:172.21.15.138:6809/2521889499] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f75e00befc0 con 0x7f75e00bc450 2024-05-03T21:20:03.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:03.978+0000 7f75eb7fe700 1 -- 172.21.15.67:0/2331367607 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_get_version_reply(handle=1 version=84) ==== 24+0+0 (secure 0 0 0) 0x7f75dc0e2c70 con 0x7f75f406bb40 2024-05-03T21:20:03.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:03.978+0000 7f75f259c700 1 --2- 172.21.15.67:0/2331367607 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f75e00b2910 0x7f75e00b4dd0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f75f4100b60 tx=0x7f75ec0079e0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T21:20:03.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:03.978+0000 7f75f2d9d700 1 --2- 172.21.15.67:0/2331367607 >> [v2:172.21.15.138:6808/2521889499,v1:172.21.15.138:6809/2521889499] conn(0x7f75e00bc450 0x7f75e00be8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:20:03.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:03.979+0000 7f75f2d9d700 1 --2- 172.21.15.67:0/2331367607 >> [v2:172.21.15.138:6808/2521889499,v1:172.21.15.138:6809/2521889499] conn(0x7f75e00bc450 0x7f75e00be8f0 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T21:20:03.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:03.980+0000 7f75eb7fe700 1 -- 172.21.15.67:0/2331367607 <== osd.5 v2:172.21.15.138:6808/2521889499 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f75e00befc0 con 0x7f75e00bc450 2024-05-03T21:20:03.998 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:03.997+0000 7f75f8b07700 1 -- 172.21.15.67:0/2331367607 --> [v2:172.21.15.138:6808/2521889499,v1:172.21.15.138:6809/2521889499] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f75f40640b0 con 0x7f75e00bc450 2024-05-03T21:20:03.998 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:03.997+0000 7f75eb7fe700 1 -- 172.21.15.67:0/2331367607 <== osd.5 v2:172.21.15.138:6808/2521889499 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f75f40640b0 con 0x7f75e00bc450 2024-05-03T21:20:03.999 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:03.997+0000 7f75f8b07700 1 -- 172.21.15.67:0/2331367607 >> [v2:172.21.15.138:6808/2521889499,v1:172.21.15.138:6809/2521889499] conn(0x7f75e00bc450 msgr2=0x7f75e00be8f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:20:03.999 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:03.997+0000 7f75f8b07700 1 --2- 172.21.15.67:0/2331367607 >> [v2:172.21.15.138:6808/2521889499,v1:172.21.15.138:6809/2521889499] conn(0x7f75e00bc450 0x7f75e00be8f0 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:20:03.999 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:03.997+0000 7f75f8b07700 1 -- 172.21.15.67:0/2331367607 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f75e00b2910 msgr2=0x7f75e00b4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:20:03.999 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:03.997+0000 7f75f8b07700 1 --2- 172.21.15.67:0/2331367607 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f75e00b2910 0x7f75e00b4dd0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f75f4100b60 tx=0x7f75ec0079e0 comp rx=0 tx=0).stop 2024-05-03T21:20:03.999 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:03.997+0000 7f75f8b07700 1 -- 172.21.15.67:0/2331367607 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f75f406bb40 msgr2=0x7f75f419fdc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:20:03.999 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:03.997+0000 7f75f8b07700 1 --2- 172.21.15.67:0/2331367607 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f75f406bb40 0x7f75f419fdc0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f75dc00d980 tx=0x7f75dc00dc90 comp rx=0 tx=0).stop 2024-05-03T21:20:04.000 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:03.997+0000 7f75f2d9d700 1 -- 172.21.15.67:0/2331367607 reap_dead start 2024-05-03T21:20:04.000 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:03.998+0000 7f75f8b07700 1 -- 172.21.15.67:0/2331367607 shutdown_connections 2024-05-03T21:20:04.000 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:03.998+0000 7f75f8b07700 1 -- 172.21.15.67:0/2331367607 >> 172.21.15.67:0/2331367607 conn(0x7f75f4079350 msgr2=0x7f75f4100e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T21:20:04.000 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:03.998+0000 7f75f8b07700 1 -- 172.21.15.67:0/2331367607 shutdown_connections 2024-05-03T21:20:04.000 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:03.998+0000 7f75f8b07700 1 -- 172.21.15.67:0/2331367607 wait complete. 2024-05-03T21:20:04.587 INFO:tasks.cephadm.ceph_manager.ceph:need seq 34359738451 got 34359738455 for osd.0 2024-05-03T21:20:04.588 DEBUG:teuthology.parallel:result is None 2024-05-03T21:20:05.168 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:20:04 smithi067 ceph-mon[29730]: pgmap v329: 97 pgs: 97 active+clean; 632 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-05-03T21:20:05.205 INFO:teuthology.orchestra.run.smithi067.stdout:55834574930 2024-05-03T21:20:05.206 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9a047fe7599a549042d06a3fb5b6813a203eabea shell --fsid 713bc57a-0991-11ef-bc97-c7b262605968 -- ceph osd last-stat-seq osd.1 2024-05-03T21:20:05.245 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:20:04 smithi082 ceph-mon[32933]: pgmap v329: 97 pgs: 97 active+clean; 632 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-05-03T21:20:05.325 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:20:04 smithi138 ceph-mon[33110]: pgmap v329: 97 pgs: 97 active+clean; 632 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-05-03T21:20:05.763 INFO:teuthology.orchestra.run.smithi067.stdout:244813135920 2024-05-03T21:20:05.764 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9a047fe7599a549042d06a3fb5b6813a203eabea shell --fsid 713bc57a-0991-11ef-bc97-c7b262605968 -- ceph osd last-stat-seq osd.8 2024-05-03T21:20:06.378 INFO:teuthology.orchestra.run.smithi067.stdout:167503724607 2024-05-03T21:20:06.379 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9a047fe7599a549042d06a3fb5b6813a203eabea shell --fsid 713bc57a-0991-11ef-bc97-c7b262605968 -- ceph osd last-stat-seq osd.5 2024-05-03T21:20:06.938 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:06.936+0000 7fc0858ab700 1 -- 172.21.15.67:0/2620748177 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc08010b940 msgr2=0x7fc08010efc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:20:06.938 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:06.936+0000 7fc0858ab700 1 --2- 172.21.15.67:0/2620748177 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc08010b940 0x7fc08010efc0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7fc0740099c0 tx=0x7fc074009cd0 comp rx=0 tx=0).stop 2024-05-03T21:20:06.938 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:06.937+0000 7fc0858ab700 1 -- 172.21.15.67:0/2620748177 shutdown_connections 2024-05-03T21:20:06.938 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:06.937+0000 7fc0858ab700 1 --2- 172.21.15.67:0/2620748177 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fc08010f6a0 0x7fc0801172b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:20:06.939 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:06.937+0000 7fc0858ab700 1 --2- 172.21.15.67:0/2620748177 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fc08010af90 0x7fc08010b370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:20:06.939 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:06.937+0000 7fc0858ab700 1 --2- 172.21.15.67:0/2620748177 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc08010b940 0x7fc08010efc0 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:20:06.939 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:06.937+0000 7fc0858ab700 1 -- 172.21.15.67:0/2620748177 >> 172.21.15.67:0/2620748177 conn(0x7fc08006e610 msgr2=0x7fc08006ea20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T21:20:06.939 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:06.937+0000 7fc0858ab700 1 -- 172.21.15.67:0/2620748177 shutdown_connections 2024-05-03T21:20:06.939 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:06.937+0000 7fc0858ab700 1 -- 172.21.15.67:0/2620748177 wait complete. 2024-05-03T21:20:06.939 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:06.938+0000 7fc0858ab700 1 Processor -- start 2024-05-03T21:20:06.939 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:06.938+0000 7fc0858ab700 1 -- start start 2024-05-03T21:20:06.942 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:06.939+0000 7fc0858ab700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc08010af90 0x7fc0801aa430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:20:06.942 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:06.939+0000 7fc0858ab700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fc08010b940 0x7fc0801aa970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:20:06.942 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:06.939+0000 7fc0858ab700 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fc08010f6a0 0x7fc0801a45f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:20:06.942 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:06.939+0000 7fc07effd700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc08010af90 0x7fc0801aa430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:20:06.942 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:06.939+0000 7fc07effd700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc08010af90 0x7fc0801aa430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:50712/0 (socket says 172.21.15.67:50712) 2024-05-03T21:20:06.942 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:06.939+0000 7fc0858ab700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fc080119f50 con 0x7fc08010af90 2024-05-03T21:20:06.942 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:06.939+0000 7fc07effd700 1 -- 172.21.15.67:0/1399535887 learned_addr learned my addr 172.21.15.67:0/1399535887 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-03T21:20:06.943 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:06.939+0000 7fc07e7fc700 1 --2- 172.21.15.67:0/1399535887 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fc08010b940 0x7fc0801aa970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:20:06.943 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:06.939+0000 7fc0858ab700 1 -- 172.21.15.67:0/1399535887 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7fc080119dd0 con 0x7fc08010b940 2024-05-03T21:20:06.943 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:06.940+0000 7fc0858ab700 1 -- 172.21.15.67:0/1399535887 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7fc08011a0d0 con 0x7fc08010f6a0 2024-05-03T21:20:06.943 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:06.940+0000 7fc07f7fe700 1 --2- 172.21.15.67:0/1399535887 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fc08010f6a0 0x7fc0801a45f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:20:06.943 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:06.940+0000 7fc07effd700 1 -- 172.21.15.67:0/1399535887 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fc08010f6a0 msgr2=0x7fc0801a45f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:20:06.943 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:06.940+0000 7fc07effd700 1 --2- 172.21.15.67:0/1399535887 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fc08010f6a0 0x7fc0801a45f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:20:06.943 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:06.940+0000 7fc07effd700 1 -- 172.21.15.67:0/1399535887 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fc08010b940 msgr2=0x7fc0801aa970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:20:06.943 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:06.940+0000 7fc07effd700 1 --2- 172.21.15.67:0/1399535887 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fc08010b940 0x7fc0801aa970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:20:06.943 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:06.940+0000 7fc07effd700 1 -- 172.21.15.67:0/1399535887 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc0801a4d50 con 0x7fc08010af90 2024-05-03T21:20:06.943 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:06.940+0000 7fc07e7fc700 1 --2- 172.21.15.67:0/1399535887 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fc08010b940 0x7fc0801aa970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-03T21:20:06.943 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:06.940+0000 7fc07f7fe700 1 --2- 172.21.15.67:0/1399535887 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fc08010f6a0 0x7fc0801a45f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-03T21:20:06.943 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:06.940+0000 7fc07effd700 1 --2- 172.21.15.67:0/1399535887 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc08010af90 0x7fc0801aa430 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7fc07000d950 tx=0x7fc07000dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T21:20:06.944 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:06.941+0000 7fc0848a9700 1 -- 172.21.15.67:0/1399535887 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc070016070 con 0x7fc08010af90 2024-05-03T21:20:06.944 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:06.941+0000 7fc0858ab700 1 -- 172.21.15.67:0/1399535887 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc0801a5040 con 0x7fc08010af90 2024-05-03T21:20:06.944 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:06.941+0000 7fc0858ab700 1 -- 172.21.15.67:0/1399535887 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc0801b1300 con 0x7fc08010af90 2024-05-03T21:20:06.944 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:06.942+0000 7fc0848a9700 1 -- 172.21.15.67:0/1399535887 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(28 keys) ==== 1170+0+0 (secure 0 0 0) 0x7fc07000fd90 con 0x7fc08010af90 2024-05-03T21:20:06.944 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:06.942+0000 7fc0848a9700 1 -- 172.21.15.67:0/1399535887 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc07000e610 con 0x7fc08010af90 2024-05-03T21:20:06.944 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:06.942+0000 7fc0858ab700 1 -- 172.21.15.67:0/1399535887 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc08006a190 con 0x7fc08010af90 2024-05-03T21:20:06.945 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:06.944+0000 7fc0848a9700 1 -- 172.21.15.67:0/1399535887 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 17) ==== 149703+0+0 (secure 0 0 0) 0x7fc070019070 con 0x7fc08010af90 2024-05-03T21:20:06.946 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:06.945+0000 7fc0848a9700 1 --2- 172.21.15.67:0/1399535887 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7fc0680b2910 0x7fc0680b4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:20:06.946 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:06.945+0000 7fc07e7fc700 1 --2- 172.21.15.67:0/1399535887 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7fc0680b2910 0x7fc0680b4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:20:06.946 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:06.945+0000 7fc0848a9700 1 -- 172.21.15.67:0/1399535887 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 8211+0+0 (secure 0 0 0) 0x7fc0700e2340 con 0x7fc08010af90 2024-05-03T21:20:06.947 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:06.946+0000 7fc07e7fc700 1 --2- 172.21.15.67:0/1399535887 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7fc0680b2910 0x7fc0680b4dd0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fc074000c00 tx=0x7fc074005fb0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T21:20:06.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:06.949+0000 7fc0848a9700 1 -- 172.21.15.67:0/1399535887 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191839 (secure 0 0 0) 0x7fc0700a9f10 con 0x7fc08010af90 2024-05-03T21:20:06.971 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:20:06 smithi067 ceph-mon[29730]: pgmap v330: 97 pgs: 97 active+clean; 632 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:20:07.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:07.169+0000 7fc0858ab700 1 -- 172.21.15.67:0/1399535887 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 9} v 0) -- 0x7fc0801a5b20 con 0x7fc08010af90 2024-05-03T21:20:07.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:07.170+0000 7fc0848a9700 1 -- 172.21.15.67:0/1399535887 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 9}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7fc0700a9810 con 0x7fc08010af90 2024-05-03T21:20:07.171 INFO:teuthology.orchestra.run.smithi067.stdout:270582939690 2024-05-03T21:20:07.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:07.172+0000 7fc0858ab700 1 -- 172.21.15.67:0/1399535887 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7fc0680b2910 msgr2=0x7fc0680b4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:20:07.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:07.172+0000 7fc0858ab700 1 --2- 172.21.15.67:0/1399535887 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7fc0680b2910 0x7fc0680b4dd0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fc074000c00 tx=0x7fc074005fb0 comp rx=0 tx=0).stop 2024-05-03T21:20:07.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:07.172+0000 7fc0858ab700 1 -- 172.21.15.67:0/1399535887 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc08010af90 msgr2=0x7fc0801aa430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:20:07.174 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:07.172+0000 7fc0858ab700 1 --2- 172.21.15.67:0/1399535887 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc08010af90 0x7fc0801aa430 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7fc07000d950 tx=0x7fc07000dc60 comp rx=0 tx=0).stop 2024-05-03T21:20:07.174 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:07.172+0000 7fc0858ab700 1 -- 172.21.15.67:0/1399535887 shutdown_connections 2024-05-03T21:20:07.174 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:07.172+0000 7fc0858ab700 1 --2- 172.21.15.67:0/1399535887 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7fc0680b2910 0x7fc0680b4dd0 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:20:07.174 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:07.172+0000 7fc0858ab700 1 --2- 172.21.15.67:0/1399535887 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc08010af90 0x7fc0801aa430 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:20:07.174 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:07.172+0000 7fc0858ab700 1 --2- 172.21.15.67:0/1399535887 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fc08010f6a0 0x7fc0801a45f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:20:07.174 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:07.172+0000 7fc0858ab700 1 --2- 172.21.15.67:0/1399535887 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fc08010b940 0x7fc0801aa970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:20:07.174 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:07.172+0000 7fc0858ab700 1 -- 172.21.15.67:0/1399535887 >> 172.21.15.67:0/1399535887 conn(0x7fc08006e610 msgr2=0x7fc080115bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T21:20:07.174 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:07.172+0000 7fc0858ab700 1 -- 172.21.15.67:0/1399535887 shutdown_connections 2024-05-03T21:20:07.174 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:07.172+0000 7fc0858ab700 1 -- 172.21.15.67:0/1399535887 wait complete. 2024-05-03T21:20:07.245 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:20:06 smithi082 ceph-mon[32933]: pgmap v330: 97 pgs: 97 active+clean; 632 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:20:07.324 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:20:06 smithi138 ceph-mon[33110]: pgmap v330: 97 pgs: 97 active+clean; 632 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:20:08.000 INFO:tasks.cephadm.ceph_manager.ceph:need seq 270582939687 got 270582939690 for osd.9 2024-05-03T21:20:08.000 DEBUG:teuthology.parallel:result is None 2024-05-03T21:20:08.221 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:20:07 smithi067 ceph-mon[29730]: from='client.? 172.21.15.67:0/1399535887' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-05-03T21:20:08.245 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:20:07 smithi082 ceph-mon[32933]: from='client.? 172.21.15.67:0/1399535887' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-05-03T21:20:08.324 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:20:07 smithi138 ceph-mon[33110]: from='client.? 172.21.15.67:0/1399535887' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-05-03T21:20:08.480 INFO:teuthology.orchestra.run.smithi067.stdout:292057776164 2024-05-03T21:20:08.480 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9a047fe7599a549042d06a3fb5b6813a203eabea shell --fsid 713bc57a-0991-11ef-bc97-c7b262605968 -- ceph osd last-stat-seq osd.10 2024-05-03T21:20:09.074 INFO:teuthology.orchestra.run.smithi067.stdout:317827579935 2024-05-03T21:20:09.075 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9a047fe7599a549042d06a3fb5b6813a203eabea shell --fsid 713bc57a-0991-11ef-bc97-c7b262605968 -- ceph osd last-stat-seq osd.11 2024-05-03T21:20:09.221 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:20:08 smithi067 ceph-mon[29730]: pgmap v331: 97 pgs: 97 active+clean; 632 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:20:09.245 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:20:08 smithi082 ceph-mon[32933]: pgmap v331: 97 pgs: 97 active+clean; 632 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:20:09.324 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:20:08 smithi138 ceph-mon[33110]: pgmap v331: 97 pgs: 97 active+clean; 632 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:20:09.613 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:09.610+0000 7fce75423700 1 -- 172.21.15.67:0/545427244 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fce70102690 msgr2=0x7fce70102a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:20:09.613 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:09.610+0000 7fce75423700 1 --2- 172.21.15.67:0/545427244 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fce70102690 0x7fce70102a70 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7fce6800b0d0 tx=0x7fce6800b3e0 comp rx=0 tx=0).stop 2024-05-03T21:20:09.613 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:09.610+0000 7fce75423700 1 -- 172.21.15.67:0/545427244 shutdown_connections 2024-05-03T21:20:09.613 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:09.610+0000 7fce75423700 1 --2- 172.21.15.67:0/545427244 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fce70102690 0x7fce70102a70 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:20:09.613 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:09.610+0000 7fce75423700 1 --2- 172.21.15.67:0/545427244 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fce701076e0 0x7fce7010f2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:20:09.613 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:09.610+0000 7fce75423700 1 --2- 172.21.15.67:0/545427244 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fce70103040 0x7fce70107000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:20:09.613 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:09.610+0000 7fce75423700 1 -- 172.21.15.67:0/545427244 >> 172.21.15.67:0/545427244 conn(0x7fce700783f0 msgr2=0x7fce70078800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T21:20:09.614 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:09.610+0000 7fce75423700 1 -- 172.21.15.67:0/545427244 shutdown_connections 2024-05-03T21:20:09.614 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:09.610+0000 7fce75423700 1 -- 172.21.15.67:0/545427244 wait complete. 2024-05-03T21:20:09.614 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:09.611+0000 7fce75423700 1 Processor -- start 2024-05-03T21:20:09.614 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:09.611+0000 7fce75423700 1 -- start start 2024-05-03T21:20:09.615 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:09.611+0000 7fce75423700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fce70102690 0x7fce7019f8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:20:09.615 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:09.611+0000 7fce75423700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fce70103040 0x7fce7019fe30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:20:09.616 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:09.611+0000 7fce75423700 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fce701076e0 0x7fce70199b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:20:09.616 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:09.611+0000 7fce75423700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fce70112850 con 0x7fce70102690 2024-05-03T21:20:09.616 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:09.611+0000 7fce75423700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7fce701126d0 con 0x7fce70103040 2024-05-03T21:20:09.616 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:09.611+0000 7fce75423700 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7fce701129d0 con 0x7fce701076e0 2024-05-03T21:20:09.616 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:09.611+0000 7fce6effd700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fce70102690 0x7fce7019f8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:20:09.616 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:09.612+0000 7fce6effd700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fce70102690 0x7fce7019f8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:50716/0 (socket says 172.21.15.67:50716) 2024-05-03T21:20:09.616 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:09.612+0000 7fce6effd700 1 -- 172.21.15.67:0/4086526244 learned_addr learned my addr 172.21.15.67:0/4086526244 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-03T21:20:09.616 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:09.612+0000 7fce6e7fc700 1 --2- 172.21.15.67:0/4086526244 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fce70103040 0x7fce7019fe30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:20:09.616 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:09.612+0000 7fce6f7fe700 1 --2- 172.21.15.67:0/4086526244 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fce701076e0 0x7fce70199b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:20:09.616 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:09.612+0000 7fce6effd700 1 -- 172.21.15.67:0/4086526244 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fce701076e0 msgr2=0x7fce70199b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:20:09.616 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:09.612+0000 7fce6effd700 1 --2- 172.21.15.67:0/4086526244 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fce701076e0 0x7fce70199b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:20:09.616 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:09.612+0000 7fce6effd700 1 -- 172.21.15.67:0/4086526244 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fce70103040 msgr2=0x7fce7019fe30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:20:09.616 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:09.612+0000 7fce6effd700 1 --2- 172.21.15.67:0/4086526244 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fce70103040 0x7fce7019fe30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:20:09.616 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:09.612+0000 7fce6effd700 1 -- 172.21.15.67:0/4086526244 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fce7019a260 con 0x7fce70102690 2024-05-03T21:20:09.617 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:09.612+0000 7fce6f7fe700 1 --2- 172.21.15.67:0/4086526244 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fce701076e0 0x7fce70199b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-03T21:20:09.617 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:09.612+0000 7fce6effd700 1 --2- 172.21.15.67:0/4086526244 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fce70102690 0x7fce7019f8f0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7fce6800d010 tx=0x7fce68007a20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T21:20:09.617 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:09.612+0000 7fce67fff700 1 -- 172.21.15.67:0/4086526244 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce68021070 con 0x7fce70102690 2024-05-03T21:20:09.617 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:09.612+0000 7fce75423700 1 -- 172.21.15.67:0/4086526244 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fce7019a4f0 con 0x7fce70102690 2024-05-03T21:20:09.617 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:09.612+0000 7fce67fff700 1 -- 172.21.15.67:0/4086526244 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(28 keys) ==== 1170+0+0 (secure 0 0 0) 0x7fce6801db50 con 0x7fce70102690 2024-05-03T21:20:09.617 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:09.612+0000 7fce67fff700 1 -- 172.21.15.67:0/4086526244 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce6801bd30 con 0x7fce70102690 2024-05-03T21:20:09.617 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:09.612+0000 7fce75423700 1 -- 172.21.15.67:0/4086526244 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fce701a67c0 con 0x7fce70102690 2024-05-03T21:20:09.617 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:09.613+0000 7fce75423700 1 -- 172.21.15.67:0/4086526244 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fce70063c30 con 0x7fce70102690 2024-05-03T21:20:09.618 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:09.615+0000 7fce67fff700 1 -- 172.21.15.67:0/4086526244 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 17) ==== 149703+0+0 (secure 0 0 0) 0x7fce68024070 con 0x7fce70102690 2024-05-03T21:20:09.618 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:09.615+0000 7fce67fff700 1 --2- 172.21.15.67:0/4086526244 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7fce5c0b2910 0x7fce5c0b4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:20:09.618 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:09.615+0000 7fce6e7fc700 1 --2- 172.21.15.67:0/4086526244 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7fce5c0b2910 0x7fce5c0b4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:20:09.618 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:09.615+0000 7fce67fff700 1 -- 172.21.15.67:0/4086526244 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 8211+0+0 (secure 0 0 0) 0x7fce6800f020 con 0x7fce70102690 2024-05-03T21:20:09.618 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:09.616+0000 7fce67fff700 1 --2- 172.21.15.67:0/4086526244 >> [v2:172.21.15.67:6818/3269682731,v1:172.21.15.67:6819/3269682731] conn(0x7fce5c0bc450 0x7fce5c0be8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:20:09.618 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:09.616+0000 7fce67fff700 1 -- 172.21.15.67:0/4086526244 --> [v2:172.21.15.67:6818/3269682731,v1:172.21.15.67:6819/3269682731] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fce6802ae70 con 0x7fce5c0bc450 2024-05-03T21:20:09.619 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:09.616+0000 7fce67fff700 1 -- 172.21.15.67:0/4086526244 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_get_version_reply(handle=1 version=84) ==== 24+0+0 (secure 0 0 0) 0x7fce680e44d0 con 0x7fce70102690 2024-05-03T21:20:09.619 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:09.616+0000 7fce6f7fe700 1 --2- 172.21.15.67:0/4086526244 >> [v2:172.21.15.67:6818/3269682731,v1:172.21.15.67:6819/3269682731] conn(0x7fce5c0bc450 0x7fce5c0be8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:20:09.619 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:09.616+0000 7fce6e7fc700 1 --2- 172.21.15.67:0/4086526244 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7fce5c0b2910 0x7fce5c0b4dd0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fce701089d0 tx=0x7fce5800a380 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T21:20:09.619 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:09.616+0000 7fce6f7fe700 1 --2- 172.21.15.67:0/4086526244 >> [v2:172.21.15.67:6818/3269682731,v1:172.21.15.67:6819/3269682731] conn(0x7fce5c0bc450 0x7fce5c0be8f0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T21:20:09.619 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:09.617+0000 7fce67fff700 1 -- 172.21.15.67:0/4086526244 <== osd.6 v2:172.21.15.67:6818/3269682731 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fce6802ae70 con 0x7fce5c0bc450 2024-05-03T21:20:09.635 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:09.633+0000 7fce75423700 1 -- 172.21.15.67:0/4086526244 --> [v2:172.21.15.67:6818/3269682731,v1:172.21.15.67:6819/3269682731] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fce700640b0 con 0x7fce5c0bc450 2024-05-03T21:20:09.636 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:09.634+0000 7fce67fff700 1 -- 172.21.15.67:0/4086526244 <== osd.6 v2:172.21.15.67:6818/3269682731 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7fce700640b0 con 0x7fce5c0bc450 2024-05-03T21:20:09.636 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:09.634+0000 7fce75423700 1 -- 172.21.15.67:0/4086526244 >> [v2:172.21.15.67:6818/3269682731,v1:172.21.15.67:6819/3269682731] conn(0x7fce5c0bc450 msgr2=0x7fce5c0be8f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:20:09.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:09.634+0000 7fce75423700 1 --2- 172.21.15.67:0/4086526244 >> [v2:172.21.15.67:6818/3269682731,v1:172.21.15.67:6819/3269682731] conn(0x7fce5c0bc450 0x7fce5c0be8f0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:20:09.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:09.634+0000 7fce75423700 1 -- 172.21.15.67:0/4086526244 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7fce5c0b2910 msgr2=0x7fce5c0b4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:20:09.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:09.634+0000 7fce75423700 1 --2- 172.21.15.67:0/4086526244 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7fce5c0b2910 0x7fce5c0b4dd0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fce701089d0 tx=0x7fce5800a380 comp rx=0 tx=0).stop 2024-05-03T21:20:09.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:09.634+0000 7fce75423700 1 -- 172.21.15.67:0/4086526244 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fce70102690 msgr2=0x7fce7019f8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:20:09.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:09.634+0000 7fce75423700 1 --2- 172.21.15.67:0/4086526244 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fce70102690 0x7fce7019f8f0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7fce6800d010 tx=0x7fce68007a20 comp rx=0 tx=0).stop 2024-05-03T21:20:09.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:09.634+0000 7fce6f7fe700 1 -- 172.21.15.67:0/4086526244 reap_dead start 2024-05-03T21:20:09.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:09.634+0000 7fce75423700 1 -- 172.21.15.67:0/4086526244 shutdown_connections 2024-05-03T21:20:09.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:09.634+0000 7fce75423700 1 -- 172.21.15.67:0/4086526244 >> 172.21.15.67:0/4086526244 conn(0x7fce700783f0 msgr2=0x7fce7006bdf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T21:20:09.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:09.634+0000 7fce75423700 1 -- 172.21.15.67:0/4086526244 shutdown_connections 2024-05-03T21:20:09.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:09.634+0000 7fce75423700 1 -- 172.21.15.67:0/4086526244 wait complete. 2024-05-03T21:20:10.057 INFO:teuthology.orchestra.run.smithi067.stdout:141733920834 2024-05-03T21:20:10.057 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9a047fe7599a549042d06a3fb5b6813a203eabea shell --fsid 713bc57a-0991-11ef-bc97-c7b262605968 -- ceph osd last-stat-seq osd.4 2024-05-03T21:20:10.279 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:10.257+0000 7fb1ab5b4700 1 -- 172.21.15.67:0/1547994741 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb1a4108410 msgr2=0x7fb1a41087f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:20:10.279 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:10.257+0000 7fb1ab5b4700 1 --2- 172.21.15.67:0/1547994741 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb1a4108410 0x7fb1a41087f0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7fb19400b0a0 tx=0x7fb19400b3b0 comp rx=0 tx=0).stop 2024-05-03T21:20:10.280 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:10.257+0000 7fb1ab5b4700 1 -- 172.21.15.67:0/1547994741 shutdown_connections 2024-05-03T21:20:10.280 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:10.257+0000 7fb1ab5b4700 1 --2- 172.21.15.67:0/1547994741 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb1a4108410 0x7fb1a41087f0 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:20:10.280 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:10.257+0000 7fb1ab5b4700 1 --2- 172.21.15.67:0/1547994741 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fb1a4102d60 0x7fb1a410f250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:20:10.280 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:10.257+0000 7fb1ab5b4700 1 --2- 172.21.15.67:0/1547994741 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fb1a41023a0 0x7fb1a4102820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:20:10.280 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:10.257+0000 7fb1ab5b4700 1 -- 172.21.15.67:0/1547994741 >> 172.21.15.67:0/1547994741 conn(0x7fb1a4079350 msgr2=0x7fb1a4079760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T21:20:10.280 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:10.257+0000 7fb1ab5b4700 1 -- 172.21.15.67:0/1547994741 shutdown_connections 2024-05-03T21:20:10.280 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:10.257+0000 7fb1ab5b4700 1 -- 172.21.15.67:0/1547994741 wait complete. 2024-05-03T21:20:10.280 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:10.258+0000 7fb1ab5b4700 1 Processor -- start 2024-05-03T21:20:10.280 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:10.258+0000 7fb1ab5b4700 1 -- start start 2024-05-03T21:20:10.280 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:10.258+0000 7fb1ab5b4700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fb1a41023a0 0x7fb1a41043f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:20:10.281 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:10.258+0000 7fb1ab5b4700 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fb1a4102d60 0x7fb1a4104930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:20:10.281 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:10.258+0000 7fb1ab5b4700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb1a4108410 0x7fb1a4104e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:20:10.281 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:10.258+0000 7fb1ab5b4700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fb1a4112910 con 0x7fb1a4108410 2024-05-03T21:20:10.281 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:10.258+0000 7fb1ab5b4700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7fb1a4112790 con 0x7fb1a41023a0 2024-05-03T21:20:10.281 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:10.258+0000 7fb1ab5b4700 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7fb1a4112a90 con 0x7fb1a4102d60 2024-05-03T21:20:10.281 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:10.258+0000 7fb1a9b51700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb1a4108410 0x7fb1a4104e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:20:10.281 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:10.258+0000 7fb1a9b51700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb1a4108410 0x7fb1a4104e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:50734/0 (socket says 172.21.15.67:50734) 2024-05-03T21:20:10.281 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:10.258+0000 7fb1a9b51700 1 -- 172.21.15.67:0/1016766490 learned_addr learned my addr 172.21.15.67:0/1016766490 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-03T21:20:10.281 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:10.258+0000 7fb1a9350700 1 --2- 172.21.15.67:0/1016766490 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fb1a41023a0 0x7fb1a41043f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:20:10.281 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:10.258+0000 7fb1a8b4f700 1 --2- 172.21.15.67:0/1016766490 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fb1a4102d60 0x7fb1a4104930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:20:10.281 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:10.259+0000 7fb1a9b51700 1 -- 172.21.15.67:0/1016766490 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fb1a4102d60 msgr2=0x7fb1a4104930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:20:10.281 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:10.259+0000 7fb1a9b51700 1 --2- 172.21.15.67:0/1016766490 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fb1a4102d60 0x7fb1a4104930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:20:10.281 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:10.259+0000 7fb1a9b51700 1 -- 172.21.15.67:0/1016766490 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fb1a41023a0 msgr2=0x7fb1a41043f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:20:10.282 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:10.259+0000 7fb1a9b51700 1 --2- 172.21.15.67:0/1016766490 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fb1a41023a0 0x7fb1a41043f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:20:10.282 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:10.259+0000 7fb1a9b51700 1 -- 172.21.15.67:0/1016766490 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb1a41a84a0 con 0x7fb1a4108410 2024-05-03T21:20:10.282 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:10.259+0000 7fb1a9b51700 1 --2- 172.21.15.67:0/1016766490 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb1a4108410 0x7fb1a4104e70 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7fb19c00b550 tx=0x7fb19c00b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T21:20:10.282 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:10.259+0000 7fb1a27fc700 1 -- 172.21.15.67:0/1016766490 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb19c015020 con 0x7fb1a4108410 2024-05-03T21:20:10.282 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:10.259+0000 7fb1ab5b4700 1 -- 172.21.15.67:0/1016766490 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb1a41a8790 con 0x7fb1a4108410 2024-05-03T21:20:10.282 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:10.259+0000 7fb1a27fc700 1 -- 172.21.15.67:0/1016766490 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(28 keys) ==== 1170+0+0 (secure 0 0 0) 0x7fb19c011d30 con 0x7fb1a4108410 2024-05-03T21:20:10.282 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:10.259+0000 7fb1a27fc700 1 -- 172.21.15.67:0/1016766490 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb19c010e00 con 0x7fb1a4108410 2024-05-03T21:20:10.282 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:10.259+0000 7fb1ab5b4700 1 -- 172.21.15.67:0/1016766490 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb1a41a8d20 con 0x7fb1a4108410 2024-05-03T21:20:10.282 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:10.260+0000 7fb1ab5b4700 1 -- 172.21.15.67:0/1016766490 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fb1a41010f0 con 0x7fb1a4108410 2024-05-03T21:20:10.282 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:10.261+0000 7fb1a27fc700 1 -- 172.21.15.67:0/1016766490 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 17) ==== 149703+0+0 (secure 0 0 0) 0x7fb19c018020 con 0x7fb1a4108410 2024-05-03T21:20:10.282 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:10.262+0000 7fb1a27fc700 1 --2- 172.21.15.67:0/1016766490 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7fb18c0b2910 0x7fb18c0b4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:20:10.282 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:10.262+0000 7fb1a9350700 1 --2- 172.21.15.67:0/1016766490 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7fb18c0b2910 0x7fb18c0b4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:20:10.282 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:10.262+0000 7fb1a27fc700 1 -- 172.21.15.67:0/1016766490 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 8211+0+0 (secure 0 0 0) 0x7fb19c0e1ed0 con 0x7fb1a4108410 2024-05-03T21:20:10.283 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:10.262+0000 7fb1a27fc700 1 --2- 172.21.15.67:0/1016766490 >> [v2:172.21.15.82:6816/3008589976,v1:172.21.15.82:6817/3008589976] conn(0x7fb18c0bc4b0 0x7fb18c0be950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:20:10.283 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:10.262+0000 7fb1a27fc700 1 -- 172.21.15.67:0/1016766490 --> [v2:172.21.15.82:6816/3008589976,v1:172.21.15.82:6817/3008589976] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fb18c0bf020 con 0x7fb18c0bc4b0 2024-05-03T21:20:10.283 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:10.262+0000 7fb1a27fc700 1 -- 172.21.15.67:0/1016766490 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_get_version_reply(handle=1 version=84) ==== 24+0+0 (secure 0 0 0) 0x7fb19c0e2320 con 0x7fb1a4108410 2024-05-03T21:20:10.283 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:10.263+0000 7fb1a9350700 1 --2- 172.21.15.67:0/1016766490 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7fb18c0b2910 0x7fb18c0b4dd0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fb194009fd0 tx=0x7fb19400b7f0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T21:20:10.283 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:10.263+0000 7fb1a8b4f700 1 --2- 172.21.15.67:0/1016766490 >> [v2:172.21.15.82:6816/3008589976,v1:172.21.15.82:6817/3008589976] conn(0x7fb18c0bc4b0 0x7fb18c0be950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:20:10.283 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:10.263+0000 7fb1a8b4f700 1 --2- 172.21.15.67:0/1016766490 >> [v2:172.21.15.82:6816/3008589976,v1:172.21.15.82:6817/3008589976] conn(0x7fb18c0bc4b0 0x7fb18c0be950 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T21:20:10.283 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:10.264+0000 7fb1a27fc700 1 -- 172.21.15.67:0/1016766490 <== osd.7 v2:172.21.15.82:6816/3008589976 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fb18c0bf020 con 0x7fb18c0bc4b0 2024-05-03T21:20:10.283 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:10.281+0000 7fb1ab5b4700 1 -- 172.21.15.67:0/1016766490 --> [v2:172.21.15.82:6816/3008589976,v1:172.21.15.82:6817/3008589976] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fb1a4103ac0 con 0x7fb18c0bc4b0 2024-05-03T21:20:10.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:10.282+0000 7fb1a27fc700 1 -- 172.21.15.67:0/1016766490 <== osd.7 v2:172.21.15.82:6816/3008589976 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7fb1a4103ac0 con 0x7fb18c0bc4b0 2024-05-03T21:20:10.285 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:10.282+0000 7fb1ab5b4700 1 -- 172.21.15.67:0/1016766490 >> [v2:172.21.15.82:6816/3008589976,v1:172.21.15.82:6817/3008589976] conn(0x7fb18c0bc4b0 msgr2=0x7fb18c0be950 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:20:10.285 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:10.282+0000 7fb1ab5b4700 1 --2- 172.21.15.67:0/1016766490 >> [v2:172.21.15.82:6816/3008589976,v1:172.21.15.82:6817/3008589976] conn(0x7fb18c0bc4b0 0x7fb18c0be950 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:20:10.285 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:10.282+0000 7fb1ab5b4700 1 -- 172.21.15.67:0/1016766490 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7fb18c0b2910 msgr2=0x7fb18c0b4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:20:10.285 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:10.282+0000 7fb1ab5b4700 1 --2- 172.21.15.67:0/1016766490 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7fb18c0b2910 0x7fb18c0b4dd0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fb194009fd0 tx=0x7fb19400b7f0 comp rx=0 tx=0).stop 2024-05-03T21:20:10.285 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:10.282+0000 7fb1ab5b4700 1 -- 172.21.15.67:0/1016766490 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb1a4108410 msgr2=0x7fb1a4104e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:20:10.286 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:10.282+0000 7fb1ab5b4700 1 --2- 172.21.15.67:0/1016766490 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb1a4108410 0x7fb1a4104e70 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7fb19c00b550 tx=0x7fb19c00b860 comp rx=0 tx=0).stop 2024-05-03T21:20:10.286 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:10.282+0000 7fb1a9b51700 1 -- 172.21.15.67:0/1016766490 reap_dead start 2024-05-03T21:20:10.286 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:10.282+0000 7fb1ab5b4700 1 -- 172.21.15.67:0/1016766490 shutdown_connections 2024-05-03T21:20:10.286 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:10.282+0000 7fb1ab5b4700 1 -- 172.21.15.67:0/1016766490 >> 172.21.15.67:0/1016766490 conn(0x7fb1a4079350 msgr2=0x7fb1a4106770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T21:20:10.286 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:10.282+0000 7fb1ab5b4700 1 -- 172.21.15.67:0/1016766490 shutdown_connections 2024-05-03T21:20:10.286 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:10.282+0000 7fb1ab5b4700 1 -- 172.21.15.67:0/1016766490 wait complete. 2024-05-03T21:20:10.989 INFO:teuthology.orchestra.run.smithi067.stdout:214748364853 2024-05-03T21:20:10.989 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9a047fe7599a549042d06a3fb5b6813a203eabea shell --fsid 713bc57a-0991-11ef-bc97-c7b262605968 -- ceph osd last-stat-seq osd.7 2024-05-03T21:20:11.182 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:20:10 smithi067 ceph-mon[29730]: pgmap v332: 97 pgs: 97 active+clean; 632 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-05-03T21:20:11.245 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:20:10 smithi082 ceph-mon[32933]: pgmap v332: 97 pgs: 97 active+clean; 632 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-05-03T21:20:11.324 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:20:10 smithi138 ceph-mon[33110]: pgmap v332: 97 pgs: 97 active+clean; 632 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-05-03T21:20:11.859 INFO:teuthology.orchestra.run.smithi067.stdout:193273528378 2024-05-03T21:20:11.859 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9a047fe7599a549042d06a3fb5b6813a203eabea shell --fsid 713bc57a-0991-11ef-bc97-c7b262605968 -- ceph osd last-stat-seq osd.6 2024-05-03T21:20:11.913 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/713bc57a-0991-11ef-bc97-c7b262605968/mon.a/config 2024-05-03T21:20:12.015 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:11.180+0000 7fcc00fc8700 1 -- 172.21.15.67:0/3465270616 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcbfc102d50 msgr2=0x7fcbfc10f240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:20:12.015 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:11.180+0000 7fcc00fc8700 1 --2- 172.21.15.67:0/3465270616 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcbfc102d50 0x7fcbfc10f240 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7fcbf400b0a0 tx=0x7fcbf400b3b0 comp rx=0 tx=0).stop 2024-05-03T21:20:12.015 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:11.180+0000 7fcc00fc8700 1 -- 172.21.15.67:0/3465270616 shutdown_connections 2024-05-03T21:20:12.015 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:11.180+0000 7fcc00fc8700 1 --2- 172.21.15.67:0/3465270616 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcbfc102d50 0x7fcbfc10f240 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:20:12.015 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:11.180+0000 7fcc00fc8700 1 --2- 172.21.15.67:0/3465270616 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fcbfc108400 0x7fcbfc1087e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:20:12.015 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:11.180+0000 7fcc00fc8700 1 --2- 172.21.15.67:0/3465270616 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fcbfc102390 0x7fcbfc102810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:20:12.015 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:11.180+0000 7fcc00fc8700 1 -- 172.21.15.67:0/3465270616 >> 172.21.15.67:0/3465270616 conn(0x7fcbfc079310 msgr2=0x7fcbfc079720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T21:20:12.015 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:11.180+0000 7fcc00fc8700 1 -- 172.21.15.67:0/3465270616 shutdown_connections 2024-05-03T21:20:12.015 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:11.180+0000 7fcc00fc8700 1 -- 172.21.15.67:0/3465270616 wait complete. 2024-05-03T21:20:12.016 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:11.181+0000 7fcc00fc8700 1 Processor -- start 2024-05-03T21:20:12.016 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:11.181+0000 7fcc00fc8700 1 -- start start 2024-05-03T21:20:12.016 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:11.181+0000 7fcc00fc8700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcbfc102390 0x7fcbfc104070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:20:12.016 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:11.181+0000 7fcc00fc8700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fcbfc102d50 0x7fcbfc1045b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:20:12.016 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:11.181+0000 7fcc00fc8700 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fcbfc108400 0x7fcbfc104af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:20:12.016 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:11.181+0000 7fcc00fc8700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fcbfc1127b0 con 0x7fcbfc102390 2024-05-03T21:20:12.016 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:11.181+0000 7fcc00fc8700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7fcbfc112630 con 0x7fcbfc102d50 2024-05-03T21:20:12.016 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:11.181+0000 7fcc00fc8700 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7fcbfc112930 con 0x7fcbfc108400 2024-05-03T21:20:12.016 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:11.182+0000 7fcbfa59c700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcbfc102390 0x7fcbfc104070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:20:12.016 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:11.182+0000 7fcbfa59c700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcbfc102390 0x7fcbfc104070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:50750/0 (socket says 172.21.15.67:50750) 2024-05-03T21:20:12.016 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:11.182+0000 7fcbfa59c700 1 -- 172.21.15.67:0/3228498319 learned_addr learned my addr 172.21.15.67:0/3228498319 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-03T21:20:12.016 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:11.182+0000 7fcbf9d9b700 1 --2- 172.21.15.67:0/3228498319 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fcbfc102d50 0x7fcbfc1045b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:20:12.016 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:11.182+0000 7fcbfad9d700 1 --2- 172.21.15.67:0/3228498319 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fcbfc108400 0x7fcbfc104af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:20:12.017 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:11.182+0000 7fcbfa59c700 1 -- 172.21.15.67:0/3228498319 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fcbfc108400 msgr2=0x7fcbfc104af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:20:12.017 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:11.182+0000 7fcbfa59c700 1 --2- 172.21.15.67:0/3228498319 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fcbfc108400 0x7fcbfc104af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:20:12.017 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:11.182+0000 7fcbfa59c700 1 -- 172.21.15.67:0/3228498319 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fcbfc102d50 msgr2=0x7fcbfc1045b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:20:12.017 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:11.182+0000 7fcbfa59c700 1 --2- 172.21.15.67:0/3228498319 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fcbfc102d50 0x7fcbfc1045b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:20:12.017 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:11.182+0000 7fcbfa59c700 1 -- 172.21.15.67:0/3228498319 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcbfc1a81a0 con 0x7fcbfc102390 2024-05-03T21:20:12.018 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:11.182+0000 7fcbfad9d700 1 --2- 172.21.15.67:0/3228498319 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fcbfc108400 0x7fcbfc104af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-03T21:20:12.018 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:11.182+0000 7fcbfa59c700 1 --2- 172.21.15.67:0/3228498319 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcbfc102390 0x7fcbfc104070 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7fcbec00e9a0 tx=0x7fcbec00ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T21:20:12.018 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:11.182+0000 7fcbeb7fe700 1 -- 172.21.15.67:0/3228498319 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcbec00cbd0 con 0x7fcbfc102390 2024-05-03T21:20:12.018 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:11.182+0000 7fcbeb7fe700 1 -- 172.21.15.67:0/3228498319 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(28 keys) ==== 1170+0+0 (secure 0 0 0) 0x7fcbec00cd70 con 0x7fcbfc102390 2024-05-03T21:20:12.018 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:11.182+0000 7fcc00fc8700 1 -- 172.21.15.67:0/3228498319 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcbfc1a8490 con 0x7fcbfc102390 2024-05-03T21:20:12.018 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:11.183+0000 7fcbeb7fe700 1 -- 172.21.15.67:0/3228498319 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcbec018820 con 0x7fcbfc102390 2024-05-03T21:20:12.018 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:11.183+0000 7fcc00fc8700 1 -- 172.21.15.67:0/3228498319 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcbfc1a8960 con 0x7fcbfc102390 2024-05-03T21:20:12.018 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:11.184+0000 7fcc00fc8700 1 -- 172.21.15.67:0/3228498319 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fcbfc106480 con 0x7fcbfc102390 2024-05-03T21:20:12.018 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:11.185+0000 7fcbeb7fe700 1 -- 172.21.15.67:0/3228498319 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 17) ==== 149703+0+0 (secure 0 0 0) 0x7fcbec010430 con 0x7fcbfc102390 2024-05-03T21:20:12.018 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:11.185+0000 7fcbeb7fe700 1 --2- 172.21.15.67:0/3228498319 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7fcbe40b2990 0x7fcbe40b4e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:20:12.018 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:11.185+0000 7fcbf9d9b700 1 --2- 172.21.15.67:0/3228498319 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7fcbe40b2990 0x7fcbe40b4e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:20:12.019 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:11.186+0000 7fcbeb7fe700 1 -- 172.21.15.67:0/3228498319 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 8211+0+0 (secure 0 0 0) 0x7fcbec0e24b0 con 0x7fcbfc102390 2024-05-03T21:20:12.019 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:11.186+0000 7fcbeb7fe700 1 --2- 172.21.15.67:0/3228498319 >> [v2:172.21.15.138:6800/2774330123,v1:172.21.15.138:6801/2774330123] conn(0x7fcbe40bc4d0 0x7fcbe40be970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:20:12.019 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:11.186+0000 7fcbeb7fe700 1 -- 172.21.15.67:0/3228498319 --> [v2:172.21.15.138:6800/2774330123,v1:172.21.15.138:6801/2774330123] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fcbe40bf040 con 0x7fcbe40bc4d0 2024-05-03T21:20:12.019 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:11.186+0000 7fcbeb7fe700 1 -- 172.21.15.67:0/3228498319 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_get_version_reply(handle=1 version=84) ==== 24+0+0 (secure 0 0 0) 0x7fcbec0e9050 con 0x7fcbfc102390 2024-05-03T21:20:12.019 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:11.186+0000 7fcbf9d9b700 1 --2- 172.21.15.67:0/3228498319 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7fcbe40b2990 0x7fcbe40b4e50 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fcbfc100a80 tx=0x7fcbf00073d0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T21:20:12.019 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:11.186+0000 7fcbfad9d700 1 --2- 172.21.15.67:0/3228498319 >> [v2:172.21.15.138:6800/2774330123,v1:172.21.15.138:6801/2774330123] conn(0x7fcbe40bc4d0 0x7fcbe40be970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:20:12.019 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:11.187+0000 7fcbfad9d700 1 --2- 172.21.15.67:0/3228498319 >> [v2:172.21.15.138:6800/2774330123,v1:172.21.15.138:6801/2774330123] conn(0x7fcbe40bc4d0 0x7fcbe40be970 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T21:20:12.019 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:11.187+0000 7fcbeb7fe700 1 -- 172.21.15.67:0/3228498319 <== osd.2 v2:172.21.15.138:6800/2774330123 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fcbe40bf040 con 0x7fcbe40bc4d0 2024-05-03T21:20:12.019 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:11.204+0000 7fcc00fc8700 1 -- 172.21.15.67:0/3228498319 --> [v2:172.21.15.138:6800/2774330123,v1:172.21.15.138:6801/2774330123] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fcbfc103ab0 con 0x7fcbe40bc4d0 2024-05-03T21:20:12.019 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:11.205+0000 7fcbeb7fe700 1 -- 172.21.15.67:0/3228498319 <== osd.2 v2:172.21.15.138:6800/2774330123 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7fcbfc103ab0 con 0x7fcbe40bc4d0 2024-05-03T21:20:12.019 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:11.205+0000 7fcc00fc8700 1 -- 172.21.15.67:0/3228498319 >> [v2:172.21.15.138:6800/2774330123,v1:172.21.15.138:6801/2774330123] conn(0x7fcbe40bc4d0 msgr2=0x7fcbe40be970 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:20:12.019 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:11.205+0000 7fcc00fc8700 1 --2- 172.21.15.67:0/3228498319 >> [v2:172.21.15.138:6800/2774330123,v1:172.21.15.138:6801/2774330123] conn(0x7fcbe40bc4d0 0x7fcbe40be970 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:20:12.020 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:11.205+0000 7fcc00fc8700 1 -- 172.21.15.67:0/3228498319 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7fcbe40b2990 msgr2=0x7fcbe40b4e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:20:12.020 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:11.205+0000 7fcc00fc8700 1 --2- 172.21.15.67:0/3228498319 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7fcbe40b2990 0x7fcbe40b4e50 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fcbfc100a80 tx=0x7fcbf00073d0 comp rx=0 tx=0).stop 2024-05-03T21:20:12.020 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:11.205+0000 7fcc00fc8700 1 -- 172.21.15.67:0/3228498319 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcbfc102390 msgr2=0x7fcbfc104070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:20:12.020 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:11.205+0000 7fcc00fc8700 1 --2- 172.21.15.67:0/3228498319 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcbfc102390 0x7fcbfc104070 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7fcbec00e9a0 tx=0x7fcbec00ecb0 comp rx=0 tx=0).stop 2024-05-03T21:20:12.020 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:11.205+0000 7fcbfad9d700 1 -- 172.21.15.67:0/3228498319 reap_dead start 2024-05-03T21:20:12.020 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:11.205+0000 7fcc00fc8700 1 -- 172.21.15.67:0/3228498319 shutdown_connections 2024-05-03T21:20:12.020 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:11.205+0000 7fcc00fc8700 1 -- 172.21.15.67:0/3228498319 >> 172.21.15.67:0/3228498319 conn(0x7fcbfc079310 msgr2=0x7fcbfc101500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T21:20:12.020 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:11.205+0000 7fcc00fc8700 1 -- 172.21.15.67:0/3228498319 shutdown_connections 2024-05-03T21:20:12.020 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:11.205+0000 7fcc00fc8700 1 -- 172.21.15.67:0/3228498319 wait complete. 2024-05-03T21:20:12.631 INFO:teuthology.orchestra.run.smithi067.stdout:77309411406 2024-05-03T21:20:12.631 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9a047fe7599a549042d06a3fb5b6813a203eabea shell --fsid 713bc57a-0991-11ef-bc97-c7b262605968 -- ceph osd last-stat-seq osd.2 2024-05-03T21:20:12.669 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/713bc57a-0991-11ef-bc97-c7b262605968/mon.a/config 2024-05-03T21:20:12.673 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/713bc57a-0991-11ef-bc97-c7b262605968/mon.a/config 2024-05-03T21:20:12.682 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/713bc57a-0991-11ef-bc97-c7b262605968/mon.a/config 2024-05-03T21:20:13.179 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:20:12 smithi067 ceph-mon[29730]: pgmap v333: 97 pgs: 97 active+clean; 632 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 6 op/s 2024-05-03T21:20:13.245 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:20:12 smithi082 ceph-mon[32933]: pgmap v333: 97 pgs: 97 active+clean; 632 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 6 op/s 2024-05-03T21:20:13.324 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:20:12 smithi138 ceph-mon[33110]: pgmap v333: 97 pgs: 97 active+clean; 632 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 6 op/s 2024-05-03T21:20:14.009 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/713bc57a-0991-11ef-bc97-c7b262605968/mon.a/config 2024-05-03T21:20:14.971 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:20:14 smithi067 ceph-mon[29730]: pgmap v334: 97 pgs: 97 active+clean; 632 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:20:15.009 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/713bc57a-0991-11ef-bc97-c7b262605968/mon.a/config 2024-05-03T21:20:15.245 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:20:14 smithi082 ceph-mon[32933]: pgmap v334: 97 pgs: 97 active+clean; 632 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:20:15.324 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:20:14 smithi138 ceph-mon[33110]: pgmap v334: 97 pgs: 97 active+clean; 632 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:20:16.135 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/713bc57a-0991-11ef-bc97-c7b262605968/mon.a/config 2024-05-03T21:20:16.135 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/713bc57a-0991-11ef-bc97-c7b262605968/mon.a/config 2024-05-03T21:20:16.163 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/713bc57a-0991-11ef-bc97-c7b262605968/mon.a/config 2024-05-03T21:20:16.165 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/713bc57a-0991-11ef-bc97-c7b262605968/mon.a/config 2024-05-03T21:20:17.221 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:20:16 smithi067 ceph-mon[29730]: pgmap v335: 97 pgs: 97 active+clean; 632 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-05-03T21:20:17.245 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:20:16 smithi082 ceph-mon[32933]: pgmap v335: 97 pgs: 97 active+clean; 632 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-05-03T21:20:17.324 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:20:16 smithi138 ceph-mon[33110]: pgmap v335: 97 pgs: 97 active+clean; 632 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-05-03T21:20:19.221 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:20:18 smithi067 ceph-mon[29730]: pgmap v336: 97 pgs: 97 active+clean; 632 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T21:20:19.245 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:20:18 smithi082 ceph-mon[32933]: pgmap v336: 97 pgs: 97 active+clean; 632 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T21:20:19.324 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:20:18 smithi138 ceph-mon[33110]: pgmap v336: 97 pgs: 97 active+clean; 632 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T21:20:21.221 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:20:20 smithi067 ceph-mon[29730]: pgmap v337: 97 pgs: 97 active+clean; 632 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T21:20:21.245 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:20:20 smithi082 ceph-mon[32933]: pgmap v337: 97 pgs: 97 active+clean; 632 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T21:20:21.324 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:20:20 smithi138 ceph-mon[33110]: pgmap v337: 97 pgs: 97 active+clean; 632 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T21:20:21.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:21.920+0000 7f927d568700 1 -- 172.21.15.67:0/3732797470 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f927806ba10 msgr2=0x7f927806be90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:20:21.926 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:21.920+0000 7f927d568700 1 --2- 172.21.15.67:0/3732797470 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f927806ba10 0x7f927806be90 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f926c009990 tx=0x7f926c009ca0 comp rx=0 tx=0).stop 2024-05-03T21:20:21.926 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:21.921+0000 7f927d568700 1 -- 172.21.15.67:0/3732797470 shutdown_connections 2024-05-03T21:20:21.926 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:21.921+0000 7f927d568700 1 --2- 172.21.15.67:0/3732797470 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f927806c3d0 0x7f9278110320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:20:21.926 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:21.921+0000 7f927d568700 1 --2- 172.21.15.67:0/3732797470 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f9278109500 0x7f92781098e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:20:21.926 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:21.921+0000 7f927d568700 1 --2- 172.21.15.67:0/3732797470 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f927806ba10 0x7f927806be90 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:20:21.927 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:21.921+0000 7f927d568700 1 -- 172.21.15.67:0/3732797470 >> 172.21.15.67:0/3732797470 conn(0x7f9278079310 msgr2=0x7f9278079720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T21:20:21.927 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:21.921+0000 7f927d568700 1 -- 172.21.15.67:0/3732797470 shutdown_connections 2024-05-03T21:20:21.927 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:21.921+0000 7f927d568700 1 -- 172.21.15.67:0/3732797470 wait complete. 2024-05-03T21:20:21.927 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:21.921+0000 7f927d568700 1 Processor -- start 2024-05-03T21:20:21.927 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:21.921+0000 7f927d568700 1 -- start start 2024-05-03T21:20:21.927 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:21.922+0000 7f927d568700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f927806ba10 0x7f9278102460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:20:21.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:21.922+0000 7f927d568700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f927806c3d0 0x7f92781029a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:20:21.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:21.922+0000 7f927d568700 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f9278109500 0x7f9278102ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:20:21.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:21.922+0000 7f927d568700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f9278113920 con 0x7f927806c3d0 2024-05-03T21:20:21.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:21.922+0000 7f927d568700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f92781137a0 con 0x7f927806ba10 2024-05-03T21:20:21.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:21.922+0000 7f927d568700 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f9278113aa0 con 0x7f9278109500 2024-05-03T21:20:21.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:21.922+0000 7f92767fc700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f927806c3d0 0x7f92781029a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:20:21.929 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:21.922+0000 7f92767fc700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f927806c3d0 0x7f92781029a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:55466/0 (socket says 172.21.15.67:55466) 2024-05-03T21:20:21.929 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:21.922+0000 7f92767fc700 1 -- 172.21.15.67:0/1756683273 learned_addr learned my addr 172.21.15.67:0/1756683273 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-03T21:20:21.929 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:21.922+0000 7f9276ffd700 1 --2- 172.21.15.67:0/1756683273 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f927806ba10 0x7f9278102460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:20:21.929 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:21.922+0000 7f92777fe700 1 --2- 172.21.15.67:0/1756683273 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f9278109500 0x7f9278102ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:20:21.929 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:21.922+0000 7f92767fc700 1 -- 172.21.15.67:0/1756683273 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f9278109500 msgr2=0x7f9278102ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:20:21.929 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:21.922+0000 7f92767fc700 1 --2- 172.21.15.67:0/1756683273 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f9278109500 0x7f9278102ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:20:21.929 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:21.922+0000 7f92767fc700 1 -- 172.21.15.67:0/1756683273 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f927806ba10 msgr2=0x7f9278102460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:20:21.930 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:21.922+0000 7f92767fc700 1 --2- 172.21.15.67:0/1756683273 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f927806ba10 0x7f9278102460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:20:21.930 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:21.922+0000 7f92767fc700 1 -- 172.21.15.67:0/1756683273 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f92781a8280 con 0x7f927806c3d0 2024-05-03T21:20:21.930 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:21.922+0000 7f92767fc700 1 --2- 172.21.15.67:0/1756683273 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f927806c3d0 0x7f92781029a0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f926c000c00 tx=0x7f926c00f670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T21:20:21.930 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:21.923+0000 7f925ffff700 1 -- 172.21.15.67:0/1756683273 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f926c027070 con 0x7f927806c3d0 2024-05-03T21:20:21.930 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:21.923+0000 7f925ffff700 1 -- 172.21.15.67:0/1756683273 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(28 keys) ==== 1170+0+0 (secure 0 0 0) 0x7f926c022470 con 0x7f927806c3d0 2024-05-03T21:20:21.930 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:21.923+0000 7f927d568700 1 -- 172.21.15.67:0/1756683273 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f92781a8510 con 0x7f927806c3d0 2024-05-03T21:20:21.930 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:21.923+0000 7f925ffff700 1 -- 172.21.15.67:0/1756683273 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f926c017770 con 0x7f927806c3d0 2024-05-03T21:20:21.930 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:21.923+0000 7f927d568700 1 -- 172.21.15.67:0/1756683273 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f92781a8910 con 0x7f927806c3d0 2024-05-03T21:20:21.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:21.924+0000 7f927d568700 1 -- 172.21.15.67:0/1756683273 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f927806a190 con 0x7f927806c3d0 2024-05-03T21:20:21.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:21.926+0000 7f925ffff700 1 -- 172.21.15.67:0/1756683273 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 17) ==== 149703+0+0 (secure 0 0 0) 0x7f926c02a070 con 0x7f927806c3d0 2024-05-03T21:20:21.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:21.927+0000 7f925ffff700 1 --2- 172.21.15.67:0/1756683273 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f92600b2910 0x7f92600b4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:20:21.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:21.927+0000 7f9276ffd700 1 --2- 172.21.15.67:0/1756683273 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f92600b2910 0x7f92600b4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:20:21.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:21.927+0000 7f925ffff700 1 -- 172.21.15.67:0/1756683273 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 8211+0+0 (secure 0 0 0) 0x7f926c0e3000 con 0x7f927806c3d0 2024-05-03T21:20:21.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:21.928+0000 7f9276ffd700 1 --2- 172.21.15.67:0/1756683273 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f92600b2910 0x7f92600b4dd0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f9268005950 tx=0x7f926800b410 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T21:20:21.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:21.930+0000 7f925ffff700 1 -- 172.21.15.67:0/1756683273 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191839 (secure 0 0 0) 0x7f926c013070 con 0x7f927806c3d0 2024-05-03T21:20:22.152 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:22.149+0000 7f927d568700 1 -- 172.21.15.67:0/1756683273 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 8} v 0) -- 0x7f927802e770 con 0x7f927806c3d0 2024-05-03T21:20:22.152 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:22.149+0000 7f925ffff700 1 -- 172.21.15.67:0/1756683273 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 8}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f926c0aac60 con 0x7f927806c3d0 2024-05-03T21:20:22.152 INFO:teuthology.orchestra.run.smithi067.stdout:244813135924 2024-05-03T21:20:22.154 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:22.152+0000 7f927d568700 1 -- 172.21.15.67:0/1756683273 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f92600b2910 msgr2=0x7f92600b4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:20:22.155 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:22.152+0000 7f927d568700 1 --2- 172.21.15.67:0/1756683273 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f92600b2910 0x7f92600b4dd0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f9268005950 tx=0x7f926800b410 comp rx=0 tx=0).stop 2024-05-03T21:20:22.155 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:22.152+0000 7f927d568700 1 -- 172.21.15.67:0/1756683273 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f927806c3d0 msgr2=0x7f92781029a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:20:22.155 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:22.152+0000 7f927d568700 1 --2- 172.21.15.67:0/1756683273 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f927806c3d0 0x7f92781029a0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f926c000c00 tx=0x7f926c00f670 comp rx=0 tx=0).stop 2024-05-03T21:20:22.155 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:22.152+0000 7f927d568700 1 -- 172.21.15.67:0/1756683273 shutdown_connections 2024-05-03T21:20:22.155 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:22.152+0000 7f927d568700 1 --2- 172.21.15.67:0/1756683273 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f92600b2910 0x7f92600b4dd0 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:20:22.155 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:22.152+0000 7f927d568700 1 --2- 172.21.15.67:0/1756683273 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f9278109500 0x7f9278102ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:20:22.155 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:22.152+0000 7f927d568700 1 --2- 172.21.15.67:0/1756683273 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f927806ba10 0x7f9278102460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:20:22.155 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:22.152+0000 7f927d568700 1 --2- 172.21.15.67:0/1756683273 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f927806c3d0 0x7f92781029a0 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:20:22.155 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:22.152+0000 7f927d568700 1 -- 172.21.15.67:0/1756683273 >> 172.21.15.67:0/1756683273 conn(0x7f9278079310 msgr2=0x7f9278100ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T21:20:22.155 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:22.152+0000 7f927d568700 1 -- 172.21.15.67:0/1756683273 shutdown_connections 2024-05-03T21:20:22.155 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:22.152+0000 7f927d568700 1 -- 172.21.15.67:0/1756683273 wait complete. 2024-05-03T21:20:22.385 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:22.381+0000 7fba8885b700 1 -- 172.21.15.67:0/2749127610 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fba84108400 msgr2=0x7fba841087e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:20:22.385 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:22.381+0000 7fba8885b700 1 --2- 172.21.15.67:0/2749127610 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fba84108400 0x7fba841087e0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7fba74009990 tx=0x7fba74009ca0 comp rx=0 tx=0).stop 2024-05-03T21:20:22.385 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:22.381+0000 7fba8885b700 1 -- 172.21.15.67:0/2749127610 shutdown_connections 2024-05-03T21:20:22.385 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:22.381+0000 7fba8885b700 1 --2- 172.21.15.67:0/2749127610 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fba84108400 0x7fba841087e0 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:20:22.385 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:22.381+0000 7fba8885b700 1 --2- 172.21.15.67:0/2749127610 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fba84102d50 0x7fba8410f240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:20:22.385 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:22.381+0000 7fba8885b700 1 --2- 172.21.15.67:0/2749127610 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fba84102390 0x7fba84102810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:20:22.385 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:22.381+0000 7fba8885b700 1 -- 172.21.15.67:0/2749127610 >> 172.21.15.67:0/2749127610 conn(0x7fba84079310 msgr2=0x7fba84079720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T21:20:22.385 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:22.381+0000 7fba8885b700 1 -- 172.21.15.67:0/2749127610 shutdown_connections 2024-05-03T21:20:22.385 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:22.381+0000 7fba8885b700 1 -- 172.21.15.67:0/2749127610 wait complete. 2024-05-03T21:20:22.386 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:22.382+0000 7fba8885b700 1 Processor -- start 2024-05-03T21:20:22.386 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:22.382+0000 7fba8885b700 1 -- start start 2024-05-03T21:20:22.386 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:22.382+0000 7fba8885b700 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fba84102390 0x7fba84199d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:20:22.386 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:22.382+0000 7fba8885b700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fba84102d50 0x7fba8419a260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:20:22.386 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:22.382+0000 7fba8885b700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fba84108400 0x7fba8419e5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:20:22.386 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:22.382+0000 7fba8885b700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fba84112830 con 0x7fba84108400 2024-05-03T21:20:22.386 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:22.382+0000 7fba8885b700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7fba841126b0 con 0x7fba84102d50 2024-05-03T21:20:22.386 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:22.382+0000 7fba8885b700 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7fba841129b0 con 0x7fba84102390 2024-05-03T21:20:22.386 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:22.382+0000 7fba82d9d700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fba84108400 0x7fba8419e5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:20:22.386 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:22.382+0000 7fba82d9d700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fba84108400 0x7fba8419e5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:55486/0 (socket says 172.21.15.67:55486) 2024-05-03T21:20:22.386 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:22.382+0000 7fba82d9d700 1 -- 172.21.15.67:0/4290889237 learned_addr learned my addr 172.21.15.67:0/4290889237 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-03T21:20:22.386 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:22.382+0000 7fba81d9b700 1 --2- 172.21.15.67:0/4290889237 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fba84102d50 0x7fba8419a260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:20:22.387 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:22.382+0000 7fba8259c700 1 --2- 172.21.15.67:0/4290889237 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fba84102390 0x7fba84199d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:20:22.387 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:22.382+0000 7fba82d9d700 1 -- 172.21.15.67:0/4290889237 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fba84102390 msgr2=0x7fba84199d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:20:22.387 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:22.382+0000 7fba82d9d700 1 --2- 172.21.15.67:0/4290889237 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fba84102390 0x7fba84199d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:20:22.387 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:22.382+0000 7fba82d9d700 1 -- 172.21.15.67:0/4290889237 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fba84102d50 msgr2=0x7fba8419a260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:20:22.387 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:22.382+0000 7fba82d9d700 1 --2- 172.21.15.67:0/4290889237 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fba84102d50 0x7fba8419a260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:20:22.387 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:22.382+0000 7fba82d9d700 1 -- 172.21.15.67:0/4290889237 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fba8419ed30 con 0x7fba84108400 2024-05-03T21:20:22.387 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:22.382+0000 7fba8259c700 1 --2- 172.21.15.67:0/4290889237 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fba84102390 0x7fba84199d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-03T21:20:22.387 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:22.383+0000 7fba81d9b700 1 --2- 172.21.15.67:0/4290889237 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fba84102d50 0x7fba8419a260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-03T21:20:22.387 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:22.383+0000 7fba82d9d700 1 --2- 172.21.15.67:0/4290889237 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fba84108400 0x7fba8419e5f0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7fba7c00b970 tx=0x7fba7c00bc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T21:20:22.387 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:22.383+0000 7fba737fe700 1 -- 172.21.15.67:0/4290889237 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba7c019070 con 0x7fba84108400 2024-05-03T21:20:22.388 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:22.383+0000 7fba737fe700 1 -- 172.21.15.67:0/4290889237 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(28 keys) ==== 1170+0+0 (secure 0 0 0) 0x7fba7c00ce30 con 0x7fba84108400 2024-05-03T21:20:22.388 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:22.383+0000 7fba8885b700 1 -- 172.21.15.67:0/4290889237 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fba8419f020 con 0x7fba84108400 2024-05-03T21:20:22.388 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:22.383+0000 7fba737fe700 1 -- 172.21.15.67:0/4290889237 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba7c0075d0 con 0x7fba84108400 2024-05-03T21:20:22.388 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:22.383+0000 7fba8885b700 1 -- 172.21.15.67:0/4290889237 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fba841a68a0 con 0x7fba84108400 2024-05-03T21:20:22.388 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:22.384+0000 7fba8885b700 1 -- 172.21.15.67:0/4290889237 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fba8406a190 con 0x7fba84108400 2024-05-03T21:20:22.388 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:22.386+0000 7fba737fe700 1 -- 172.21.15.67:0/4290889237 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 17) ==== 149703+0+0 (secure 0 0 0) 0x7fba7c01c070 con 0x7fba84108400 2024-05-03T21:20:22.389 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:22.386+0000 7fba737fe700 1 --2- 172.21.15.67:0/4290889237 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7fba6c0b2910 0x7fba6c0b4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:20:22.390 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:22.387+0000 7fba8259c700 1 --2- 172.21.15.67:0/4290889237 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7fba6c0b2910 0x7fba6c0b4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:20:22.390 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:22.387+0000 7fba737fe700 1 -- 172.21.15.67:0/4290889237 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 8211+0+0 (secure 0 0 0) 0x7fba7c0e1fb0 con 0x7fba84108400 2024-05-03T21:20:22.390 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:22.388+0000 7fba8259c700 1 --2- 172.21.15.67:0/4290889237 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7fba6c0b2910 0x7fba6c0b4dd0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fba74006010 tx=0x7fba7400b540 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T21:20:22.396 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:22.394+0000 7fba737fe700 1 -- 172.21.15.67:0/4290889237 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191839 (secure 0 0 0) 0x7fba7c023070 con 0x7fba84108400 2024-05-03T21:20:22.617 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:22.615+0000 7fba8885b700 1 -- 172.21.15.67:0/4290889237 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) -- 0x7fba841a6ad0 con 0x7fba84108400 2024-05-03T21:20:22.618 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:22.615+0000 7fba737fe700 1 -- 172.21.15.67:0/4290889237 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7fba7c0a9d30 con 0x7fba84108400 2024-05-03T21:20:22.618 INFO:teuthology.orchestra.run.smithi067.stdout:115964117067 2024-05-03T21:20:22.620 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:22.617+0000 7fba8885b700 1 -- 172.21.15.67:0/4290889237 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7fba6c0b2910 msgr2=0x7fba6c0b4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:20:22.620 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:22.617+0000 7fba8885b700 1 --2- 172.21.15.67:0/4290889237 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7fba6c0b2910 0x7fba6c0b4dd0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fba74006010 tx=0x7fba7400b540 comp rx=0 tx=0).stop 2024-05-03T21:20:22.620 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:22.617+0000 7fba8885b700 1 -- 172.21.15.67:0/4290889237 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fba84108400 msgr2=0x7fba8419e5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:20:22.620 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:22.617+0000 7fba8885b700 1 --2- 172.21.15.67:0/4290889237 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fba84108400 0x7fba8419e5f0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7fba7c00b970 tx=0x7fba7c00bc80 comp rx=0 tx=0).stop 2024-05-03T21:20:22.620 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:22.618+0000 7fba8885b700 1 -- 172.21.15.67:0/4290889237 shutdown_connections 2024-05-03T21:20:22.620 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:22.618+0000 7fba8885b700 1 --2- 172.21.15.67:0/4290889237 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7fba6c0b2910 0x7fba6c0b4dd0 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:20:22.620 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:22.618+0000 7fba8885b700 1 --2- 172.21.15.67:0/4290889237 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fba84108400 0x7fba8419e5f0 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:20:22.620 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:22.618+0000 7fba8885b700 1 --2- 172.21.15.67:0/4290889237 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fba84102390 0x7fba84199d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:20:22.621 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:22.618+0000 7fba8885b700 1 --2- 172.21.15.67:0/4290889237 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fba84102d50 0x7fba8419a260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:20:22.621 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:22.618+0000 7fba8885b700 1 -- 172.21.15.67:0/4290889237 >> 172.21.15.67:0/4290889237 conn(0x7fba84079310 msgr2=0x7fba84101610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T21:20:22.621 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:22.618+0000 7fba8885b700 1 -- 172.21.15.67:0/4290889237 shutdown_connections 2024-05-03T21:20:22.621 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:22.618+0000 7fba8885b700 1 -- 172.21.15.67:0/4290889237 wait complete. 2024-05-03T21:20:23.177 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:20:22 smithi067 ceph-mon[29730]: pgmap v338: 97 pgs: 97 active+clean; 632 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-05-03T21:20:23.177 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:20:22 smithi067 ceph-mon[29730]: from='client.? 172.21.15.67:0/1756683273' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-05-03T21:20:23.177 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:20:22 smithi067 ceph-mon[29730]: from='client.? 172.21.15.67:0/4290889237' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-05-03T21:20:23.245 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:20:22 smithi082 ceph-mon[32933]: pgmap v338: 97 pgs: 97 active+clean; 632 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-05-03T21:20:23.245 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:20:22 smithi082 ceph-mon[32933]: from='client.? 172.21.15.67:0/1756683273' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-05-03T21:20:23.246 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:20:22 smithi082 ceph-mon[32933]: from='client.? 172.21.15.67:0/4290889237' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-05-03T21:20:23.324 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:20:22 smithi138 ceph-mon[33110]: pgmap v338: 97 pgs: 97 active+clean; 632 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-05-03T21:20:23.325 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:20:22 smithi138 ceph-mon[33110]: from='client.? 172.21.15.67:0/1756683273' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-05-03T21:20:23.325 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:20:22 smithi138 ceph-mon[33110]: from='client.? 172.21.15.67:0/4290889237' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-05-03T21:20:24.326 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:24.321+0000 7f5cbcbd9700 1 -- 172.21.15.67:0/1909984334 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5cb8108b70 msgr2=0x7f5cb810f330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:20:24.327 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:24.321+0000 7f5cbcbd9700 1 --2- 172.21.15.67:0/1909984334 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5cb8108b70 0x7f5cb810f330 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f5cb000b0d0 tx=0x7f5cb000b3e0 comp rx=0 tx=0).stop 2024-05-03T21:20:24.327 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:24.321+0000 7f5cbcbd9700 1 -- 172.21.15.67:0/1909984334 shutdown_connections 2024-05-03T21:20:24.327 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:24.321+0000 7f5cbcbd9700 1 --2- 172.21.15.67:0/1909984334 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5cb8108b70 0x7f5cb810f330 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:20:24.327 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:24.321+0000 7f5cbcbd9700 1 --2- 172.21.15.67:0/1909984334 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f5cb8106340 0x7f5cb8106720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:20:24.327 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:24.321+0000 7f5cbcbd9700 1 --2- 172.21.15.67:0/1909984334 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f5cb81081b0 0x7f5cb8108630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:20:24.327 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:24.321+0000 7f5cbcbd9700 1 -- 172.21.15.67:0/1909984334 >> 172.21.15.67:0/1909984334 conn(0x7f5cb80776c0 msgr2=0x7f5cb8077ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T21:20:24.327 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:24.321+0000 7f5cbcbd9700 1 -- 172.21.15.67:0/1909984334 shutdown_connections 2024-05-03T21:20:24.327 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:24.321+0000 7f5cbcbd9700 1 -- 172.21.15.67:0/1909984334 wait complete. 2024-05-03T21:20:24.327 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:24.322+0000 7f5cbcbd9700 1 Processor -- start 2024-05-03T21:20:24.328 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:24.322+0000 7f5cbcbd9700 1 -- start start 2024-05-03T21:20:24.328 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:24.322+0000 7f5cbcbd9700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f5cb8106340 0x7f5cb819e7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:20:24.328 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:24.322+0000 7f5cbcbd9700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5cb81081b0 0x7f5cb819ed30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:20:24.328 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:24.322+0000 7f5cbcbd9700 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f5cb8108b70 0x7f5cb8198a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:20:24.328 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:24.322+0000 7f5cbcbd9700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f5cb8112880 con 0x7f5cb81081b0 2024-05-03T21:20:24.328 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:24.322+0000 7f5cbcbd9700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f5cb8112700 con 0x7f5cb8106340 2024-05-03T21:20:24.328 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:24.322+0000 7f5cbcbd9700 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f5cb8112a00 con 0x7f5cb8108b70 2024-05-03T21:20:24.328 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:24.322+0000 7f5cb5d9b700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5cb81081b0 0x7f5cb819ed30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:20:24.328 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:24.322+0000 7f5cb5d9b700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5cb81081b0 0x7f5cb819ed30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:55512/0 (socket says 172.21.15.67:55512) 2024-05-03T21:20:24.328 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:24.322+0000 7f5cb5d9b700 1 -- 172.21.15.67:0/3367952965 learned_addr learned my addr 172.21.15.67:0/3367952965 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-03T21:20:24.328 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:24.322+0000 7f5cb659c700 1 --2- 172.21.15.67:0/3367952965 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f5cb8106340 0x7f5cb819e7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:20:24.328 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:24.323+0000 7f5cb6d9d700 1 --2- 172.21.15.67:0/3367952965 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f5cb8108b70 0x7f5cb8198a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:20:24.328 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:24.323+0000 7f5cb5d9b700 1 -- 172.21.15.67:0/3367952965 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f5cb8108b70 msgr2=0x7f5cb8198a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:20:24.328 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:24.323+0000 7f5cb5d9b700 1 --2- 172.21.15.67:0/3367952965 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f5cb8108b70 0x7f5cb8198a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:20:24.329 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:24.323+0000 7f5cb5d9b700 1 -- 172.21.15.67:0/3367952965 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f5cb8106340 msgr2=0x7f5cb819e7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:20:24.329 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:24.323+0000 7f5cb5d9b700 1 --2- 172.21.15.67:0/3367952965 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f5cb8106340 0x7f5cb819e7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:20:24.329 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:24.323+0000 7f5cb5d9b700 1 -- 172.21.15.67:0/3367952965 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5cb8199160 con 0x7f5cb81081b0 2024-05-03T21:20:24.329 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:24.323+0000 7f5cb5d9b700 1 --2- 172.21.15.67:0/3367952965 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5cb81081b0 0x7f5cb819ed30 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f5cac00d980 tx=0x7f5cac00dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T21:20:24.329 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:24.323+0000 7f5ca77fe700 1 -- 172.21.15.67:0/3367952965 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5cac016070 con 0x7f5cb81081b0 2024-05-03T21:20:24.329 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:24.323+0000 7f5cbcbd9700 1 -- 172.21.15.67:0/3367952965 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5cb8199450 con 0x7f5cb81081b0 2024-05-03T21:20:24.329 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:24.323+0000 7f5ca77fe700 1 -- 172.21.15.67:0/3367952965 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(28 keys) ==== 1170+0+0 (secure 0 0 0) 0x7f5cac004d30 con 0x7f5cb81081b0 2024-05-03T21:20:24.329 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:24.323+0000 7f5ca77fe700 1 -- 172.21.15.67:0/3367952965 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5cac005680 con 0x7f5cb81081b0 2024-05-03T21:20:24.329 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:24.323+0000 7f5cbcbd9700 1 -- 172.21.15.67:0/3367952965 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5cb81a56c0 con 0x7f5cb81081b0 2024-05-03T21:20:24.329 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:24.324+0000 7f5cbcbd9700 1 -- 172.21.15.67:0/3367952965 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5cb806a190 con 0x7f5cb81081b0 2024-05-03T21:20:24.330 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:24.325+0000 7f5ca77fe700 1 -- 172.21.15.67:0/3367952965 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 17) ==== 149703+0+0 (secure 0 0 0) 0x7f5cac004750 con 0x7f5cb81081b0 2024-05-03T21:20:24.330 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:24.326+0000 7f5ca77fe700 1 --2- 172.21.15.67:0/3367952965 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f5ca00b2910 0x7f5ca00b4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:20:24.330 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:24.326+0000 7f5cb659c700 1 --2- 172.21.15.67:0/3367952965 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f5ca00b2910 0x7f5ca00b4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:20:24.330 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:24.327+0000 7f5ca77fe700 1 -- 172.21.15.67:0/3367952965 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 8211+0+0 (secure 0 0 0) 0x7f5cac0e2700 con 0x7f5cb81081b0 2024-05-03T21:20:24.330 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:24.327+0000 7f5cb659c700 1 --2- 172.21.15.67:0/3367952965 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f5ca00b2910 0x7f5ca00b4dd0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f5cb8107980 tx=0x7f5ca8009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T21:20:24.332 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:24.330+0000 7f5ca77fe700 1 -- 172.21.15.67:0/3367952965 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191839 (secure 0 0 0) 0x7f5cac01f020 con 0x7f5cb81081b0 2024-05-03T21:20:24.558 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:24.555+0000 7f5cbcbd9700 1 -- 172.21.15.67:0/3367952965 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) -- 0x7f5cb8199ee0 con 0x7f5cb81081b0 2024-05-03T21:20:24.558 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:24.556+0000 7f5ca77fe700 1 -- 172.21.15.67:0/3367952965 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f5cac0cae20 con 0x7f5cb81081b0 2024-05-03T21:20:24.558 INFO:teuthology.orchestra.run.smithi067.stdout:167503724611 2024-05-03T21:20:24.560 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:24.558+0000 7f5cbcbd9700 1 -- 172.21.15.67:0/3367952965 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f5ca00b2910 msgr2=0x7f5ca00b4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:20:24.560 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:24.558+0000 7f5cbcbd9700 1 --2- 172.21.15.67:0/3367952965 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f5ca00b2910 0x7f5ca00b4dd0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f5cb8107980 tx=0x7f5ca8009260 comp rx=0 tx=0).stop 2024-05-03T21:20:24.560 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:24.558+0000 7f5cbcbd9700 1 -- 172.21.15.67:0/3367952965 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5cb81081b0 msgr2=0x7f5cb819ed30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:20:24.560 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:24.558+0000 7f5cbcbd9700 1 --2- 172.21.15.67:0/3367952965 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5cb81081b0 0x7f5cb819ed30 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f5cac00d980 tx=0x7f5cac00dc90 comp rx=0 tx=0).stop 2024-05-03T21:20:24.560 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:24.558+0000 7f5cbcbd9700 1 -- 172.21.15.67:0/3367952965 shutdown_connections 2024-05-03T21:20:24.560 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:24.558+0000 7f5cbcbd9700 1 --2- 172.21.15.67:0/3367952965 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f5ca00b2910 0x7f5ca00b4dd0 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:20:24.560 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:24.558+0000 7f5cbcbd9700 1 --2- 172.21.15.67:0/3367952965 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f5cb8108b70 0x7f5cb8198a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:20:24.561 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:24.558+0000 7f5cbcbd9700 1 --2- 172.21.15.67:0/3367952965 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f5cb8106340 0x7f5cb819e7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:20:24.561 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:24.558+0000 7f5cbcbd9700 1 --2- 172.21.15.67:0/3367952965 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5cb81081b0 0x7f5cb819ed30 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:20:24.561 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:24.558+0000 7f5cbcbd9700 1 -- 172.21.15.67:0/3367952965 >> 172.21.15.67:0/3367952965 conn(0x7f5cb80776c0 msgr2=0x7f5cb80ffd70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T21:20:24.561 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:24.558+0000 7f5cbcbd9700 1 -- 172.21.15.67:0/3367952965 shutdown_connections 2024-05-03T21:20:24.561 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:24.558+0000 7f5cbcbd9700 1 -- 172.21.15.67:0/3367952965 wait complete. 2024-05-03T21:20:24.810 INFO:tasks.cephadm.ceph_manager.ceph:need seq 115964117061 got 115964117067 for osd.3 2024-05-03T21:20:24.810 DEBUG:teuthology.parallel:result is None 2024-05-03T21:20:24.996 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:24.994+0000 7f933a7db700 1 -- 172.21.15.67:0/682813762 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9334102390 msgr2=0x7f9334102810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:20:24.997 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:24.994+0000 7f933a7db700 1 --2- 172.21.15.67:0/682813762 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9334102390 0x7f9334102810 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f9328009990 tx=0x7f9328009ca0 comp rx=0 tx=0).stop 2024-05-03T21:20:24.997 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:24.994+0000 7f933a7db700 1 -- 172.21.15.67:0/682813762 shutdown_connections 2024-05-03T21:20:24.997 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:24.994+0000 7f933a7db700 1 --2- 172.21.15.67:0/682813762 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f9334102d50 0x7f933410f240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:20:24.997 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:24.994+0000 7f933a7db700 1 --2- 172.21.15.67:0/682813762 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f9334108400 0x7f93341087e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:20:24.997 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:24.994+0000 7f933a7db700 1 --2- 172.21.15.67:0/682813762 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9334102390 0x7f9334102810 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:20:24.997 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:24.994+0000 7f933a7db700 1 -- 172.21.15.67:0/682813762 >> 172.21.15.67:0/682813762 conn(0x7f9334079310 msgr2=0x7f9334079720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T21:20:24.997 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:24.994+0000 7f933a7db700 1 -- 172.21.15.67:0/682813762 shutdown_connections 2024-05-03T21:20:24.997 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:24.994+0000 7f933a7db700 1 -- 172.21.15.67:0/682813762 wait complete. 2024-05-03T21:20:24.999 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:24.995+0000 7f933a7db700 1 Processor -- start 2024-05-03T21:20:24.999 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:24.995+0000 7f933a7db700 1 -- start start 2024-05-03T21:20:24.999 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:24.995+0000 7f933a7db700 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f9334102390 0x7f933419ab10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:20:24.999 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:24.995+0000 7f933a7db700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9334102d50 0x7f933419b050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:20:24.999 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:24.995+0000 7f933a7db700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f9334108400 0x7f93341a5b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:20:24.999 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:24.995+0000 7f933a7db700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f93341127b0 con 0x7f9334102d50 2024-05-03T21:20:24.999 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:24.995+0000 7f933a7db700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f9334112630 con 0x7f9334108400 2024-05-03T21:20:24.999 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:24.995+0000 7f933a7db700 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f9334112930 con 0x7f9334102390 2024-05-03T21:20:24.999 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:24.995+0000 7f93337fe700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9334102d50 0x7f933419b050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:20:24.999 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:24.995+0000 7f93337fe700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9334102d50 0x7f933419b050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:55530/0 (socket says 172.21.15.67:55530) 2024-05-03T21:20:24.999 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:24.995+0000 7f93337fe700 1 -- 172.21.15.67:0/1648579100 learned_addr learned my addr 172.21.15.67:0/1648579100 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-03T21:20:24.999 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:24.996+0000 7f9333fff700 1 --2- 172.21.15.67:0/1648579100 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f9334102390 0x7f933419ab10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:20:24.999 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:24.996+0000 7f9338d78700 1 --2- 172.21.15.67:0/1648579100 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f9334108400 0x7f93341a5b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:20:25.000 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:24.996+0000 7f93337fe700 1 -- 172.21.15.67:0/1648579100 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f9334102390 msgr2=0x7f933419ab10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:20:25.000 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:24.996+0000 7f93337fe700 1 --2- 172.21.15.67:0/1648579100 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f9334102390 0x7f933419ab10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:20:25.000 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:24.996+0000 7f93337fe700 1 -- 172.21.15.67:0/1648579100 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f9334108400 msgr2=0x7f93341a5b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:20:25.000 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:24.996+0000 7f93337fe700 1 --2- 172.21.15.67:0/1648579100 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f9334108400 0x7f93341a5b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:20:25.000 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:24.996+0000 7f93337fe700 1 -- 172.21.15.67:0/1648579100 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f93341a60d0 con 0x7f9334102d50 2024-05-03T21:20:25.000 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:24.996+0000 7f93337fe700 1 --2- 172.21.15.67:0/1648579100 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9334102d50 0x7f933419b050 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f9328000c00 tx=0x7f932800f670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T21:20:25.000 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:24.996+0000 7f93317fa700 1 -- 172.21.15.67:0/1648579100 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9328027070 con 0x7f9334102d50 2024-05-03T21:20:25.000 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:24.996+0000 7f933a7db700 1 -- 172.21.15.67:0/1648579100 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f93341a62a0 con 0x7f9334102d50 2024-05-03T21:20:25.000 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:24.996+0000 7f93317fa700 1 -- 172.21.15.67:0/1648579100 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(28 keys) ==== 1170+0+0 (secure 0 0 0) 0x7f9328022470 con 0x7f9334102d50 2024-05-03T21:20:25.000 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:24.996+0000 7f93317fa700 1 -- 172.21.15.67:0/1648579100 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9328017630 con 0x7f9334102d50 2024-05-03T21:20:25.000 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:24.996+0000 7f933a7db700 1 -- 172.21.15.67:0/1648579100 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f93341a66f0 con 0x7f9334102d50 2024-05-03T21:20:25.001 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:24.997+0000 7f933a7db700 1 -- 172.21.15.67:0/1648579100 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f933406a190 con 0x7f9334102d50 2024-05-03T21:20:25.001 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:24.999+0000 7f93317fa700 1 -- 172.21.15.67:0/1648579100 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 17) ==== 149703+0+0 (secure 0 0 0) 0x7f932802a070 con 0x7f9334102d50 2024-05-03T21:20:25.002 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:25.000+0000 7f93317fa700 1 --2- 172.21.15.67:0/1648579100 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f931c0b2910 0x7f931c0b4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:20:25.002 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:25.000+0000 7f9333fff700 1 --2- 172.21.15.67:0/1648579100 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f931c0b2910 0x7f931c0b4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:20:25.002 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:25.000+0000 7f93317fa700 1 -- 172.21.15.67:0/1648579100 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 8211+0+0 (secure 0 0 0) 0x7f93280e3410 con 0x7f9334102d50 2024-05-03T21:20:25.002 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:25.000+0000 7f9333fff700 1 --2- 172.21.15.67:0/1648579100 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f931c0b2910 0x7f931c0b4dd0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f9324005950 tx=0x7f932400a300 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T21:20:25.006 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:25.004+0000 7f93317fa700 1 -- 172.21.15.67:0/1648579100 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191839 (secure 0 0 0) 0x7f9328013070 con 0x7f9334102d50 2024-05-03T21:20:25.106 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:20:24 smithi067 ceph-mon[29730]: pgmap v339: 97 pgs: 97 active+clean; 632 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 6 op/s 2024-05-03T21:20:25.106 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:20:24 smithi067 ceph-mon[29730]: from='client.? 172.21.15.67:0/3367952965' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-05-03T21:20:25.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:25.222+0000 7f933a7db700 1 -- 172.21.15.67:0/1648579100 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) -- 0x7f933402e510 con 0x7f9334102d50 2024-05-03T21:20:25.225 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:25.222+0000 7f93317fa700 1 -- 172.21.15.67:0/1648579100 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f93280ab070 con 0x7f9334102d50 2024-05-03T21:20:25.225 INFO:teuthology.orchestra.run.smithi067.stdout:77309411409 2024-05-03T21:20:25.227 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:25.224+0000 7f933a7db700 1 -- 172.21.15.67:0/1648579100 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f931c0b2910 msgr2=0x7f931c0b4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:20:25.227 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:25.224+0000 7f933a7db700 1 --2- 172.21.15.67:0/1648579100 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f931c0b2910 0x7f931c0b4dd0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f9324005950 tx=0x7f932400a300 comp rx=0 tx=0).stop 2024-05-03T21:20:25.227 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:25.224+0000 7f933a7db700 1 -- 172.21.15.67:0/1648579100 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9334102d50 msgr2=0x7f933419b050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:20:25.227 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:25.224+0000 7f933a7db700 1 --2- 172.21.15.67:0/1648579100 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9334102d50 0x7f933419b050 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f9328000c00 tx=0x7f932800f670 comp rx=0 tx=0).stop 2024-05-03T21:20:25.227 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:25.224+0000 7f933a7db700 1 -- 172.21.15.67:0/1648579100 shutdown_connections 2024-05-03T21:20:25.227 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:25.224+0000 7f933a7db700 1 --2- 172.21.15.67:0/1648579100 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f931c0b2910 0x7f931c0b4dd0 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:20:25.227 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:25.224+0000 7f933a7db700 1 --2- 172.21.15.67:0/1648579100 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f9334108400 0x7f93341a5b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:20:25.227 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:25.224+0000 7f933a7db700 1 --2- 172.21.15.67:0/1648579100 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f9334102390 0x7f933419ab10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:20:25.227 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:25.224+0000 7f933a7db700 1 --2- 172.21.15.67:0/1648579100 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9334102d50 0x7f933419b050 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:20:25.227 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:25.224+0000 7f933a7db700 1 -- 172.21.15.67:0/1648579100 >> 172.21.15.67:0/1648579100 conn(0x7f9334079310 msgr2=0x7f9334100ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T21:20:25.228 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:25.224+0000 7f933a7db700 1 -- 172.21.15.67:0/1648579100 shutdown_connections 2024-05-03T21:20:25.228 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:25.224+0000 7f933a7db700 1 -- 172.21.15.67:0/1648579100 wait complete. 2024-05-03T21:20:25.245 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:20:24 smithi082 ceph-mon[32933]: pgmap v339: 97 pgs: 97 active+clean; 632 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 6 op/s 2024-05-03T21:20:25.245 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:20:24 smithi082 ceph-mon[32933]: from='client.? 172.21.15.67:0/3367952965' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-05-03T21:20:25.324 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:20:24 smithi138 ceph-mon[33110]: pgmap v339: 97 pgs: 97 active+clean; 632 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 6 op/s 2024-05-03T21:20:25.325 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:20:24 smithi138 ceph-mon[33110]: from='client.? 172.21.15.67:0/3367952965' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-05-03T21:20:26.215 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:20:25 smithi067 ceph-mon[29730]: from='client.? 172.21.15.67:0/1648579100' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-05-03T21:20:26.245 INFO:tasks.cephadm.ceph_manager.ceph:need seq 77309411406 got 77309411409 for osd.2 2024-05-03T21:20:26.245 DEBUG:teuthology.parallel:result is None 2024-05-03T21:20:26.245 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:20:25 smithi082 ceph-mon[32933]: from='client.? 172.21.15.67:0/1648579100' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-05-03T21:20:26.324 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:20:25 smithi138 ceph-mon[33110]: from='client.? 172.21.15.67:0/1648579100' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-05-03T21:20:26.810 INFO:tasks.cephadm.ceph_manager.ceph:need seq 244813135920 got 244813135924 for osd.8 2024-05-03T21:20:26.810 DEBUG:teuthology.parallel:result is None 2024-05-03T21:20:27.105 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:20:26 smithi067 ceph-mon[29730]: pgmap v340: 97 pgs: 97 active+clean; 632 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:20:27.246 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:20:26 smithi082 ceph-mon[32933]: pgmap v340: 97 pgs: 97 active+clean; 632 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:20:27.329 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:20:26 smithi138 ceph-mon[33110]: pgmap v340: 97 pgs: 97 active+clean; 632 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:20:28.137 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:28.134+0000 7ff532d31700 1 -- 172.21.15.67:0/2522529229 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff52c104130 msgr2=0x7ff52c108140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:20:28.137 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:28.134+0000 7ff532d31700 1 --2- 172.21.15.67:0/2522529229 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff52c104130 0x7ff52c108140 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7ff51c0099c0 tx=0x7ff51c009cd0 comp rx=0 tx=0).stop 2024-05-03T21:20:28.137 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:28.134+0000 7ff532d31700 1 -- 172.21.15.67:0/2522529229 shutdown_connections 2024-05-03T21:20:28.137 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:28.134+0000 7ff532d31700 1 --2- 172.21.15.67:0/2522529229 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7ff52c108870 0x7ff52c1103f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:20:28.137 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:28.134+0000 7ff532d31700 1 --2- 172.21.15.67:0/2522529229 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ff52c103780 0x7ff52c103b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:20:28.137 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:28.134+0000 7ff532d31700 1 --2- 172.21.15.67:0/2522529229 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff52c104130 0x7ff52c108140 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:20:28.137 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:28.134+0000 7ff532d31700 1 -- 172.21.15.67:0/2522529229 >> 172.21.15.67:0/2522529229 conn(0x7ff52c07a610 msgr2=0x7ff52c07aa20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T21:20:28.137 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:28.134+0000 7ff532d31700 1 -- 172.21.15.67:0/2522529229 shutdown_connections 2024-05-03T21:20:28.137 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:28.134+0000 7ff532d31700 1 -- 172.21.15.67:0/2522529229 wait complete. 2024-05-03T21:20:28.139 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:28.135+0000 7ff532d31700 1 Processor -- start 2024-05-03T21:20:28.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:28.135+0000 7ff532d31700 1 -- start start 2024-05-03T21:20:28.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:28.135+0000 7ff532d31700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ff52c103780 0x7ff52c19f8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:20:28.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:28.135+0000 7ff532d31700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff52c104130 0x7ff52c19fe20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:20:28.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:28.135+0000 7ff532d31700 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7ff52c108870 0x7ff52c199aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:20:28.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:28.135+0000 7ff532d31700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7ff52c1138e0 con 0x7ff52c104130 2024-05-03T21:20:28.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:28.135+0000 7ff532d31700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7ff52c113760 con 0x7ff52c103780 2024-05-03T21:20:28.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:28.135+0000 7ff532d31700 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7ff52c113a60 con 0x7ff52c108870 2024-05-03T21:20:28.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:28.135+0000 7ff523fff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff52c104130 0x7ff52c19fe20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:20:28.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:28.135+0000 7ff523fff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff52c104130 0x7ff52c19fe20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:58578/0 (socket says 172.21.15.67:58578) 2024-05-03T21:20:28.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:28.135+0000 7ff523fff700 1 -- 172.21.15.67:0/1198548833 learned_addr learned my addr 172.21.15.67:0/1198548833 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-03T21:20:28.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:28.135+0000 7ff5312ce700 1 --2- 172.21.15.67:0/1198548833 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7ff52c108870 0x7ff52c199aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:20:28.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:28.135+0000 7ff530acd700 1 --2- 172.21.15.67:0/1198548833 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ff52c103780 0x7ff52c19f8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:20:28.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:28.136+0000 7ff523fff700 1 -- 172.21.15.67:0/1198548833 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7ff52c108870 msgr2=0x7ff52c199aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:20:28.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:28.136+0000 7ff523fff700 1 --2- 172.21.15.67:0/1198548833 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7ff52c108870 0x7ff52c199aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:20:28.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:28.136+0000 7ff523fff700 1 -- 172.21.15.67:0/1198548833 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ff52c103780 msgr2=0x7ff52c19f8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:20:28.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:28.136+0000 7ff523fff700 1 --2- 172.21.15.67:0/1198548833 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ff52c103780 0x7ff52c19f8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:20:28.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:28.136+0000 7ff523fff700 1 -- 172.21.15.67:0/1198548833 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff52c19a200 con 0x7ff52c104130 2024-05-03T21:20:28.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:28.136+0000 7ff523fff700 1 --2- 172.21.15.67:0/1198548833 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff52c104130 0x7ff52c19fe20 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7ff51c0059d0 tx=0x7ff51c00bb00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T21:20:28.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:28.136+0000 7ff521ffb700 1 -- 172.21.15.67:0/1198548833 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff51c027070 con 0x7ff52c104130 2024-05-03T21:20:28.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:28.136+0000 7ff521ffb700 1 -- 172.21.15.67:0/1198548833 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(28 keys) ==== 1170+0+0 (secure 0 0 0) 0x7ff51c003f70 con 0x7ff52c104130 2024-05-03T21:20:28.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:28.136+0000 7ff532d31700 1 -- 172.21.15.67:0/1198548833 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff52c19a490 con 0x7ff52c104130 2024-05-03T21:20:28.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:28.136+0000 7ff521ffb700 1 -- 172.21.15.67:0/1198548833 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff51c022dd0 con 0x7ff52c104130 2024-05-03T21:20:28.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:28.136+0000 7ff532d31700 1 -- 172.21.15.67:0/1198548833 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff52c1a67b0 con 0x7ff52c104130 2024-05-03T21:20:28.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:28.137+0000 7ff532d31700 1 -- 172.21.15.67:0/1198548833 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff52c104c00 con 0x7ff52c104130 2024-05-03T21:20:28.142 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:28.139+0000 7ff521ffb700 1 -- 172.21.15.67:0/1198548833 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 17) ==== 149703+0+0 (secure 0 0 0) 0x7ff51c02a070 con 0x7ff52c104130 2024-05-03T21:20:28.142 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:28.139+0000 7ff521ffb700 1 --2- 172.21.15.67:0/1198548833 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7ff5140b2910 0x7ff5140b4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:20:28.142 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:28.140+0000 7ff530acd700 1 --2- 172.21.15.67:0/1198548833 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7ff5140b2910 0x7ff5140b4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:20:28.142 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:28.140+0000 7ff521ffb700 1 -- 172.21.15.67:0/1198548833 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 8211+0+0 (secure 0 0 0) 0x7ff51c0e3190 con 0x7ff52c104130 2024-05-03T21:20:28.142 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:28.140+0000 7ff530acd700 1 --2- 172.21.15.67:0/1198548833 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7ff5140b2910 0x7ff5140b4dd0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7ff52c19abe0 tx=0x7ff52800b410 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T21:20:28.145 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:28.143+0000 7ff521ffb700 1 -- 172.21.15.67:0/1198548833 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191839 (secure 0 0 0) 0x7ff51c031080 con 0x7ff52c104130 2024-05-03T21:20:28.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:28.356+0000 7f7cd5b6b700 1 -- 172.21.15.67:0/2342247193 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7cd01095e0 msgr2=0x7f7cd01099c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:20:28.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:28.356+0000 7f7cd5b6b700 1 --2- 172.21.15.67:0/2342247193 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7cd01095e0 0x7f7cd01099c0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f7cc00099c0 tx=0x7f7cc0009cd0 comp rx=0 tx=0).stop 2024-05-03T21:20:28.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:28.356+0000 7f7cd5b6b700 1 -- 172.21.15.67:0/2342247193 shutdown_connections 2024-05-03T21:20:28.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:28.356+0000 7f7cd5b6b700 1 --2- 172.21.15.67:0/2342247193 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7cd01095e0 0x7f7cd01099c0 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:20:28.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:28.356+0000 7f7cd5b6b700 1 --2- 172.21.15.67:0/2342247193 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f7cd0103f00 0x7f7cd01103b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:20:28.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:28.356+0000 7f7cd5b6b700 1 --2- 172.21.15.67:0/2342247193 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f7cd0103540 0x7f7cd01039c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:20:28.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:28.356+0000 7f7cd5b6b700 1 -- 172.21.15.67:0/2342247193 >> 172.21.15.67:0/2342247193 conn(0x7f7cd007a650 msgr2=0x7f7cd007aa60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T21:20:28.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:28.356+0000 7f7cd5b6b700 1 -- 172.21.15.67:0/2342247193 shutdown_connections 2024-05-03T21:20:28.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:28.356+0000 7f7cd5b6b700 1 -- 172.21.15.67:0/2342247193 wait complete. 2024-05-03T21:20:28.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:28.357+0000 7f7cd5b6b700 1 Processor -- start 2024-05-03T21:20:28.361 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:28.357+0000 7f7cd5b6b700 1 -- start start 2024-05-03T21:20:28.361 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:28.357+0000 7f7cd5b6b700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f7cd0103540 0x7f7cd00757a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:20:28.361 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:28.357+0000 7f7cd5b6b700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7cd0103f00 0x7f7cd0075ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:20:28.361 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:28.357+0000 7f7cd5b6b700 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f7cd01095e0 0x7f7cd0076220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:20:28.362 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:28.357+0000 7f7cd5b6b700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f7cd0113950 con 0x7f7cd0103f00 2024-05-03T21:20:28.362 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:28.357+0000 7f7cd5b6b700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f7cd01137d0 con 0x7f7cd0103540 2024-05-03T21:20:28.362 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:28.357+0000 7f7cd5b6b700 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f7cd0113ad0 con 0x7f7cd01095e0 2024-05-03T21:20:28.362 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:28.357+0000 7f7cceffd700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7cd0103f00 0x7f7cd0075ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:20:28.362 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:28.357+0000 7f7cceffd700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7cd0103f00 0x7f7cd0075ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:58596/0 (socket says 172.21.15.67:58596) 2024-05-03T21:20:28.362 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:28.357+0000 7f7cceffd700 1 -- 172.21.15.67:0/1369965547 learned_addr learned my addr 172.21.15.67:0/1369965547 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-03T21:20:28.362 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:28.357+0000 7f7ccffff700 1 --2- 172.21.15.67:0/1369965547 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f7cd01095e0 0x7f7cd0076220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:20:28.362 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:28.357+0000 7f7ccf7fe700 1 --2- 172.21.15.67:0/1369965547 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f7cd0103540 0x7f7cd00757a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:20:28.362 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:28.358+0000 7f7cceffd700 1 -- 172.21.15.67:0/1369965547 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f7cd01095e0 msgr2=0x7f7cd0076220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:20:28.362 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:28.358+0000 7f7cceffd700 1 --2- 172.21.15.67:0/1369965547 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f7cd01095e0 0x7f7cd0076220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:20:28.362 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:28.358+0000 7f7cceffd700 1 -- 172.21.15.67:0/1369965547 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f7cd0103540 msgr2=0x7f7cd00757a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:20:28.362 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:28.358+0000 7f7cceffd700 1 --2- 172.21.15.67:0/1369965547 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f7cd0103540 0x7f7cd00757a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:20:28.362 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:28.358+0000 7f7cceffd700 1 -- 172.21.15.67:0/1369965547 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7cd01a8290 con 0x7f7cd0103f00 2024-05-03T21:20:28.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:28.358+0000 7f7cceffd700 1 --2- 172.21.15.67:0/1369965547 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7cd0103f00 0x7f7cd0075ce0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f7cb800d980 tx=0x7f7cb800dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T21:20:28.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:28.358+0000 7f7cccff9700 1 -- 172.21.15.67:0/1369965547 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7cb8016070 con 0x7f7cd0103f00 2024-05-03T21:20:28.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:28.358+0000 7f7cd5b6b700 1 -- 172.21.15.67:0/1369965547 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7cd01a8580 con 0x7f7cd0103f00 2024-05-03T21:20:28.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:28.358+0000 7f7cccff9700 1 -- 172.21.15.67:0/1369965547 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(28 keys) ==== 1170+0+0 (secure 0 0 0) 0x7f7cb8004d30 con 0x7f7cd0103f00 2024-05-03T21:20:28.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:28.358+0000 7f7cccff9700 1 -- 172.21.15.67:0/1369965547 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7cb80056c0 con 0x7f7cd0103f00 2024-05-03T21:20:28.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:28.358+0000 7f7cd5b6b700 1 -- 172.21.15.67:0/1369965547 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7cd01a8ac0 con 0x7f7cd0103f00 2024-05-03T21:20:28.364 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:28.360+0000 7f7cd5b6b700 1 -- 172.21.15.67:0/1369965547 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7cd010ba70 con 0x7f7cd0103f00 2024-05-03T21:20:28.364 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:28.360+0000 7f7cccff9700 1 -- 172.21.15.67:0/1369965547 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 17) ==== 149703+0+0 (secure 0 0 0) 0x7f7cb8004750 con 0x7f7cd0103f00 2024-05-03T21:20:28.364 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:28.361+0000 7f7cccff9700 1 --2- 172.21.15.67:0/1369965547 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f7cbc0b2910 0x7f7cbc0b4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:20:28.364 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:28.361+0000 7f7ccf7fe700 1 --2- 172.21.15.67:0/1369965547 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f7cbc0b2910 0x7f7cbc0b4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:20:28.364 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:28.361+0000 7f7cccff9700 1 -- 172.21.15.67:0/1369965547 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 8211+0+0 (secure 0 0 0) 0x7f7cb80e20b0 con 0x7f7cd0103f00 2024-05-03T21:20:28.364 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:28.362+0000 7f7ccf7fe700 1 --2- 172.21.15.67:0/1369965547 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f7cbc0b2910 0x7f7cbc0b4dd0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f7cc0000c00 tx=0x7f7cc0005fb0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T21:20:28.364 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:28.362+0000 7ff532d31700 1 -- 172.21.15.67:0/1198548833 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 11} v 0) -- 0x7ff52c02e510 con 0x7ff52c104130 2024-05-03T21:20:28.368 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:28.366+0000 7ff521ffb700 1 -- 172.21.15.67:0/1198548833 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 11}]=0 v0) ==== 75+0+13 (secure 0 0 0) 0x7ff51c0b1e20 con 0x7ff52c104130 2024-05-03T21:20:28.368 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:28.366+0000 7f7cccff9700 1 -- 172.21.15.67:0/1369965547 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191839 (secure 0 0 0) 0x7f7cb801f020 con 0x7f7cd0103f00 2024-05-03T21:20:28.368 INFO:teuthology.orchestra.run.smithi067.stdout:317827579941 2024-05-03T21:20:28.371 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:28.369+0000 7ff532d31700 1 -- 172.21.15.67:0/1198548833 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7ff5140b2910 msgr2=0x7ff5140b4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:20:28.371 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:28.369+0000 7ff532d31700 1 --2- 172.21.15.67:0/1198548833 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7ff5140b2910 0x7ff5140b4dd0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7ff52c19abe0 tx=0x7ff52800b410 comp rx=0 tx=0).stop 2024-05-03T21:20:28.371 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:28.369+0000 7ff532d31700 1 -- 172.21.15.67:0/1198548833 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff52c104130 msgr2=0x7ff52c19fe20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:20:28.371 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:28.369+0000 7ff532d31700 1 --2- 172.21.15.67:0/1198548833 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff52c104130 0x7ff52c19fe20 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7ff51c0059d0 tx=0x7ff51c00bb00 comp rx=0 tx=0).stop 2024-05-03T21:20:28.371 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:28.369+0000 7ff532d31700 1 -- 172.21.15.67:0/1198548833 shutdown_connections 2024-05-03T21:20:28.372 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:28.369+0000 7ff532d31700 1 --2- 172.21.15.67:0/1198548833 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7ff5140b2910 0x7ff5140b4dd0 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:20:28.372 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:28.369+0000 7ff532d31700 1 --2- 172.21.15.67:0/1198548833 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7ff52c108870 0x7ff52c199aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:20:28.372 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:28.369+0000 7ff532d31700 1 --2- 172.21.15.67:0/1198548833 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ff52c103780 0x7ff52c19f8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:20:28.372 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:28.369+0000 7ff532d31700 1 --2- 172.21.15.67:0/1198548833 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff52c104130 0x7ff52c19fe20 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:20:28.372 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:28.369+0000 7ff532d31700 1 -- 172.21.15.67:0/1198548833 >> 172.21.15.67:0/1198548833 conn(0x7ff52c07a610 msgr2=0x7ff52c102740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T21:20:28.372 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:28.369+0000 7ff532d31700 1 -- 172.21.15.67:0/1198548833 shutdown_connections 2024-05-03T21:20:28.372 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:28.369+0000 7ff532d31700 1 -- 172.21.15.67:0/1198548833 wait complete. 2024-05-03T21:20:28.626 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:28.623+0000 7f7cd5b6b700 1 -- 172.21.15.67:0/1369965547 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) -- 0x7f7cd0076cd0 con 0x7f7cd0103f00 2024-05-03T21:20:28.626 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:28.623+0000 7f7cccff9700 1 -- 172.21.15.67:0/1369965547 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f7cb80a9c80 con 0x7f7cd0103f00 2024-05-03T21:20:28.626 INFO:teuthology.orchestra.run.smithi067.stdout:193273528382 2024-05-03T21:20:28.628 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:28.626+0000 7f7cd5b6b700 1 -- 172.21.15.67:0/1369965547 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f7cbc0b2910 msgr2=0x7f7cbc0b4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:20:28.628 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:28.626+0000 7f7cd5b6b700 1 --2- 172.21.15.67:0/1369965547 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f7cbc0b2910 0x7f7cbc0b4dd0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f7cc0000c00 tx=0x7f7cc0005fb0 comp rx=0 tx=0).stop 2024-05-03T21:20:28.629 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:28.626+0000 7f7cd5b6b700 1 -- 172.21.15.67:0/1369965547 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7cd0103f00 msgr2=0x7f7cd0075ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:20:28.629 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:28.626+0000 7f7cd5b6b700 1 --2- 172.21.15.67:0/1369965547 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7cd0103f00 0x7f7cd0075ce0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f7cb800d980 tx=0x7f7cb800dc90 comp rx=0 tx=0).stop 2024-05-03T21:20:28.629 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:28.626+0000 7f7cd5b6b700 1 -- 172.21.15.67:0/1369965547 shutdown_connections 2024-05-03T21:20:28.629 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:28.626+0000 7f7cd5b6b700 1 --2- 172.21.15.67:0/1369965547 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f7cbc0b2910 0x7f7cbc0b4dd0 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:20:28.629 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:28.626+0000 7f7cd5b6b700 1 --2- 172.21.15.67:0/1369965547 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f7cd01095e0 0x7f7cd0076220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:20:28.629 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:28.626+0000 7f7cd5b6b700 1 --2- 172.21.15.67:0/1369965547 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f7cd0103540 0x7f7cd00757a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:20:28.629 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:28.626+0000 7f7cd5b6b700 1 --2- 172.21.15.67:0/1369965547 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7cd0103f00 0x7f7cd0075ce0 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:20:28.629 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:28.626+0000 7f7cd5b6b700 1 -- 172.21.15.67:0/1369965547 >> 172.21.15.67:0/1369965547 conn(0x7f7cd007a650 msgr2=0x7f7cd0102160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T21:20:28.629 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:28.626+0000 7f7cd5b6b700 1 -- 172.21.15.67:0/1369965547 shutdown_connections 2024-05-03T21:20:28.629 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:28.626+0000 7f7cd5b6b700 1 -- 172.21.15.67:0/1369965547 wait complete. 2024-05-03T21:20:29.064 INFO:tasks.cephadm.ceph_manager.ceph:need seq 167503724607 got 167503724611 for osd.5 2024-05-03T21:20:29.064 DEBUG:teuthology.parallel:result is None 2024-05-03T21:20:29.221 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:20:28 smithi067 ceph-mon[29730]: pgmap v341: 97 pgs: 97 active+clean; 632 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:20:29.221 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:20:28 smithi067 ceph-mon[29730]: from='client.? 172.21.15.67:0/1198548833' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-05-03T21:20:29.221 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:20:28 smithi067 ceph-mon[29730]: from='client.? 172.21.15.67:0/1369965547' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-05-03T21:20:29.245 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:20:28 smithi082 ceph-mon[32933]: pgmap v341: 97 pgs: 97 active+clean; 632 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:20:29.245 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:20:28 smithi082 ceph-mon[32933]: from='client.? 172.21.15.67:0/1198548833' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-05-03T21:20:29.245 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:20:28 smithi082 ceph-mon[32933]: from='client.? 172.21.15.67:0/1369965547' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-05-03T21:20:29.324 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:20:28 smithi138 ceph-mon[33110]: pgmap v341: 97 pgs: 97 active+clean; 632 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:20:29.325 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:20:28 smithi138 ceph-mon[33110]: from='client.? 172.21.15.67:0/1198548833' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-05-03T21:20:29.325 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:20:28 smithi138 ceph-mon[33110]: from='client.? 172.21.15.67:0/1369965547' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-05-03T21:20:29.526 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:29.524+0000 7f5b9a846700 1 -- 172.21.15.67:0/2298216984 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5b94108300 msgr2=0x7f5b941086e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:20:29.526 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:29.524+0000 7f5b9a846700 1 --2- 172.21.15.67:0/2298216984 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5b94108300 0x7f5b941086e0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f5b84009990 tx=0x7f5b84009ca0 comp rx=0 tx=0).stop 2024-05-03T21:20:29.526 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:29.524+0000 7f5b9a846700 1 -- 172.21.15.67:0/2298216984 shutdown_connections 2024-05-03T21:20:29.527 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:29.524+0000 7f5b9a846700 1 --2- 172.21.15.67:0/2298216984 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5b94108300 0x7f5b941086e0 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:20:29.527 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:29.524+0000 7f5b9a846700 1 --2- 172.21.15.67:0/2298216984 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f5b941076e0 0x7f5b9410f2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:20:29.527 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:29.524+0000 7f5b9a846700 1 --2- 172.21.15.67:0/2298216984 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f5b94108cb0 0x7f5b94107000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:20:29.527 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:29.524+0000 7f5b9a846700 1 -- 172.21.15.67:0/2298216984 >> 172.21.15.67:0/2298216984 conn(0x7f5b940785a0 msgr2=0x7f5b940ff950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T21:20:29.527 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:29.524+0000 7f5b9a846700 1 -- 172.21.15.67:0/2298216984 shutdown_connections 2024-05-03T21:20:29.527 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:29.524+0000 7f5b9a846700 1 -- 172.21.15.67:0/2298216984 wait complete. 2024-05-03T21:20:29.529 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:29.525+0000 7f5b9a846700 1 Processor -- start 2024-05-03T21:20:29.529 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:29.525+0000 7f5b9a846700 1 -- start start 2024-05-03T21:20:29.529 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:29.525+0000 7f5b9a846700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5b941076e0 0x7f5b94103090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:20:29.529 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:29.525+0000 7f5b9a846700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f5b94108300 0x7f5b941035d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:20:29.530 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:29.525+0000 7f5b9a846700 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f5b94108cb0 0x7f5b94103b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:20:29.530 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:29.525+0000 7f5b9a846700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f5b941127e0 con 0x7f5b941076e0 2024-05-03T21:20:29.530 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:29.525+0000 7f5b9a846700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f5b94112660 con 0x7f5b94108300 2024-05-03T21:20:29.530 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:29.525+0000 7f5b9a846700 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f5b94112960 con 0x7f5b94108cb0 2024-05-03T21:20:29.530 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:29.525+0000 7f5b93fff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5b941076e0 0x7f5b94103090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:20:29.530 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:29.525+0000 7f5b93fff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5b941076e0 0x7f5b94103090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:58624/0 (socket says 172.21.15.67:58624) 2024-05-03T21:20:29.530 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:29.525+0000 7f5b93fff700 1 -- 172.21.15.67:0/291597340 learned_addr learned my addr 172.21.15.67:0/291597340 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-03T21:20:29.530 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:29.525+0000 7f5b937fe700 1 --2- 172.21.15.67:0/291597340 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f5b94108300 0x7f5b941035d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:20:29.530 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:29.525+0000 7f5b98de3700 1 --2- 172.21.15.67:0/291597340 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f5b94108cb0 0x7f5b94103b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:20:29.530 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:29.526+0000 7f5b93fff700 1 -- 172.21.15.67:0/291597340 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f5b94108cb0 msgr2=0x7f5b94103b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:20:29.530 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:29.526+0000 7f5b93fff700 1 --2- 172.21.15.67:0/291597340 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f5b94108cb0 0x7f5b94103b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:20:29.530 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:29.526+0000 7f5b93fff700 1 -- 172.21.15.67:0/291597340 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f5b94108300 msgr2=0x7f5b941035d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:20:29.530 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:29.526+0000 7f5b93fff700 1 --2- 172.21.15.67:0/291597340 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f5b94108300 0x7f5b941035d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:20:29.531 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:29.526+0000 7f5b93fff700 1 -- 172.21.15.67:0/291597340 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5b941a8210 con 0x7f5b941076e0 2024-05-03T21:20:29.531 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:29.526+0000 7f5b93fff700 1 --2- 172.21.15.67:0/291597340 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5b941076e0 0x7f5b94103090 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f5b84000c00 tx=0x7f5b8400f670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T21:20:29.531 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:29.526+0000 7f5b917fa700 1 -- 172.21.15.67:0/291597340 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b84027070 con 0x7f5b941076e0 2024-05-03T21:20:29.531 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:29.526+0000 7f5b9a846700 1 -- 172.21.15.67:0/291597340 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5b941a84a0 con 0x7f5b941076e0 2024-05-03T21:20:29.531 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:29.526+0000 7f5b917fa700 1 -- 172.21.15.67:0/291597340 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(28 keys) ==== 1170+0+0 (secure 0 0 0) 0x7f5b84022470 con 0x7f5b941076e0 2024-05-03T21:20:29.531 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:29.526+0000 7f5b917fa700 1 -- 172.21.15.67:0/291597340 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b84017630 con 0x7f5b941076e0 2024-05-03T21:20:29.531 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:29.526+0000 7f5b9a846700 1 -- 172.21.15.67:0/291597340 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5b941a8950 con 0x7f5b941076e0 2024-05-03T21:20:29.531 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:29.527+0000 7f5b9a846700 1 -- 172.21.15.67:0/291597340 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5b941019a0 con 0x7f5b941076e0 2024-05-03T21:20:29.532 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:29.528+0000 7f5b917fa700 1 -- 172.21.15.67:0/291597340 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 17) ==== 149703+0+0 (secure 0 0 0) 0x7f5b8402a070 con 0x7f5b941076e0 2024-05-03T21:20:29.532 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:29.529+0000 7f5b917fa700 1 --2- 172.21.15.67:0/291597340 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f5b800b2990 0x7f5b800b4e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:20:29.532 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:29.529+0000 7f5b937fe700 1 --2- 172.21.15.67:0/291597340 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f5b800b2990 0x7f5b800b4e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:20:29.532 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:29.529+0000 7f5b917fa700 1 -- 172.21.15.67:0/291597340 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 8211+0+0 (secure 0 0 0) 0x7f5b840e3310 con 0x7f5b941076e0 2024-05-03T21:20:29.532 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:29.529+0000 7f5b937fe700 1 --2- 172.21.15.67:0/291597340 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f5b800b2990 0x7f5b800b4e50 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f5b940ffa10 tx=0x7f5b7c0073d0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T21:20:29.534 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:29.532+0000 7f5b917fa700 1 -- 172.21.15.67:0/291597340 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191839 (secure 0 0 0) 0x7f5b84013070 con 0x7f5b941076e0 2024-05-03T21:20:29.752 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:29.749+0000 7f5b9a846700 1 -- 172.21.15.67:0/291597340 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) -- 0x7f5b9402e510 con 0x7f5b941076e0 2024-05-03T21:20:29.752 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:29.750+0000 7f5b917fa700 1 -- 172.21.15.67:0/291597340 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f5b840aaf70 con 0x7f5b941076e0 2024-05-03T21:20:29.752 INFO:teuthology.orchestra.run.smithi067.stdout:55834574935 2024-05-03T21:20:29.754 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:29.752+0000 7f5b9a846700 1 -- 172.21.15.67:0/291597340 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f5b800b2990 msgr2=0x7f5b800b4e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:20:29.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:29.752+0000 7f5b9a846700 1 --2- 172.21.15.67:0/291597340 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f5b800b2990 0x7f5b800b4e50 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f5b940ffa10 tx=0x7f5b7c0073d0 comp rx=0 tx=0).stop 2024-05-03T21:20:29.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:29.752+0000 7f5b9a846700 1 -- 172.21.15.67:0/291597340 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5b941076e0 msgr2=0x7f5b94103090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:20:29.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:29.752+0000 7f5b9a846700 1 --2- 172.21.15.67:0/291597340 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5b941076e0 0x7f5b94103090 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f5b84000c00 tx=0x7f5b8400f670 comp rx=0 tx=0).stop 2024-05-03T21:20:29.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:29.752+0000 7f5b9a846700 1 -- 172.21.15.67:0/291597340 shutdown_connections 2024-05-03T21:20:29.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:29.752+0000 7f5b9a846700 1 --2- 172.21.15.67:0/291597340 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f5b800b2990 0x7f5b800b4e50 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:20:29.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:29.752+0000 7f5b9a846700 1 --2- 172.21.15.67:0/291597340 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5b941076e0 0x7f5b94103090 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:20:29.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:29.752+0000 7f5b9a846700 1 --2- 172.21.15.67:0/291597340 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f5b94108cb0 0x7f5b94103b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:20:29.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:29.752+0000 7f5b9a846700 1 --2- 172.21.15.67:0/291597340 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f5b94108300 0x7f5b941035d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:20:29.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:29.752+0000 7f5b9a846700 1 -- 172.21.15.67:0/291597340 >> 172.21.15.67:0/291597340 conn(0x7f5b940785a0 msgr2=0x7f5b94102160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T21:20:29.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:29.752+0000 7f5b9a846700 1 -- 172.21.15.67:0/291597340 shutdown_connections 2024-05-03T21:20:29.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:29.752+0000 7f5b9a846700 1 -- 172.21.15.67:0/291597340 wait complete. 2024-05-03T21:20:30.471 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:20:30 smithi067 ceph-mon[29730]: from='client.? 172.21.15.67:0/291597340' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-05-03T21:20:30.495 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:20:30 smithi082 ceph-mon[32933]: from='client.? 172.21.15.67:0/291597340' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-05-03T21:20:30.574 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:20:30 smithi138 ceph-mon[33110]: from='client.? 172.21.15.67:0/291597340' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-05-03T21:20:30.639 INFO:tasks.cephadm.ceph_manager.ceph:need seq 55834574930 got 55834574935 for osd.1 2024-05-03T21:20:30.639 DEBUG:teuthology.parallel:result is None 2024-05-03T21:20:31.228 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:31.226+0000 7f9346542700 1 -- 172.21.15.67:0/249457909 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9340104100 msgr2=0x7f9340108120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:20:31.229 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:31.226+0000 7f9346542700 1 --2- 172.21.15.67:0/249457909 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9340104100 0x7f9340108120 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f93300099c0 tx=0x7f9330009cd0 comp rx=0 tx=0).stop 2024-05-03T21:20:31.229 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:31.226+0000 7f9346542700 1 -- 172.21.15.67:0/249457909 shutdown_connections 2024-05-03T21:20:31.229 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:31.226+0000 7f9346542700 1 --2- 172.21.15.67:0/249457909 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f9340108850 0x7f93401103d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:20:31.229 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:31.226+0000 7f9346542700 1 --2- 172.21.15.67:0/249457909 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f9340103750 0x7f9340103b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:20:31.229 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:31.226+0000 7f9346542700 1 --2- 172.21.15.67:0/249457909 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9340104100 0x7f9340108120 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:20:31.229 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:31.226+0000 7f9346542700 1 -- 172.21.15.67:0/249457909 >> 172.21.15.67:0/249457909 conn(0x7f934007a600 msgr2=0x7f934007aa10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T21:20:31.229 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:31.226+0000 7f9346542700 1 -- 172.21.15.67:0/249457909 shutdown_connections 2024-05-03T21:20:31.229 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:31.226+0000 7f9346542700 1 -- 172.21.15.67:0/249457909 wait complete. 2024-05-03T21:20:31.229 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:31.226+0000 7f9346542700 1 Processor -- start 2024-05-03T21:20:31.229 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:31.226+0000 7f9346542700 1 -- start start 2024-05-03T21:20:31.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:31.227+0000 7f9346542700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f9340103750 0x7f934019beb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:20:31.232 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:31.227+0000 7f9346542700 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f9340104100 0x7f934019c3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:20:31.232 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:31.227+0000 7f9346542700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9340108850 0x7f93401a0780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:20:31.232 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:31.227+0000 7f9346542700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f93401138c0 con 0x7f9340108850 2024-05-03T21:20:31.232 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:31.227+0000 7f9346542700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f9340113740 con 0x7f9340103750 2024-05-03T21:20:31.232 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:31.227+0000 7f9346542700 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f9340113a40 con 0x7f9340104100 2024-05-03T21:20:31.232 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:31.227+0000 7f9344adf700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9340108850 0x7f93401a0780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:20:31.232 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:31.227+0000 7f9344adf700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9340108850 0x7f93401a0780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:58660/0 (socket says 172.21.15.67:58660) 2024-05-03T21:20:31.232 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:31.227+0000 7f9344adf700 1 -- 172.21.15.67:0/842641958 learned_addr learned my addr 172.21.15.67:0/842641958 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-03T21:20:31.232 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:31.227+0000 7f933ffff700 1 --2- 172.21.15.67:0/842641958 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f9340103750 0x7f934019beb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:20:31.233 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:31.227+0000 7f933f7fe700 1 --2- 172.21.15.67:0/842641958 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f9340104100 0x7f934019c3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:20:31.233 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:31.227+0000 7f9344adf700 1 -- 172.21.15.67:0/842641958 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f9340104100 msgr2=0x7f934019c3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:20:31.233 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:31.227+0000 7f9344adf700 1 --2- 172.21.15.67:0/842641958 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f9340104100 0x7f934019c3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:20:31.233 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:31.227+0000 7f9344adf700 1 -- 172.21.15.67:0/842641958 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f9340103750 msgr2=0x7f934019beb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:20:31.233 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:31.227+0000 7f9344adf700 1 --2- 172.21.15.67:0/842641958 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f9340103750 0x7f934019beb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:20:31.233 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:31.227+0000 7f9344adf700 1 -- 172.21.15.67:0/842641958 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f93401a0ec0 con 0x7f9340108850 2024-05-03T21:20:31.234 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:31.227+0000 7f933ffff700 1 --2- 172.21.15.67:0/842641958 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f9340103750 0x7f934019beb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-03T21:20:31.234 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:31.227+0000 7f9344adf700 1 --2- 172.21.15.67:0/842641958 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9340108850 0x7f93401a0780 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f9338007ee0 tx=0x7f933800d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T21:20:31.234 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:31.228+0000 7f933d7fa700 1 -- 172.21.15.67:0/842641958 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f933800dbd0 con 0x7f9340108850 2024-05-03T21:20:31.234 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:31.228+0000 7f9346542700 1 -- 172.21.15.67:0/842641958 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f93401a11b0 con 0x7f9340108850 2024-05-03T21:20:31.234 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:31.228+0000 7f933d7fa700 1 -- 172.21.15.67:0/842641958 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(28 keys) ==== 1170+0+0 (secure 0 0 0) 0x7f933800dd70 con 0x7f9340108850 2024-05-03T21:20:31.235 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:31.228+0000 7f933d7fa700 1 -- 172.21.15.67:0/842641958 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f93380133f0 con 0x7f9340108850 2024-05-03T21:20:31.235 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:31.228+0000 7f9346542700 1 -- 172.21.15.67:0/842641958 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f93401a8a30 con 0x7f9340108850 2024-05-03T21:20:31.235 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:31.229+0000 7f9346542700 1 -- 172.21.15.67:0/842641958 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f934006a190 con 0x7f9340108850 2024-05-03T21:20:31.236 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:31.230+0000 7f933d7fa700 1 -- 172.21.15.67:0/842641958 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 17) ==== 149703+0+0 (secure 0 0 0) 0x7f933801a020 con 0x7f9340108850 2024-05-03T21:20:31.236 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:31.231+0000 7f933d7fa700 1 --2- 172.21.15.67:0/842641958 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f932c0b2990 0x7f932c0b4e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:20:31.236 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:31.231+0000 7f933ffff700 1 --2- 172.21.15.67:0/842641958 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f932c0b2990 0x7f932c0b4e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:20:31.236 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:31.231+0000 7f933d7fa700 1 -- 172.21.15.67:0/842641958 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 8211+0+0 (secure 0 0 0) 0x7f93380e2110 con 0x7f9340108850 2024-05-03T21:20:31.237 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:31.231+0000 7f933ffff700 1 --2- 172.21.15.67:0/842641958 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f932c0b2990 0x7f932c0b4e50 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f9328005950 tx=0x7f9328005f50 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T21:20:31.238 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:31.235+0000 7f933d7fa700 1 -- 172.21.15.67:0/842641958 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191839 (secure 0 0 0) 0x7f9338010070 con 0x7f9340108850 2024-05-03T21:20:31.356 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:31.124+0000 7f39e7e2e700 1 -- 172.21.15.67:0/2600807729 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f39e010ff40 msgr2=0x7f39e0112330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:20:31.356 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:31.124+0000 7f39e7e2e700 1 --2- 172.21.15.67:0/2600807729 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f39e010ff40 0x7f39e0112330 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f39d800b0a0 tx=0x7f39d800b3b0 comp rx=0 tx=0).stop 2024-05-03T21:20:31.356 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:31.124+0000 7f39e7e2e700 1 -- 172.21.15.67:0/2600807729 shutdown_connections 2024-05-03T21:20:31.356 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:31.124+0000 7f39e7e2e700 1 --2- 172.21.15.67:0/2600807729 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f39e010ff40 0x7f39e0112330 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:20:31.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:31.124+0000 7f39e7e2e700 1 --2- 172.21.15.67:0/2600807729 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f39e006ba40 0x7f39e006be20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:20:31.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:31.124+0000 7f39e7e2e700 1 --2- 172.21.15.67:0/2600807729 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f39e006c360 0x7f39e010fa00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:20:31.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:31.124+0000 7f39e7e2e700 1 -- 172.21.15.67:0/2600807729 >> 172.21.15.67:0/2600807729 conn(0x7f39e0079310 msgr2=0x7f39e0079720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T21:20:31.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:31.125+0000 7f39e7e2e700 1 -- 172.21.15.67:0/2600807729 shutdown_connections 2024-05-03T21:20:31.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:31.125+0000 7f39e7e2e700 1 -- 172.21.15.67:0/2600807729 wait complete. 2024-05-03T21:20:31.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:31.125+0000 7f39e7e2e700 1 Processor -- start 2024-05-03T21:20:31.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:31.125+0000 7f39e7e2e700 1 -- start start 2024-05-03T21:20:31.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:31.126+0000 7f39e7e2e700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f39e006ba40 0x7f39e019be80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:20:31.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:31.126+0000 7f39e7e2e700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f39e006c360 0x7f39e019c3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:20:31.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:31.126+0000 7f39e7e2e700 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f39e010ff40 0x7f39e01a0700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:20:31.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:31.126+0000 7f39e7e2e700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f39e0115af0 con 0x7f39e006c360 2024-05-03T21:20:31.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:31.126+0000 7f39e7e2e700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f39e0115970 con 0x7f39e006ba40 2024-05-03T21:20:31.358 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:31.126+0000 7f39e7e2e700 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f39e0115c70 con 0x7f39e010ff40 2024-05-03T21:20:31.358 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:31.126+0000 7f39e53c9700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f39e006c360 0x7f39e019c3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:20:31.358 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:31.126+0000 7f39e53c9700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f39e006c360 0x7f39e019c3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:58644/0 (socket says 172.21.15.67:58644) 2024-05-03T21:20:31.358 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:31.126+0000 7f39e53c9700 1 -- 172.21.15.67:0/1035783852 learned_addr learned my addr 172.21.15.67:0/1035783852 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-03T21:20:31.358 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:31.126+0000 7f39e5bca700 1 --2- 172.21.15.67:0/1035783852 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f39e006ba40 0x7f39e019be80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:20:31.358 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:31.126+0000 7f39e63cb700 1 --2- 172.21.15.67:0/1035783852 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f39e010ff40 0x7f39e01a0700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:20:31.358 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:31.126+0000 7f39e53c9700 1 -- 172.21.15.67:0/1035783852 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f39e010ff40 msgr2=0x7f39e01a0700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:20:31.358 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:31.126+0000 7f39e53c9700 1 --2- 172.21.15.67:0/1035783852 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f39e010ff40 0x7f39e01a0700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:20:31.358 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:31.126+0000 7f39e53c9700 1 -- 172.21.15.67:0/1035783852 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f39e006ba40 msgr2=0x7f39e019be80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:20:31.358 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:31.126+0000 7f39e53c9700 1 --2- 172.21.15.67:0/1035783852 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f39e006ba40 0x7f39e019be80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:20:31.358 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:31.126+0000 7f39e53c9700 1 -- 172.21.15.67:0/1035783852 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f39e01a0e40 con 0x7f39e006c360 2024-05-03T21:20:31.358 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:31.126+0000 7f39e63cb700 1 --2- 172.21.15.67:0/1035783852 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f39e010ff40 0x7f39e01a0700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-03T21:20:31.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:31.126+0000 7f39e53c9700 1 --2- 172.21.15.67:0/1035783852 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f39e006c360 0x7f39e019c3c0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f39d000c970 tx=0x7f39d000cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T21:20:31.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:31.127+0000 7f39d6ffd700 1 -- 172.21.15.67:0/1035783852 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39d0007aa0 con 0x7f39e006c360 2024-05-03T21:20:31.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:31.127+0000 7f39e7e2e700 1 -- 172.21.15.67:0/1035783852 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f39e01a1130 con 0x7f39e006c360 2024-05-03T21:20:31.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:31.127+0000 7f39d6ffd700 1 -- 172.21.15.67:0/1035783852 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(28 keys) ==== 1170+0+0 (secure 0 0 0) 0x7f39d0004d30 con 0x7f39e006c360 2024-05-03T21:20:31.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:31.127+0000 7f39d6ffd700 1 -- 172.21.15.67:0/1035783852 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39d0005690 con 0x7f39e006c360 2024-05-03T21:20:31.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:31.127+0000 7f39e7e2e700 1 -- 172.21.15.67:0/1035783852 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f39e01a89b0 con 0x7f39e006c360 2024-05-03T21:20:31.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:31.128+0000 7f39e7e2e700 1 -- 172.21.15.67:0/1035783852 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f39e006a190 con 0x7f39e006c360 2024-05-03T21:20:31.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:31.129+0000 7f39d6ffd700 1 -- 172.21.15.67:0/1035783852 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 17) ==== 149703+0+0 (secure 0 0 0) 0x7f39d0004750 con 0x7f39e006c360 2024-05-03T21:20:31.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:31.130+0000 7f39d6ffd700 1 --2- 172.21.15.67:0/1035783852 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f39cc0b2910 0x7f39cc0b4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:20:31.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:31.130+0000 7f39d6ffd700 1 -- 172.21.15.67:0/1035783852 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 8211+0+0 (secure 0 0 0) 0x7f39d00e26a0 con 0x7f39e006c360 2024-05-03T21:20:31.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:31.130+0000 7f39e5bca700 1 --2- 172.21.15.67:0/1035783852 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f39cc0b2910 0x7f39cc0b4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:20:31.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:31.130+0000 7f39e5bca700 1 --2- 172.21.15.67:0/1035783852 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f39cc0b2910 0x7f39cc0b4dd0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f39dc00ac50 tx=0x7f39dc00a380 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T21:20:31.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:31.133+0000 7f39d6ffd700 1 -- 172.21.15.67:0/1035783852 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191839 (secure 0 0 0) 0x7f39d0025020 con 0x7f39e006c360 2024-05-03T21:20:31.360 INFO:teuthology.orchestra.run.smithi067.stdout:214748364856 2024-05-03T21:20:31.360 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:31.352+0000 7f39e7e2e700 1 -- 172.21.15.67:0/1035783852 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) -- 0x7f39e019d190 con 0x7f39e006c360 2024-05-03T21:20:31.361 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:31.353+0000 7f39d6ffd700 1 -- 172.21.15.67:0/1035783852 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f39e019d190 con 0x7f39e006c360 2024-05-03T21:20:31.361 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:31.355+0000 7f39e7e2e700 1 -- 172.21.15.67:0/1035783852 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f39cc0b2910 msgr2=0x7f39cc0b4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:20:31.361 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:31.355+0000 7f39e7e2e700 1 --2- 172.21.15.67:0/1035783852 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f39cc0b2910 0x7f39cc0b4dd0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f39dc00ac50 tx=0x7f39dc00a380 comp rx=0 tx=0).stop 2024-05-03T21:20:31.361 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:31.355+0000 7f39e7e2e700 1 -- 172.21.15.67:0/1035783852 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f39e006c360 msgr2=0x7f39e019c3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:20:31.361 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:31.355+0000 7f39e7e2e700 1 --2- 172.21.15.67:0/1035783852 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f39e006c360 0x7f39e019c3c0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f39d000c970 tx=0x7f39d000cc80 comp rx=0 tx=0).stop 2024-05-03T21:20:31.361 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:31.355+0000 7f39e7e2e700 1 -- 172.21.15.67:0/1035783852 shutdown_connections 2024-05-03T21:20:31.361 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:31.355+0000 7f39e7e2e700 1 --2- 172.21.15.67:0/1035783852 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f39cc0b2910 0x7f39cc0b4dd0 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:20:31.361 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:31.355+0000 7f39e7e2e700 1 --2- 172.21.15.67:0/1035783852 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f39e010ff40 0x7f39e01a0700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:20:31.361 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:31.355+0000 7f39e7e2e700 1 --2- 172.21.15.67:0/1035783852 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f39e006ba40 0x7f39e019be80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:20:31.361 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:31.355+0000 7f39e7e2e700 1 --2- 172.21.15.67:0/1035783852 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f39e006c360 0x7f39e019c3c0 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:20:31.361 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:31.355+0000 7f39e7e2e700 1 -- 172.21.15.67:0/1035783852 >> 172.21.15.67:0/1035783852 conn(0x7f39e0079310 msgr2=0x7f39e0111250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T21:20:31.361 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:31.355+0000 7f39e7e2e700 1 -- 172.21.15.67:0/1035783852 shutdown_connections 2024-05-03T21:20:31.361 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:31.355+0000 7f39e7e2e700 1 -- 172.21.15.67:0/1035783852 wait complete. 2024-05-03T21:20:31.454 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:31.452+0000 7f9346542700 1 -- 172.21.15.67:0/842641958 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 10} v 0) -- 0x7f93401a8c90 con 0x7f9340108850 2024-05-03T21:20:31.455 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:20:31 smithi067 ceph-mon[29730]: pgmap v342: 97 pgs: 97 active+clean; 632 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T21:20:31.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:31.452+0000 7f933d7fa700 1 -- 172.21.15.67:0/842641958 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 10}]=0 v0) ==== 75+0+13 (secure 0 0 0) 0x7f93380a9ce0 con 0x7f9340108850 2024-05-03T21:20:31.455 INFO:teuthology.orchestra.run.smithi067.stdout:292057776169 2024-05-03T21:20:31.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:31.454+0000 7f9346542700 1 -- 172.21.15.67:0/842641958 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f932c0b2990 msgr2=0x7f932c0b4e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:20:31.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:31.454+0000 7f9346542700 1 --2- 172.21.15.67:0/842641958 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f932c0b2990 0x7f932c0b4e50 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f9328005950 tx=0x7f9328005f50 comp rx=0 tx=0).stop 2024-05-03T21:20:31.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:31.454+0000 7f9346542700 1 -- 172.21.15.67:0/842641958 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9340108850 msgr2=0x7f93401a0780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:20:31.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:31.454+0000 7f9346542700 1 --2- 172.21.15.67:0/842641958 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9340108850 0x7f93401a0780 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f9338007ee0 tx=0x7f933800d3b0 comp rx=0 tx=0).stop 2024-05-03T21:20:31.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:31.455+0000 7f9346542700 1 -- 172.21.15.67:0/842641958 shutdown_connections 2024-05-03T21:20:31.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:31.455+0000 7f9346542700 1 --2- 172.21.15.67:0/842641958 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f932c0b2990 0x7f932c0b4e50 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:20:31.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:31.455+0000 7f9346542700 1 --2- 172.21.15.67:0/842641958 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9340108850 0x7f93401a0780 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:20:31.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:31.455+0000 7f9346542700 1 --2- 172.21.15.67:0/842641958 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f9340103750 0x7f934019beb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:20:31.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:31.455+0000 7f9346542700 1 --2- 172.21.15.67:0/842641958 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f9340104100 0x7f934019c3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:20:31.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:31.455+0000 7f9346542700 1 -- 172.21.15.67:0/842641958 >> 172.21.15.67:0/842641958 conn(0x7f934007a600 msgr2=0x7f9340107070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T21:20:31.458 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:31.455+0000 7f9346542700 1 -- 172.21.15.67:0/842641958 shutdown_connections 2024-05-03T21:20:31.458 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:31.455+0000 7f9346542700 1 -- 172.21.15.67:0/842641958 wait complete. 2024-05-03T21:20:31.495 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:20:31 smithi082 ceph-mon[32933]: pgmap v342: 97 pgs: 97 active+clean; 632 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T21:20:31.574 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:20:31 smithi138 ceph-mon[33110]: pgmap v342: 97 pgs: 97 active+clean; 632 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T21:20:31.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:31.922+0000 7f37d99fd700 1 -- 172.21.15.67:0/2925237022 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f37d4108440 msgr2=0x7f37d4108820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:20:31.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:31.922+0000 7f37d99fd700 1 --2- 172.21.15.67:0/2925237022 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f37d4108440 0x7f37d4108820 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f37c4009990 tx=0x7f37c4009ca0 comp rx=0 tx=0).stop 2024-05-03T21:20:31.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:31.922+0000 7f37d99fd700 1 -- 172.21.15.67:0/2925237022 shutdown_connections 2024-05-03T21:20:31.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:31.922+0000 7f37d99fd700 1 --2- 172.21.15.67:0/2925237022 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f37d4108440 0x7f37d4108820 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:20:31.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:31.922+0000 7f37d99fd700 1 --2- 172.21.15.67:0/2925237022 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f37d4102d90 0x7f37d410f280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:20:31.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:31.922+0000 7f37d99fd700 1 --2- 172.21.15.67:0/2925237022 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f37d41023d0 0x7f37d4102850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:20:31.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:31.922+0000 7f37d99fd700 1 -- 172.21.15.67:0/2925237022 >> 172.21.15.67:0/2925237022 conn(0x7f37d40793e0 msgr2=0x7f37d40797f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T21:20:31.926 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:31.922+0000 7f37d99fd700 1 -- 172.21.15.67:0/2925237022 shutdown_connections 2024-05-03T21:20:31.926 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:31.922+0000 7f37d99fd700 1 -- 172.21.15.67:0/2925237022 wait complete. 2024-05-03T21:20:31.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:31.923+0000 7f37d99fd700 1 Processor -- start 2024-05-03T21:20:31.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:31.923+0000 7f37d99fd700 1 -- start start 2024-05-03T21:20:31.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:31.923+0000 7f37d99fd700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f37d41023d0 0x7f37d41040b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:20:31.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:31.923+0000 7f37d99fd700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f37d4102d90 0x7f37d41045f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:20:31.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:31.923+0000 7f37d99fd700 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f37d4108440 0x7f37d4104b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:20:31.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:31.923+0000 7f37d99fd700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f37d4112870 con 0x7f37d41023d0 2024-05-03T21:20:31.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:31.923+0000 7f37d99fd700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f37d41126f0 con 0x7f37d4102d90 2024-05-03T21:20:31.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:31.923+0000 7f37d99fd700 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f37d41129f0 con 0x7f37d4108440 2024-05-03T21:20:31.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:31.923+0000 7f37d2ffd700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f37d41023d0 0x7f37d41040b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:20:31.929 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:31.923+0000 7f37d2ffd700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f37d41023d0 0x7f37d41040b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:58666/0 (socket says 172.21.15.67:58666) 2024-05-03T21:20:31.929 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:31.923+0000 7f37d2ffd700 1 -- 172.21.15.67:0/2134467450 learned_addr learned my addr 172.21.15.67:0/2134467450 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-03T21:20:31.929 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:31.924+0000 7f37cbfff700 1 --2- 172.21.15.67:0/2134467450 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f37d4102d90 0x7f37d41045f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:20:31.929 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:31.924+0000 7f37d37fe700 1 --2- 172.21.15.67:0/2134467450 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f37d4108440 0x7f37d4104b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:20:31.929 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:31.924+0000 7f37d2ffd700 1 -- 172.21.15.67:0/2134467450 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f37d4108440 msgr2=0x7f37d4104b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:20:31.929 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:31.924+0000 7f37d2ffd700 1 --2- 172.21.15.67:0/2134467450 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f37d4108440 0x7f37d4104b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:20:31.929 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:31.924+0000 7f37d2ffd700 1 -- 172.21.15.67:0/2134467450 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f37d4102d90 msgr2=0x7f37d41045f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:20:31.929 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:31.924+0000 7f37d2ffd700 1 --2- 172.21.15.67:0/2134467450 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f37d4102d90 0x7f37d41045f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:20:31.929 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:31.924+0000 7f37d2ffd700 1 -- 172.21.15.67:0/2134467450 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f37d41a8220 con 0x7f37d41023d0 2024-05-03T21:20:31.929 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:31.924+0000 7f37d2ffd700 1 --2- 172.21.15.67:0/2134467450 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f37d41023d0 0x7f37d41040b0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f37c4000c00 tx=0x7f37c400f7d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T21:20:31.930 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:31.924+0000 7f37d89fb700 1 -- 172.21.15.67:0/2134467450 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37c4027070 con 0x7f37d41023d0 2024-05-03T21:20:31.930 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:31.924+0000 7f37d89fb700 1 -- 172.21.15.67:0/2134467450 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(28 keys) ==== 1170+0+0 (secure 0 0 0) 0x7f37c400fcc0 con 0x7f37d41023d0 2024-05-03T21:20:31.930 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:31.924+0000 7f37d99fd700 1 -- 172.21.15.67:0/2134467450 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f37d41a84b0 con 0x7f37d41023d0 2024-05-03T21:20:31.930 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:31.924+0000 7f37d89fb700 1 -- 172.21.15.67:0/2134467450 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37c40178b0 con 0x7f37d41023d0 2024-05-03T21:20:31.930 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:31.924+0000 7f37d99fd700 1 -- 172.21.15.67:0/2134467450 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f37d41a8900 con 0x7f37d41023d0 2024-05-03T21:20:31.930 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:31.925+0000 7f37d99fd700 1 -- 172.21.15.67:0/2134467450 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f37d4103af0 con 0x7f37d41023d0 2024-05-03T21:20:31.931 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:31.927+0000 7f37d89fb700 1 -- 172.21.15.67:0/2134467450 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 17) ==== 149703+0+0 (secure 0 0 0) 0x7f37c402a070 con 0x7f37d41023d0 2024-05-03T21:20:31.931 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:31.928+0000 7f37d89fb700 1 --2- 172.21.15.67:0/2134467450 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f37c00b2910 0x7f37c00b4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:20:31.931 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:31.928+0000 7f37cbfff700 1 --2- 172.21.15.67:0/2134467450 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f37c00b2910 0x7f37c00b4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:20:31.931 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:31.928+0000 7f37d89fb700 1 -- 172.21.15.67:0/2134467450 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 8211+0+0 (secure 0 0 0) 0x7f37c40e3290 con 0x7f37d41023d0 2024-05-03T21:20:31.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:31.928+0000 7f37cbfff700 1 --2- 172.21.15.67:0/2134467450 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f37c00b2910 0x7f37c00b4dd0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f37d40799c0 tx=0x7f37bc00a380 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T21:20:31.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:31.931+0000 7f37d89fb700 1 -- 172.21.15.67:0/2134467450 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191839 (secure 0 0 0) 0x7f37c4013070 con 0x7f37d41023d0 2024-05-03T21:20:32.149 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:32.146+0000 7f37d99fd700 1 -- 172.21.15.67:0/2134467450 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) -- 0x7f37d41a8c30 con 0x7f37d41023d0 2024-05-03T21:20:32.149 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:32.147+0000 7f37d89fb700 1 -- 172.21.15.67:0/2134467450 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f37c40aaef0 con 0x7f37d41023d0 2024-05-03T21:20:32.149 INFO:teuthology.orchestra.run.smithi067.stdout:141733920839 2024-05-03T21:20:32.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:32.149+0000 7f37d99fd700 1 -- 172.21.15.67:0/2134467450 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f37c00b2910 msgr2=0x7f37c00b4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:20:32.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:32.149+0000 7f37d99fd700 1 --2- 172.21.15.67:0/2134467450 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f37c00b2910 0x7f37c00b4dd0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f37d40799c0 tx=0x7f37bc00a380 comp rx=0 tx=0).stop 2024-05-03T21:20:32.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:32.149+0000 7f37d99fd700 1 -- 172.21.15.67:0/2134467450 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f37d41023d0 msgr2=0x7f37d41040b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:20:32.152 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:32.149+0000 7f37d99fd700 1 --2- 172.21.15.67:0/2134467450 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f37d41023d0 0x7f37d41040b0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f37c4000c00 tx=0x7f37c400f7d0 comp rx=0 tx=0).stop 2024-05-03T21:20:32.152 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:32.149+0000 7f37d99fd700 1 -- 172.21.15.67:0/2134467450 shutdown_connections 2024-05-03T21:20:32.152 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:32.149+0000 7f37d99fd700 1 --2- 172.21.15.67:0/2134467450 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f37c00b2910 0x7f37c00b4dd0 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:20:32.152 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:32.149+0000 7f37d99fd700 1 --2- 172.21.15.67:0/2134467450 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f37d41023d0 0x7f37d41040b0 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:20:32.152 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:32.149+0000 7f37d99fd700 1 --2- 172.21.15.67:0/2134467450 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f37d4108440 0x7f37d4104b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:20:32.152 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:32.149+0000 7f37d99fd700 1 --2- 172.21.15.67:0/2134467450 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f37d4102d90 0x7f37d41045f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:20:32.152 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:32.149+0000 7f37d99fd700 1 -- 172.21.15.67:0/2134467450 >> 172.21.15.67:0/2134467450 conn(0x7f37d40793e0 msgr2=0x7f37d4101630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T21:20:32.152 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:32.149+0000 7f37d99fd700 1 -- 172.21.15.67:0/2134467450 shutdown_connections 2024-05-03T21:20:32.152 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:32.149+0000 7f37d99fd700 1 -- 172.21.15.67:0/2134467450 wait complete. 2024-05-03T21:20:32.455 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:20:32 smithi067 ceph-mon[29730]: from='client.? 172.21.15.67:0/1035783852' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-05-03T21:20:32.455 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:20:32 smithi067 ceph-mon[29730]: from='client.? 172.21.15.67:0/842641958' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-05-03T21:20:32.482 INFO:tasks.cephadm.ceph_manager.ceph:need seq 317827579935 got 317827579941 for osd.11 2024-05-03T21:20:32.482 DEBUG:teuthology.parallel:result is None 2024-05-03T21:20:32.495 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:20:32 smithi082 ceph-mon[32933]: from='client.? 172.21.15.67:0/1035783852' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-05-03T21:20:32.495 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:20:32 smithi082 ceph-mon[32933]: from='client.? 172.21.15.67:0/842641958' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-05-03T21:20:32.574 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:20:32 smithi138 ceph-mon[33110]: from='client.? 172.21.15.67:0/1035783852' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-05-03T21:20:32.575 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:20:32 smithi138 ceph-mon[33110]: from='client.? 172.21.15.67:0/842641958' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-05-03T21:20:33.022 INFO:tasks.cephadm.ceph_manager.ceph:need seq 193273528378 got 193273528382 for osd.6 2024-05-03T21:20:33.022 DEBUG:teuthology.parallel:result is None 2024-05-03T21:20:33.221 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:20:33 smithi067 ceph-mon[29730]: pgmap v343: 97 pgs: 97 active+clean; 632 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:20:33.221 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:20:33 smithi067 ceph-mon[29730]: from='client.? 172.21.15.67:0/2134467450' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-05-03T21:20:33.495 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:20:33 smithi082 ceph-mon[32933]: pgmap v343: 97 pgs: 97 active+clean; 632 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:20:33.496 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:20:33 smithi082 ceph-mon[32933]: from='client.? 172.21.15.67:0/2134467450' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-05-03T21:20:33.574 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:20:33 smithi138 ceph-mon[33110]: pgmap v343: 97 pgs: 97 active+clean; 632 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:20:33.575 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:20:33 smithi138 ceph-mon[33110]: from='client.? 172.21.15.67:0/2134467450' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-05-03T21:20:33.686 INFO:tasks.cephadm.ceph_manager.ceph:need seq 214748364853 got 214748364856 for osd.7 2024-05-03T21:20:33.687 DEBUG:teuthology.parallel:result is None 2024-05-03T21:20:34.167 INFO:tasks.cephadm.ceph_manager.ceph:need seq 292057776164 got 292057776169 for osd.10 2024-05-03T21:20:34.167 DEBUG:teuthology.parallel:result is None 2024-05-03T21:20:34.628 INFO:tasks.cephadm.ceph_manager.ceph:need seq 141733920834 got 141733920839 for osd.4 2024-05-03T21:20:34.628 DEBUG:teuthology.parallel:result is None 2024-05-03T21:20:34.628 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-05-03T21:20:34.629 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9a047fe7599a549042d06a3fb5b6813a203eabea shell --fsid 713bc57a-0991-11ef-bc97-c7b262605968 -- ceph pg dump --format=json 2024-05-03T21:20:34.888 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/713bc57a-0991-11ef-bc97-c7b262605968/mon.a/config 2024-05-03T21:20:35.330 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:20:35 smithi067 ceph-mon[29730]: pgmap v344: 97 pgs: 97 active+clean; 632 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:20:35.495 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:20:35 smithi082 ceph-mon[32933]: pgmap v344: 97 pgs: 97 active+clean; 632 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:20:35.574 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:20:35 smithi138 ceph-mon[33110]: pgmap v344: 97 pgs: 97 active+clean; 632 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:20:36.494 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:36.492+0000 7f3d03b61700 1 -- 172.21.15.67:0/1308636489 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3cfc109510 msgr2=0x7f3cfc1098f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:20:36.494 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:36.492+0000 7f3d03b61700 1 --2- 172.21.15.67:0/1308636489 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3cfc109510 0x7f3cfc1098f0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f3cf8009990 tx=0x7f3cf8009ca0 comp rx=0 tx=0).stop 2024-05-03T21:20:36.494 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:36.492+0000 7f3d03b61700 1 -- 172.21.15.67:0/1308636489 shutdown_connections 2024-05-03T21:20:36.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:36.492+0000 7f3d03b61700 1 --2- 172.21.15.67:0/1308636489 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3cfc109510 0x7f3cfc1098f0 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:20:36.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:36.492+0000 7f3d03b61700 1 --2- 172.21.15.67:0/1308636489 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f3cfc06c590 0x7f3cfc110330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:20:36.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:36.492+0000 7f3d03b61700 1 --2- 172.21.15.67:0/1308636489 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f3cfc06bbd0 0x7f3cfc06c050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:20:36.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:36.492+0000 7f3d03b61700 1 -- 172.21.15.67:0/1308636489 >> 172.21.15.67:0/1308636489 conn(0x7f3cfc079350 msgr2=0x7f3cfc079760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T21:20:36.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:36.492+0000 7f3d03b61700 1 -- 172.21.15.67:0/1308636489 shutdown_connections 2024-05-03T21:20:36.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:36.492+0000 7f3d03b61700 1 -- 172.21.15.67:0/1308636489 wait complete. 2024-05-03T21:20:36.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:36.493+0000 7f3d03b61700 1 Processor -- start 2024-05-03T21:20:36.497 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:36.493+0000 7f3d03b61700 1 -- start start 2024-05-03T21:20:36.497 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:36.493+0000 7f3d03b61700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3cfc06bbd0 0x7f3cfc19f8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:20:36.497 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:36.493+0000 7f3d03b61700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f3cfc06c590 0x7f3cfc19fe30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:20:36.497 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:36.493+0000 7f3d03b61700 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f3cfc109510 0x7f3cfc199ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:20:36.497 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:36.493+0000 7f3d03b61700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f3cfc1138b0 con 0x7f3cfc06bbd0 2024-05-03T21:20:36.497 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:36.493+0000 7f3d03b61700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f3cfc113730 con 0x7f3cfc06c590 2024-05-03T21:20:36.497 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:36.493+0000 7f3d03b61700 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f3cfc113a30 con 0x7f3cfc109510 2024-05-03T21:20:36.497 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:36.493+0000 7f3d018fd700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3cfc06bbd0 0x7f3cfc19f8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:20:36.497 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:36.493+0000 7f3d018fd700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3cfc06bbd0 0x7f3cfc19f8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:51378/0 (socket says 172.21.15.67:51378) 2024-05-03T21:20:36.497 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:36.493+0000 7f3d018fd700 1 -- 172.21.15.67:0/2949332832 learned_addr learned my addr 172.21.15.67:0/2949332832 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-03T21:20:36.497 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:36.493+0000 7f3d020fe700 1 --2- 172.21.15.67:0/2949332832 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f3cfc109510 0x7f3cfc199ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:20:36.497 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:36.493+0000 7f3d010fc700 1 --2- 172.21.15.67:0/2949332832 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f3cfc06c590 0x7f3cfc19fe30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:20:36.497 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:36.493+0000 7f3d018fd700 1 -- 172.21.15.67:0/2949332832 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f3cfc109510 msgr2=0x7f3cfc199ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:20:36.497 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:36.493+0000 7f3d018fd700 1 --2- 172.21.15.67:0/2949332832 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f3cfc109510 0x7f3cfc199ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:20:36.498 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:36.493+0000 7f3d018fd700 1 -- 172.21.15.67:0/2949332832 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f3cfc06c590 msgr2=0x7f3cfc19fe30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:20:36.498 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:36.493+0000 7f3d018fd700 1 --2- 172.21.15.67:0/2949332832 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f3cfc06c590 0x7f3cfc19fe30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:20:36.498 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:36.494+0000 7f3d018fd700 1 -- 172.21.15.67:0/2949332832 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3cfc19a210 con 0x7f3cfc06bbd0 2024-05-03T21:20:36.498 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:36.494+0000 7f3d010fc700 1 --2- 172.21.15.67:0/2949332832 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f3cfc06c590 0x7f3cfc19fe30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-03T21:20:36.498 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:36.494+0000 7f3d018fd700 1 --2- 172.21.15.67:0/2949332832 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3cfc06bbd0 0x7f3cfc19f8f0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f3cf8000c00 tx=0x7f3cf800f880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T21:20:36.498 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:36.494+0000 7f3cf2ffd700 1 -- 172.21.15.67:0/2949332832 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3cf8027070 con 0x7f3cfc06bbd0 2024-05-03T21:20:36.498 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:36.494+0000 7f3d03b61700 1 -- 172.21.15.67:0/2949332832 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3cfc19a4a0 con 0x7f3cfc06bbd0 2024-05-03T21:20:36.498 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:36.494+0000 7f3cf2ffd700 1 -- 172.21.15.67:0/2949332832 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(28 keys) ==== 1170+0+0 (secure 0 0 0) 0x7f3cf800fd70 con 0x7f3cfc06bbd0 2024-05-03T21:20:36.498 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:36.494+0000 7f3cf2ffd700 1 -- 172.21.15.67:0/2949332832 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3cf8017950 con 0x7f3cfc06bbd0 2024-05-03T21:20:36.498 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:36.494+0000 7f3d03b61700 1 -- 172.21.15.67:0/2949332832 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3cfc1a67c0 con 0x7f3cfc06bbd0 2024-05-03T21:20:36.499 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:36.495+0000 7f3d03b61700 1 -- 172.21.15.67:0/2949332832 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3cfc10b9f0 con 0x7f3cfc06bbd0 2024-05-03T21:20:36.499 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:36.496+0000 7f3cf2ffd700 1 -- 172.21.15.67:0/2949332832 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 17) ==== 149703+0+0 (secure 0 0 0) 0x7f3cf802a070 con 0x7f3cfc06bbd0 2024-05-03T21:20:36.499 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:36.497+0000 7f3cf2ffd700 1 --2- 172.21.15.67:0/2949332832 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f3ce80b2910 0x7f3ce80b4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:20:36.499 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:36.497+0000 7f3d010fc700 1 --2- 172.21.15.67:0/2949332832 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f3ce80b2910 0x7f3ce80b4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:20:36.499 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:36.497+0000 7f3cf2ffd700 1 -- 172.21.15.67:0/2949332832 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 8211+0+0 (secure 0 0 0) 0x7f3cf80e32e0 con 0x7f3cfc06bbd0 2024-05-03T21:20:36.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:36.498+0000 7f3d010fc700 1 --2- 172.21.15.67:0/2949332832 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f3ce80b2910 0x7f3ce80b4dd0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f3cec005fd0 tx=0x7f3cec005e20 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T21:20:36.503 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:36.501+0000 7f3cf2ffd700 1 -- 172.21.15.67:0/2949332832 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191839 (secure 0 0 0) 0x7f3cf8013070 con 0x7f3cfc06bbd0 2024-05-03T21:20:36.712 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:36.709+0000 7f3d03b61700 1 -- 172.21.15.67:0/2949332832 --> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f3cfc19afe0 con 0x7f3ce80b2910 2024-05-03T21:20:36.719 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:36.717+0000 7f3cf2ffd700 1 -- 172.21.15.67:0/2949332832 <== mgr.14150 v2:172.21.15.67:6800/604451592 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+258711 (secure 0 0 0) 0x7f3cfc19afe0 con 0x7f3ce80b2910 2024-05-03T21:20:36.723 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-05-03T21:20:36.725 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:36.720+0000 7f3d03b61700 1 -- 172.21.15.67:0/2949332832 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f3ce80b2910 msgr2=0x7f3ce80b4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:20:36.726 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:36.720+0000 7f3d03b61700 1 --2- 172.21.15.67:0/2949332832 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f3ce80b2910 0x7f3ce80b4dd0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f3cec005fd0 tx=0x7f3cec005e20 comp rx=0 tx=0).stop 2024-05-03T21:20:36.726 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:36.720+0000 7f3d03b61700 1 -- 172.21.15.67:0/2949332832 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3cfc06bbd0 msgr2=0x7f3cfc19f8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:20:36.726 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:36.720+0000 7f3d03b61700 1 --2- 172.21.15.67:0/2949332832 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3cfc06bbd0 0x7f3cfc19f8f0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f3cf8000c00 tx=0x7f3cf800f880 comp rx=0 tx=0).stop 2024-05-03T21:20:36.726 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:36.720+0000 7f3d03b61700 1 -- 172.21.15.67:0/2949332832 shutdown_connections 2024-05-03T21:20:36.726 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:36.720+0000 7f3d03b61700 1 --2- 172.21.15.67:0/2949332832 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f3ce80b2910 0x7f3ce80b4dd0 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:20:36.726 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:36.720+0000 7f3d03b61700 1 --2- 172.21.15.67:0/2949332832 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3cfc06bbd0 0x7f3cfc19f8f0 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:20:36.726 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:36.720+0000 7f3d03b61700 1 --2- 172.21.15.67:0/2949332832 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f3cfc109510 0x7f3cfc199ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:20:36.726 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:36.720+0000 7f3d03b61700 1 --2- 172.21.15.67:0/2949332832 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f3cfc06c590 0x7f3cfc19fe30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:20:36.726 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:36.720+0000 7f3d03b61700 1 -- 172.21.15.67:0/2949332832 >> 172.21.15.67:0/2949332832 conn(0x7f3cfc079350 msgr2=0x7f3cfc100e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T21:20:36.726 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:36.720+0000 7f3d03b61700 1 -- 172.21.15.67:0/2949332832 shutdown_connections 2024-05-03T21:20:36.726 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:36.720+0000 7f3d03b61700 1 -- 172.21.15.67:0/2949332832 wait complete. 2024-05-03T21:20:36.727 INFO:teuthology.orchestra.run.smithi067.stderr:dumped all 2024-05-03T21:20:37.443 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:20:37 smithi067 ceph-mon[29730]: pgmap v345: 97 pgs: 97 active+clean; 632 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:20:37.474 INFO:teuthology.orchestra.run.smithi067.stdout:{"pg_ready":true,"pg_map":{"version":345,"stamp":"2024-05-03T21:20:35.833772+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":646756,"num_objects":64,"num_object_clones":0,"num_object_copies":192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":64,"num_whiteouts":0,"num_read":663,"num_read_kb":418,"num_write":283,"num_write_kb":2252,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":2951952,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":230,"ondisk_log_size":230,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":291,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":382736,"kb_used_data":12708,"kb_used_omap":18,"kb_used_meta":369837,"kb_avail":1124559088,"statfs":{"total":1151940427776,"available":1151548506112,"internally_reserved":0,"allocated":13012992,"data_stored":7238424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19143,"internal_metadata":378713401},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":75,"num_read_kb":40,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003187"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":20,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-05-03T21:19:34.975095+0000","last_change":"2024-05-03T21:18:42.819790+0000","last_active":"2024-05-03T21:19:34.975095+0000","last_peered":"2024-05-03T21:19:34.975095+0000","last_clean":"2024-05-03T21:19:34.975095+0000","last_became_active":"2024-05-03T21:18:42.818989+0000","last_became_peered":"2024-05-03T21:18:42.818989+0000","last_unstale":"2024-05-03T21:19:34.975095+0000","last_undegraded":"2024-05-03T21:19:34.975095+0000","last_fullsized":"2024-05-03T21:19:34.975095+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_clean_scrub_stamp":"2024-05-03T21:18:41.788726+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T08:38:43.699198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":20,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-05-03T21:19:34.979895+0000","last_change":"2024-05-03T21:18:42.812454+0000","last_active":"2024-05-03T21:19:34.979895+0000","last_peered":"2024-05-03T21:19:34.979895+0000","last_clean":"2024-05-03T21:19:34.979895+0000","last_became_active":"2024-05-03T21:18:42.812140+0000","last_became_peered":"2024-05-03T21:18:42.812140+0000","last_unstale":"2024-05-03T21:19:34.979895+0000","last_undegraded":"2024-05-03T21:19:34.979895+0000","last_fullsized":"2024-05-03T21:19:34.979895+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_clean_scrub_stamp":"2024-05-03T21:18:41.788726+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-04T22:30:36.057079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":20,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-05-03T21:19:34.979632+0000","last_change":"2024-05-03T21:18:42.815547+0000","last_active":"2024-05-03T21:19:34.979632+0000","last_peered":"2024-05-03T21:19:34.979632+0000","last_clean":"2024-05-03T21:19:34.979632+0000","last_became_active":"2024-05-03T21:18:42.815192+0000","last_became_peered":"2024-05-03T21:18:42.815192+0000","last_unstale":"2024-05-03T21:19:34.979632+0000","last_undegraded":"2024-05-03T21:19:34.979632+0000","last_fullsized":"2024-05-03T21:19:34.979632+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_clean_scrub_stamp":"2024-05-03T21:18:41.788726+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T04:15:05.786275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":11,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-05-03T21:19:34.983212+0000","last_change":"2024-05-03T21:19:34.982658+0000","last_active":"2024-05-03T21:19:34.983212+0000","last_peered":"2024-05-03T21:19:34.983212+0000","last_clean":"2024-05-03T21:19:34.983212+0000","last_became_active":"2024-05-03T21:19:34.982225+0000","last_became_peered":"2024-05-03T21:19:34.982225+0000","last_unstale":"2024-05-03T21:19:34.983212+0000","last_undegraded":"2024-05-03T21:19:34.983212+0000","last_fullsized":"2024-05-03T21:19:34.983212+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_clean_scrub_stamp":"2024-05-03T21:18:41.788726+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T00:11:37.702317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,2],"acting":[9,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":20,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-05-03T21:19:34.972150+0000","last_change":"2024-05-03T21:18:42.807225+0000","last_active":"2024-05-03T21:19:34.972150+0000","last_peered":"2024-05-03T21:19:34.972150+0000","last_clean":"2024-05-03T21:19:34.972150+0000","last_became_active":"2024-05-03T21:18:42.806856+0000","last_became_peered":"2024-05-03T21:18:42.806856+0000","last_unstale":"2024-05-03T21:19:34.972150+0000","last_undegraded":"2024-05-03T21:19:34.972150+0000","last_fullsized":"2024-05-03T21:19:34.972150+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_clean_scrub_stamp":"2024-05-03T21:18:41.788726+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-04T23:59:15.221592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":20,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-05-03T21:19:35.208158+0000","last_change":"2024-05-03T21:18:42.819099+0000","last_active":"2024-05-03T21:19:35.208158+0000","last_peered":"2024-05-03T21:19:35.208158+0000","last_clean":"2024-05-03T21:19:35.208158+0000","last_became_active":"2024-05-03T21:18:42.818775+0000","last_became_peered":"2024-05-03T21:18:42.818775+0000","last_unstale":"2024-05-03T21:19:35.208158+0000","last_undegraded":"2024-05-03T21:19:35.208158+0000","last_fullsized":"2024-05-03T21:19:35.208158+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_clean_scrub_stamp":"2024-05-03T21:18:41.788726+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T03:46:43.762816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":20,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-05-03T21:19:34.973759+0000","last_change":"2024-05-03T21:18:42.811779+0000","last_active":"2024-05-03T21:19:34.973759+0000","last_peered":"2024-05-03T21:19:34.973759+0000","last_clean":"2024-05-03T21:19:34.973759+0000","last_became_active":"2024-05-03T21:18:42.811376+0000","last_became_peered":"2024-05-03T21:18:42.811376+0000","last_unstale":"2024-05-03T21:19:34.973759+0000","last_undegraded":"2024-05-03T21:19:34.973759+0000","last_fullsized":"2024-05-03T21:19:34.973759+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_clean_scrub_stamp":"2024-05-03T21:18:41.788726+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-04T22:08:00.629106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":20,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-05-03T21:19:34.978376+0000","last_change":"2024-05-03T21:18:42.820643+0000","last_active":"2024-05-03T21:19:34.978376+0000","last_peered":"2024-05-03T21:19:34.978376+0000","last_clean":"2024-05-03T21:19:34.978376+0000","last_became_active":"2024-05-03T21:18:42.820317+0000","last_became_peered":"2024-05-03T21:18:42.820317+0000","last_unstale":"2024-05-03T21:19:34.978376+0000","last_undegraded":"2024-05-03T21:19:34.978376+0000","last_fullsized":"2024-05-03T21:19:34.978376+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_clean_scrub_stamp":"2024-05-03T21:18:41.788726+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-04T21:39:44.319396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":20,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-05-03T21:19:34.972315+0000","last_change":"2024-05-03T21:18:42.819895+0000","last_active":"2024-05-03T21:19:34.972315+0000","last_peered":"2024-05-03T21:19:34.972315+0000","last_clean":"2024-05-03T21:19:34.972315+0000","last_became_active":"2024-05-03T21:18:42.818360+0000","last_became_peered":"2024-05-03T21:18:42.818360+0000","last_unstale":"2024-05-03T21:19:34.972315+0000","last_undegraded":"2024-05-03T21:19:34.972315+0000","last_fullsized":"2024-05-03T21:19:34.972315+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_clean_scrub_stamp":"2024-05-03T21:18:41.788726+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T06:49:09.368610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":20,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-05-03T21:19:35.208821+0000","last_change":"2024-05-03T21:18:42.817064+0000","last_active":"2024-05-03T21:19:35.208821+0000","last_peered":"2024-05-03T21:19:35.208821+0000","last_clean":"2024-05-03T21:19:35.208821+0000","last_became_active":"2024-05-03T21:18:42.816727+0000","last_became_peered":"2024-05-03T21:18:42.816727+0000","last_unstale":"2024-05-03T21:19:35.208821+0000","last_undegraded":"2024-05-03T21:19:35.208821+0000","last_fullsized":"2024-05-03T21:19:35.208821+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_clean_scrub_stamp":"2024-05-03T21:18:41.788726+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T06:30:20.093765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":20,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-05-03T21:19:34.974044+0000","last_change":"2024-05-03T21:18:42.818728+0000","last_active":"2024-05-03T21:19:34.974044+0000","last_peered":"2024-05-03T21:19:34.974044+0000","last_clean":"2024-05-03T21:19:34.974044+0000","last_became_active":"2024-05-03T21:18:42.818250+0000","last_became_peered":"2024-05-03T21:18:42.818250+0000","last_unstale":"2024-05-03T21:19:34.974044+0000","last_undegraded":"2024-05-03T21:19:34.974044+0000","last_fullsized":"2024-05-03T21:19:34.974044+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_clean_scrub_stamp":"2024-05-03T21:18:41.788726+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T03:52:04.613842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":20,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-05-03T21:19:35.208942+0000","last_change":"2024-05-03T21:18:42.816897+0000","last_active":"2024-05-03T21:19:35.208942+0000","last_peered":"2024-05-03T21:19:35.208942+0000","last_clean":"2024-05-03T21:19:35.208942+0000","last_became_active":"2024-05-03T21:18:42.816274+0000","last_became_peered":"2024-05-03T21:18:42.816274+0000","last_unstale":"2024-05-03T21:19:35.208942+0000","last_undegraded":"2024-05-03T21:19:35.208942+0000","last_fullsized":"2024-05-03T21:19:35.208942+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_clean_scrub_stamp":"2024-05-03T21:18:41.788726+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-04T21:48:17.744413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":20,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-05-03T21:19:34.973435+0000","last_change":"2024-05-03T21:18:42.820523+0000","last_active":"2024-05-03T21:19:34.973435+0000","last_peered":"2024-05-03T21:19:34.973435+0000","last_clean":"2024-05-03T21:19:34.973435+0000","last_became_active":"2024-05-03T21:18:42.820133+0000","last_became_peered":"2024-05-03T21:18:42.820133+0000","last_unstale":"2024-05-03T21:19:34.973435+0000","last_undegraded":"2024-05-03T21:19:34.973435+0000","last_fullsized":"2024-05-03T21:19:34.973435+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_clean_scrub_stamp":"2024-05-03T21:18:41.788726+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-04T23:25:34.304369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":20,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-05-03T21:19:34.978724+0000","last_change":"2024-05-03T21:18:42.814467+0000","last_active":"2024-05-03T21:19:34.978724+0000","last_peered":"2024-05-03T21:19:34.978724+0000","last_clean":"2024-05-03T21:19:34.978724+0000","last_became_active":"2024-05-03T21:18:42.814124+0000","last_became_peered":"2024-05-03T21:18:42.814124+0000","last_unstale":"2024-05-03T21:19:34.978724+0000","last_undegraded":"2024-05-03T21:19:34.978724+0000","last_fullsized":"2024-05-03T21:19:34.978724+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_clean_scrub_stamp":"2024-05-03T21:18:41.788726+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T08:41:03.567620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":20,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-05-03T21:19:34.978587+0000","last_change":"2024-05-03T21:18:42.822158+0000","last_active":"2024-05-03T21:19:34.978587+0000","last_peered":"2024-05-03T21:19:34.978587+0000","last_clean":"2024-05-03T21:19:34.978587+0000","last_became_active":"2024-05-03T21:18:42.821793+0000","last_became_peered":"2024-05-03T21:18:42.821793+0000","last_unstale":"2024-05-03T21:19:34.978587+0000","last_undegraded":"2024-05-03T21:19:34.978587+0000","last_fullsized":"2024-05-03T21:19:34.978587+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_clean_scrub_stamp":"2024-05-03T21:18:41.788726+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T06:44:09.269513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":20,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-05-03T21:19:34.971720+0000","last_change":"2024-05-03T21:18:42.813904+0000","last_active":"2024-05-03T21:19:34.971720+0000","last_peered":"2024-05-03T21:19:34.971720+0000","last_clean":"2024-05-03T21:19:34.971720+0000","last_became_active":"2024-05-03T21:18:42.813488+0000","last_became_peered":"2024-05-03T21:18:42.813488+0000","last_unstale":"2024-05-03T21:19:34.971720+0000","last_undegraded":"2024-05-03T21:19:34.971720+0000","last_fullsized":"2024-05-03T21:19:34.971720+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_clean_scrub_stamp":"2024-05-03T21:18:41.788726+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T00:52:20.288003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":20,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-05-03T21:19:34.975229+0000","last_change":"2024-05-03T21:18:42.820157+0000","last_active":"2024-05-03T21:19:34.975229+0000","last_peered":"2024-05-03T21:19:34.975229+0000","last_clean":"2024-05-03T21:19:34.975229+0000","last_became_active":"2024-05-03T21:18:42.819450+0000","last_became_peered":"2024-05-03T21:18:42.819450+0000","last_unstale":"2024-05-03T21:19:34.975229+0000","last_undegraded":"2024-05-03T21:19:34.975229+0000","last_fullsized":"2024-05-03T21:19:34.975229+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_clean_scrub_stamp":"2024-05-03T21:18:41.788726+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T04:34:19.252499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":20,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-05-03T21:19:34.973800+0000","last_change":"2024-05-03T21:18:42.816582+0000","last_active":"2024-05-03T21:19:34.973800+0000","last_peered":"2024-05-03T21:19:34.973800+0000","last_clean":"2024-05-03T21:19:34.973800+0000","last_became_active":"2024-05-03T21:18:42.816247+0000","last_became_peered":"2024-05-03T21:18:42.816247+0000","last_unstale":"2024-05-03T21:19:34.973800+0000","last_undegraded":"2024-05-03T21:19:34.973800+0000","last_fullsized":"2024-05-03T21:19:34.973800+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_clean_scrub_stamp":"2024-05-03T21:18:41.788726+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T00:09:02.620176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":20,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-05-03T21:19:34.973840+0000","last_change":"2024-05-03T21:18:42.821587+0000","last_active":"2024-05-03T21:19:34.973840+0000","last_peered":"2024-05-03T21:19:34.973840+0000","last_clean":"2024-05-03T21:19:34.973840+0000","last_became_active":"2024-05-03T21:18:42.821236+0000","last_became_peered":"2024-05-03T21:18:42.821236+0000","last_unstale":"2024-05-03T21:19:34.973840+0000","last_undegraded":"2024-05-03T21:19:34.973840+0000","last_fullsized":"2024-05-03T21:19:34.973840+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_clean_scrub_stamp":"2024-05-03T21:18:41.788726+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-04T22:23:46.746787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"82'9","reported_seq":35,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-05-03T21:19:34.974229+0000","last_change":"2024-05-03T21:18:39.792001+0000","last_active":"2024-05-03T21:19:34.974229+0000","last_peered":"2024-05-03T21:19:34.974229+0000","last_clean":"2024-05-03T21:19:34.974229+0000","last_became_active":"2024-05-03T21:18:39.791644+0000","last_became_peered":"2024-05-03T21:18:39.791644+0000","last_unstale":"2024-05-03T21:19:34.974229+0000","last_undegraded":"2024-05-03T21:19:34.974229+0000","last_fullsized":"2024-05-03T21:19:34.974229+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T21:18:38.766727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T21:18:38.766727+0000","last_clean_scrub_stamp":"2024-05-03T21:18:38.766727+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-04T23:15:36.169243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":20,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-05-03T21:19:34.979208+0000","last_change":"2024-05-03T21:18:42.815115+0000","last_active":"2024-05-03T21:19:34.979208+0000","last_peered":"2024-05-03T21:19:34.979208+0000","last_clean":"2024-05-03T21:19:34.979208+0000","last_became_active":"2024-05-03T21:18:42.814734+0000","last_became_peered":"2024-05-03T21:18:42.814734+0000","last_unstale":"2024-05-03T21:19:34.979208+0000","last_undegraded":"2024-05-03T21:19:34.979208+0000","last_fullsized":"2024-05-03T21:19:34.979208+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_clean_scrub_stamp":"2024-05-03T21:18:41.788726+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-04T23:25:18.232982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"82'2","reported_seq":27,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-05-03T21:19:34.979497+0000","last_change":"2024-05-03T21:18:39.814107+0000","last_active":"2024-05-03T21:19:34.979497+0000","last_peered":"2024-05-03T21:19:34.979497+0000","last_clean":"2024-05-03T21:19:34.979497+0000","last_became_active":"2024-05-03T21:18:39.813615+0000","last_became_peered":"2024-05-03T21:18:39.813615+0000","last_unstale":"2024-05-03T21:19:34.979497+0000","last_undegraded":"2024-05-03T21:19:34.979497+0000","last_fullsized":"2024-05-03T21:19:34.979497+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T21:18:38.766727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T21:18:38.766727+0000","last_clean_scrub_stamp":"2024-05-03T21:18:38.766727+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-04T23:24:00.914805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":20,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-05-03T21:19:34.978900+0000","last_change":"2024-05-03T21:18:42.816925+0000","last_active":"2024-05-03T21:19:34.978900+0000","last_peered":"2024-05-03T21:19:34.978900+0000","last_clean":"2024-05-03T21:19:34.978900+0000","last_became_active":"2024-05-03T21:18:42.814731+0000","last_became_peered":"2024-05-03T21:18:42.814731+0000","last_unstale":"2024-05-03T21:19:34.978900+0000","last_undegraded":"2024-05-03T21:19:34.978900+0000","last_fullsized":"2024-05-03T21:19:34.978900+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_clean_scrub_stamp":"2024-05-03T21:18:41.788726+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T00:18:26.859784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"82'4","reported_seq":30,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-05-03T21:19:34.973033+0000","last_change":"2024-05-03T21:18:39.792101+0000","last_active":"2024-05-03T21:19:34.973033+0000","last_peered":"2024-05-03T21:19:34.973033+0000","last_clean":"2024-05-03T21:19:34.973033+0000","last_became_active":"2024-05-03T21:18:39.791725+0000","last_became_peered":"2024-05-03T21:18:39.791725+0000","last_unstale":"2024-05-03T21:19:34.973033+0000","last_undegraded":"2024-05-03T21:19:34.973033+0000","last_fullsized":"2024-05-03T21:19:34.973033+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T21:18:38.766727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T21:18:38.766727+0000","last_clean_scrub_stamp":"2024-05-03T21:18:38.766727+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T05:09:42.900318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":20,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-05-03T21:19:34.971846+0000","last_change":"2024-05-03T21:18:42.823318+0000","last_active":"2024-05-03T21:19:34.971846+0000","last_peered":"2024-05-03T21:19:34.971846+0000","last_clean":"2024-05-03T21:19:34.971846+0000","last_became_active":"2024-05-03T21:18:42.823011+0000","last_became_peered":"2024-05-03T21:18:42.823011+0000","last_unstale":"2024-05-03T21:19:34.971846+0000","last_undegraded":"2024-05-03T21:19:34.971846+0000","last_fullsized":"2024-05-03T21:19:34.971846+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_clean_scrub_stamp":"2024-05-03T21:18:41.788726+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T01:04:51.967269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"82'2","reported_seq":28,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-05-03T21:19:34.973609+0000","last_change":"2024-05-03T21:18:39.787730+0000","last_active":"2024-05-03T21:19:34.973609+0000","last_peered":"2024-05-03T21:19:34.973609+0000","last_clean":"2024-05-03T21:19:34.973609+0000","last_became_active":"2024-05-03T21:18:39.787403+0000","last_became_peered":"2024-05-03T21:18:39.787403+0000","last_unstale":"2024-05-03T21:19:34.973609+0000","last_undegraded":"2024-05-03T21:19:34.973609+0000","last_fullsized":"2024-05-03T21:19:34.973609+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T21:18:38.766727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T21:18:38.766727+0000","last_clean_scrub_stamp":"2024-05-03T21:18:38.766727+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T06:28:41.454461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":20,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-05-03T21:19:34.971641+0000","last_change":"2024-05-03T21:18:42.818987+0000","last_active":"2024-05-03T21:19:34.971641+0000","last_peered":"2024-05-03T21:19:34.971641+0000","last_clean":"2024-05-03T21:19:34.971641+0000","last_became_active":"2024-05-03T21:18:42.818645+0000","last_became_peered":"2024-05-03T21:18:42.818645+0000","last_unstale":"2024-05-03T21:19:34.971641+0000","last_undegraded":"2024-05-03T21:19:34.971641+0000","last_fullsized":"2024-05-03T21:19:34.971641+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_clean_scrub_stamp":"2024-05-03T21:18:41.788726+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T02:26:27.470390+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"82'6","reported_seq":127,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-05-03T21:20:31.023908+0000","last_change":"2024-05-03T21:18:39.793807+0000","last_active":"2024-05-03T21:20:31.023908+0000","last_peered":"2024-05-03T21:20:31.023908+0000","last_clean":"2024-05-03T21:20:31.023908+0000","last_became_active":"2024-05-03T21:18:39.793447+0000","last_became_peered":"2024-05-03T21:18:39.793447+0000","last_unstale":"2024-05-03T21:20:31.023908+0000","last_undegraded":"2024-05-03T21:20:31.023908+0000","last_fullsized":"2024-05-03T21:20:31.023908+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T21:18:38.766727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T21:18:38.766727+0000","last_clean_scrub_stamp":"2024-05-03T21:18:38.766727+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T06:44:55.036889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1013,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":95,"num_read_kb":3,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":20,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-05-03T21:19:34.972280+0000","last_change":"2024-05-03T21:18:42.807486+0000","last_active":"2024-05-03T21:19:34.972280+0000","last_peered":"2024-05-03T21:19:34.972280+0000","last_clean":"2024-05-03T21:19:34.972280+0000","last_became_active":"2024-05-03T21:18:42.807120+0000","last_became_peered":"2024-05-03T21:18:42.807120+0000","last_unstale":"2024-05-03T21:19:34.972280+0000","last_undegraded":"2024-05-03T21:19:34.972280+0000","last_fullsized":"2024-05-03T21:19:34.972280+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_clean_scrub_stamp":"2024-05-03T21:18:41.788726+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T05:32:38.621330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1a","version":"82'3","reported_seq":28,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-05-03T21:19:35.208026+0000","last_change":"2024-05-03T21:18:39.791943+0000","last_active":"2024-05-03T21:19:35.208026+0000","last_peered":"2024-05-03T21:19:35.208026+0000","last_clean":"2024-05-03T21:19:35.208026+0000","last_became_active":"2024-05-03T21:18:39.791587+0000","last_became_peered":"2024-05-03T21:18:39.791587+0000","last_unstale":"2024-05-03T21:19:35.208026+0000","last_undegraded":"2024-05-03T21:19:35.208026+0000","last_fullsized":"2024-05-03T21:19:35.208026+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T21:18:38.766727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T21:18:38.766727+0000","last_clean_scrub_stamp":"2024-05-03T21:18:38.766727+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T05:01:39.458019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":11,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-05-03T21:19:35.210398+0000","last_change":"2024-05-03T21:19:35.209904+0000","last_active":"2024-05-03T21:19:35.210398+0000","last_peered":"2024-05-03T21:19:35.210398+0000","last_clean":"2024-05-03T21:19:35.210398+0000","last_became_active":"2024-05-03T21:19:35.209475+0000","last_became_peered":"2024-05-03T21:19:35.209475+0000","last_unstale":"2024-05-03T21:19:35.210398+0000","last_undegraded":"2024-05-03T21:19:35.210398+0000","last_fullsized":"2024-05-03T21:19:35.210398+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_clean_scrub_stamp":"2024-05-03T21:18:41.788726+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T09:14:24.101832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4,3],"acting":[9,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.19","version":"82'2","reported_seq":27,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-05-03T21:19:34.973825+0000","last_change":"2024-05-03T21:18:39.784859+0000","last_active":"2024-05-03T21:19:34.973825+0000","last_peered":"2024-05-03T21:19:34.973825+0000","last_clean":"2024-05-03T21:19:34.973825+0000","last_became_active":"2024-05-03T21:18:39.784428+0000","last_became_peered":"2024-05-03T21:18:39.784428+0000","last_unstale":"2024-05-03T21:19:34.973825+0000","last_undegraded":"2024-05-03T21:19:34.973825+0000","last_fullsized":"2024-05-03T21:19:34.973825+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T21:18:38.766727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T21:18:38.766727+0000","last_clean_scrub_stamp":"2024-05-03T21:18:38.766727+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T05:56:18.368399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":20,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-05-03T21:19:34.978436+0000","last_change":"2024-05-03T21:18:42.813285+0000","last_active":"2024-05-03T21:19:34.978436+0000","last_peered":"2024-05-03T21:19:34.978436+0000","last_clean":"2024-05-03T21:19:34.978436+0000","last_became_active":"2024-05-03T21:18:42.812954+0000","last_became_peered":"2024-05-03T21:18:42.812954+0000","last_unstale":"2024-05-03T21:19:34.978436+0000","last_undegraded":"2024-05-03T21:19:34.978436+0000","last_fullsized":"2024-05-03T21:19:34.978436+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_clean_scrub_stamp":"2024-05-03T21:18:41.788726+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T00:19:26.841514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"82'1","reported_seq":27,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-05-03T21:19:34.971312+0000","last_change":"2024-05-03T21:18:39.793485+0000","last_active":"2024-05-03T21:19:34.971312+0000","last_peered":"2024-05-03T21:19:34.971312+0000","last_clean":"2024-05-03T21:19:34.971312+0000","last_became_active":"2024-05-03T21:18:39.793055+0000","last_became_peered":"2024-05-03T21:18:39.793055+0000","last_unstale":"2024-05-03T21:19:34.971312+0000","last_undegraded":"2024-05-03T21:19:34.971312+0000","last_fullsized":"2024-05-03T21:19:34.971312+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T21:18:38.766727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T21:18:38.766727+0000","last_clean_scrub_stamp":"2024-05-03T21:18:38.766727+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T03:10:59.880330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":20,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-05-03T21:19:34.973551+0000","last_change":"2024-05-03T21:18:42.815721+0000","last_active":"2024-05-03T21:19:34.973551+0000","last_peered":"2024-05-03T21:19:34.973551+0000","last_clean":"2024-05-03T21:19:34.973551+0000","last_became_active":"2024-05-03T21:18:42.815408+0000","last_became_peered":"2024-05-03T21:18:42.815408+0000","last_unstale":"2024-05-03T21:19:34.973551+0000","last_undegraded":"2024-05-03T21:19:34.973551+0000","last_fullsized":"2024-05-03T21:19:34.973551+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_clean_scrub_stamp":"2024-05-03T21:18:41.788726+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T03:03:46.851667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"82'3","reported_seq":123,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-05-03T21:20:34.027618+0000","last_change":"2024-05-03T21:18:39.792060+0000","last_active":"2024-05-03T21:20:34.027618+0000","last_peered":"2024-05-03T21:20:34.027618+0000","last_clean":"2024-05-03T21:20:34.027618+0000","last_became_active":"2024-05-03T21:18:39.791704+0000","last_became_peered":"2024-05-03T21:18:39.791704+0000","last_unstale":"2024-05-03T21:20:34.027618+0000","last_undegraded":"2024-05-03T21:20:34.027618+0000","last_fullsized":"2024-05-03T21:20:34.027618+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T21:18:38.766727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T21:18:38.766727+0000","last_clean_scrub_stamp":"2024-05-03T21:18:38.766727+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T03:46:43.784554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":95,"num_read_kb":95,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":20,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-05-03T21:19:34.972208+0000","last_change":"2024-05-03T21:18:42.823016+0000","last_active":"2024-05-03T21:19:34.972208+0000","last_peered":"2024-05-03T21:19:34.972208+0000","last_clean":"2024-05-03T21:19:34.972208+0000","last_became_active":"2024-05-03T21:18:42.822363+0000","last_became_peered":"2024-05-03T21:18:42.822363+0000","last_unstale":"2024-05-03T21:19:34.972208+0000","last_undegraded":"2024-05-03T21:19:34.972208+0000","last_fullsized":"2024-05-03T21:19:34.972208+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_clean_scrub_stamp":"2024-05-03T21:18:41.788726+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T07:46:36.213746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"82'2","reported_seq":27,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-05-03T21:19:34.974314+0000","last_change":"2024-05-03T21:18:39.791944+0000","last_active":"2024-05-03T21:19:34.974314+0000","last_peered":"2024-05-03T21:19:34.974314+0000","last_clean":"2024-05-03T21:19:34.974314+0000","last_became_active":"2024-05-03T21:18:39.791576+0000","last_became_peered":"2024-05-03T21:18:39.791576+0000","last_unstale":"2024-05-03T21:19:34.974314+0000","last_undegraded":"2024-05-03T21:19:34.974314+0000","last_fullsized":"2024-05-03T21:19:34.974314+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T21:18:38.766727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T21:18:38.766727+0000","last_clean_scrub_stamp":"2024-05-03T21:18:38.766727+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T07:05:58.190107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":20,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-05-03T21:19:34.978861+0000","last_change":"2024-05-03T21:18:42.819587+0000","last_active":"2024-05-03T21:19:34.978861+0000","last_peered":"2024-05-03T21:19:34.978861+0000","last_clean":"2024-05-03T21:19:34.978861+0000","last_became_active":"2024-05-03T21:18:42.819286+0000","last_became_peered":"2024-05-03T21:18:42.819286+0000","last_unstale":"2024-05-03T21:19:34.978861+0000","last_undegraded":"2024-05-03T21:19:34.978861+0000","last_fullsized":"2024-05-03T21:19:34.978861+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_clean_scrub_stamp":"2024-05-03T21:18:41.788726+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T01:31:34.367410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"82'4","reported_seq":31,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-05-03T21:19:34.971743+0000","last_change":"2024-05-03T21:18:39.815394+0000","last_active":"2024-05-03T21:19:34.971743+0000","last_peered":"2024-05-03T21:19:34.971743+0000","last_clean":"2024-05-03T21:19:34.971743+0000","last_became_active":"2024-05-03T21:18:39.814874+0000","last_became_peered":"2024-05-03T21:18:39.814874+0000","last_unstale":"2024-05-03T21:19:34.971743+0000","last_undegraded":"2024-05-03T21:19:34.971743+0000","last_fullsized":"2024-05-03T21:19:34.971743+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T21:18:38.766727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T21:18:38.766727+0000","last_clean_scrub_stamp":"2024-05-03T21:18:38.766727+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T08:42:50.532572+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":20,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-05-03T21:19:34.971885+0000","last_change":"2024-05-03T21:18:42.804483+0000","last_active":"2024-05-03T21:19:34.971885+0000","last_peered":"2024-05-03T21:19:34.971885+0000","last_clean":"2024-05-03T21:19:34.971885+0000","last_became_active":"2024-05-03T21:18:42.804064+0000","last_became_peered":"2024-05-03T21:18:42.804064+0000","last_unstale":"2024-05-03T21:19:34.971885+0000","last_undegraded":"2024-05-03T21:19:34.971885+0000","last_fullsized":"2024-05-03T21:19:34.971885+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_clean_scrub_stamp":"2024-05-03T21:18:41.788726+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T01:02:22.249152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"82'27","reported_seq":249,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-05-03T21:20:35.029363+0000","last_change":"2024-05-03T21:18:39.792864+0000","last_active":"2024-05-03T21:20:35.029363+0000","last_peered":"2024-05-03T21:20:35.029363+0000","last_clean":"2024-05-03T21:20:35.029363+0000","last_became_active":"2024-05-03T21:18:39.792469+0000","last_became_peered":"2024-05-03T21:18:39.792469+0000","last_unstale":"2024-05-03T21:20:35.029363+0000","last_undegraded":"2024-05-03T21:20:35.029363+0000","last_fullsized":"2024-05-03T21:20:35.029363+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T21:18:38.766727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T21:18:38.766727+0000","last_clean_scrub_stamp":"2024-05-03T21:18:38.766727+0000","objects_scrubbed":0,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-04T22:31:09.180649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51762,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":197,"num_read_kb":148,"num_write":28,"num_write_kb":59,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":20,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-05-03T21:19:34.972725+0000","last_change":"2024-05-03T21:18:42.816299+0000","last_active":"2024-05-03T21:19:34.972725+0000","last_peered":"2024-05-03T21:19:34.972725+0000","last_clean":"2024-05-03T21:19:34.972725+0000","last_became_active":"2024-05-03T21:18:42.815947+0000","last_became_peered":"2024-05-03T21:18:42.815947+0000","last_unstale":"2024-05-03T21:19:34.972725+0000","last_undegraded":"2024-05-03T21:19:34.972725+0000","last_fullsized":"2024-05-03T21:19:34.972725+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_clean_scrub_stamp":"2024-05-03T21:18:41.788726+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T07:52:21.620885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"82'6","reported_seq":32,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-05-03T21:19:34.973877+0000","last_change":"2024-05-03T21:18:39.794552+0000","last_active":"2024-05-03T21:19:34.973877+0000","last_peered":"2024-05-03T21:19:34.973877+0000","last_clean":"2024-05-03T21:19:34.973877+0000","last_became_active":"2024-05-03T21:18:39.794183+0000","last_became_peered":"2024-05-03T21:18:39.794183+0000","last_unstale":"2024-05-03T21:19:34.973877+0000","last_undegraded":"2024-05-03T21:19:34.973877+0000","last_fullsized":"2024-05-03T21:19:34.973877+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T21:18:38.766727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T21:18:38.766727+0000","last_clean_scrub_stamp":"2024-05-03T21:18:38.766727+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T00:16:32.820261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":20,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-05-03T21:19:34.973435+0000","last_change":"2024-05-03T21:18:42.811542+0000","last_active":"2024-05-03T21:19:34.973435+0000","last_peered":"2024-05-03T21:19:34.973435+0000","last_clean":"2024-05-03T21:19:34.973435+0000","last_became_active":"2024-05-03T21:18:42.810301+0000","last_became_peered":"2024-05-03T21:18:42.810301+0000","last_unstale":"2024-05-03T21:19:34.973435+0000","last_undegraded":"2024-05-03T21:19:34.973435+0000","last_fullsized":"2024-05-03T21:19:34.973435+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_clean_scrub_stamp":"2024-05-03T21:18:41.788726+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T03:01:30.503201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"82'3","reported_seq":29,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-05-03T21:19:34.971818+0000","last_change":"2024-05-03T21:18:39.796028+0000","last_active":"2024-05-03T21:19:34.971818+0000","last_peered":"2024-05-03T21:19:34.971818+0000","last_clean":"2024-05-03T21:19:34.971818+0000","last_became_active":"2024-05-03T21:18:39.795646+0000","last_became_peered":"2024-05-03T21:18:39.795646+0000","last_unstale":"2024-05-03T21:19:34.971818+0000","last_undegraded":"2024-05-03T21:19:34.971818+0000","last_fullsized":"2024-05-03T21:19:34.971818+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T21:18:38.766727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T21:18:38.766727+0000","last_clean_scrub_stamp":"2024-05-03T21:18:38.766727+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T01:56:31.128184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":20,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-05-03T21:19:34.974135+0000","last_change":"2024-05-03T21:18:42.810690+0000","last_active":"2024-05-03T21:19:34.974135+0000","last_peered":"2024-05-03T21:19:34.974135+0000","last_clean":"2024-05-03T21:19:34.974135+0000","last_became_active":"2024-05-03T21:18:42.810397+0000","last_became_peered":"2024-05-03T21:18:42.810397+0000","last_unstale":"2024-05-03T21:19:34.974135+0000","last_undegraded":"2024-05-03T21:19:34.974135+0000","last_fullsized":"2024-05-03T21:19:34.974135+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_clean_scrub_stamp":"2024-05-03T21:18:41.788726+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T09:14:43.292353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"82'4","reported_seq":125,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-05-03T21:20:31.024031+0000","last_change":"2024-05-03T21:18:39.793102+0000","last_active":"2024-05-03T21:20:31.024031+0000","last_peered":"2024-05-03T21:20:31.024031+0000","last_clean":"2024-05-03T21:20:31.024031+0000","last_became_active":"2024-05-03T21:18:39.792700+0000","last_became_peered":"2024-05-03T21:18:39.792700+0000","last_unstale":"2024-05-03T21:20:31.024031+0000","last_undegraded":"2024-05-03T21:20:31.024031+0000","last_fullsized":"2024-05-03T21:20:31.024031+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T21:18:38.766727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T21:18:38.766727+0000","last_clean_scrub_stamp":"2024-05-03T21:18:38.766727+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-04T21:32:36.761116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":979,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":4,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":20,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-05-03T21:19:34.974364+0000","last_change":"2024-05-03T21:18:42.815975+0000","last_active":"2024-05-03T21:19:34.974364+0000","last_peered":"2024-05-03T21:19:34.974364+0000","last_clean":"2024-05-03T21:19:34.974364+0000","last_became_active":"2024-05-03T21:18:42.814667+0000","last_became_peered":"2024-05-03T21:18:42.814667+0000","last_unstale":"2024-05-03T21:19:34.974364+0000","last_undegraded":"2024-05-03T21:19:34.974364+0000","last_fullsized":"2024-05-03T21:19:34.974364+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_clean_scrub_stamp":"2024-05-03T21:18:41.788726+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T00:50:47.991863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"82'15","reported_seq":29,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-05-03T21:19:34.995443+0000","last_change":"2024-05-03T21:19:34.994980+0000","last_active":"2024-05-03T21:19:34.995443+0000","last_peered":"2024-05-03T21:19:34.995443+0000","last_clean":"2024-05-03T21:19:34.995443+0000","last_became_active":"2024-05-03T21:19:34.979988+0000","last_became_peered":"2024-05-03T21:19:34.979988+0000","last_unstale":"2024-05-03T21:19:34.995443+0000","last_undegraded":"2024-05-03T21:19:34.995443+0000","last_fullsized":"2024-05-03T21:19:34.995443+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T21:18:38.766727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T21:18:38.766727+0000","last_clean_scrub_stamp":"2024-05-03T21:18:38.766727+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T01:14:44.807458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":112,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":20,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-05-03T21:19:34.971435+0000","last_change":"2024-05-03T21:18:42.810313+0000","last_active":"2024-05-03T21:19:34.971435+0000","last_peered":"2024-05-03T21:19:34.971435+0000","last_clean":"2024-05-03T21:19:34.971435+0000","last_became_active":"2024-05-03T21:18:42.809732+0000","last_became_peered":"2024-05-03T21:18:42.809732+0000","last_unstale":"2024-05-03T21:19:34.971435+0000","last_undegraded":"2024-05-03T21:19:34.971435+0000","last_fullsized":"2024-05-03T21:19:34.971435+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_clean_scrub_stamp":"2024-05-03T21:18:41.788726+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T01:36:54.077645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"82'2","reported_seq":29,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-05-03T21:19:34.972521+0000","last_change":"2024-05-03T21:18:39.784067+0000","last_active":"2024-05-03T21:19:34.972521+0000","last_peered":"2024-05-03T21:19:34.972521+0000","last_clean":"2024-05-03T21:19:34.972521+0000","last_became_active":"2024-05-03T21:18:39.783596+0000","last_became_peered":"2024-05-03T21:18:39.783596+0000","last_unstale":"2024-05-03T21:19:34.972521+0000","last_undegraded":"2024-05-03T21:19:34.972521+0000","last_fullsized":"2024-05-03T21:19:34.972521+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T21:18:38.766727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T21:18:38.766727+0000","last_clean_scrub_stamp":"2024-05-03T21:18:38.766727+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T03:10:25.925757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":20,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-05-03T21:19:34.974825+0000","last_change":"2024-05-03T21:18:42.817617+0000","last_active":"2024-05-03T21:19:34.974825+0000","last_peered":"2024-05-03T21:19:34.974825+0000","last_clean":"2024-05-03T21:19:34.974825+0000","last_became_active":"2024-05-03T21:18:42.817267+0000","last_became_peered":"2024-05-03T21:18:42.817267+0000","last_unstale":"2024-05-03T21:19:34.974825+0000","last_undegraded":"2024-05-03T21:19:34.974825+0000","last_fullsized":"2024-05-03T21:19:34.974825+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_clean_scrub_stamp":"2024-05-03T21:18:41.788726+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T08:00:12.864215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":25,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-05-03T21:19:34.973855+0000","last_change":"2024-05-03T21:18:39.814213+0000","last_active":"2024-05-03T21:19:34.973855+0000","last_peered":"2024-05-03T21:19:34.973855+0000","last_clean":"2024-05-03T21:19:34.973855+0000","last_became_active":"2024-05-03T21:18:39.813787+0000","last_became_peered":"2024-05-03T21:18:39.813787+0000","last_unstale":"2024-05-03T21:19:34.973855+0000","last_undegraded":"2024-05-03T21:19:34.973855+0000","last_fullsized":"2024-05-03T21:19:34.973855+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T21:18:38.766727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T21:18:38.766727+0000","last_clean_scrub_stamp":"2024-05-03T21:18:38.766727+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T08:45:17.963396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"82'1","reported_seq":21,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-05-03T21:19:35.208354+0000","last_change":"2024-05-03T21:18:42.821547+0000","last_active":"2024-05-03T21:19:35.208354+0000","last_peered":"2024-05-03T21:19:35.208354+0000","last_clean":"2024-05-03T21:19:35.208354+0000","last_became_active":"2024-05-03T21:18:42.821168+0000","last_became_peered":"2024-05-03T21:18:42.821168+0000","last_unstale":"2024-05-03T21:19:35.208354+0000","last_undegraded":"2024-05-03T21:19:35.208354+0000","last_fullsized":"2024-05-03T21:19:35.208354+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_clean_scrub_stamp":"2024-05-03T21:18:41.788726+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T07:28:25.037648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"82'1","reported_seq":26,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-05-03T21:19:34.973954+0000","last_change":"2024-05-03T21:18:39.792994+0000","last_active":"2024-05-03T21:19:34.973954+0000","last_peered":"2024-05-03T21:19:34.973954+0000","last_clean":"2024-05-03T21:19:34.973954+0000","last_became_active":"2024-05-03T21:18:39.792125+0000","last_became_peered":"2024-05-03T21:18:39.792125+0000","last_unstale":"2024-05-03T21:19:34.973954+0000","last_undegraded":"2024-05-03T21:19:34.973954+0000","last_fullsized":"2024-05-03T21:19:34.973954+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T21:18:38.766727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T21:18:38.766727+0000","last_clean_scrub_stamp":"2024-05-03T21:18:38.766727+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T03:09:45.917263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":20,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-05-03T21:19:35.208007+0000","last_change":"2024-05-03T21:18:42.814513+0000","last_active":"2024-05-03T21:19:35.208007+0000","last_peered":"2024-05-03T21:19:35.208007+0000","last_clean":"2024-05-03T21:19:35.208007+0000","last_became_active":"2024-05-03T21:18:42.814097+0000","last_became_peered":"2024-05-03T21:18:42.814097+0000","last_unstale":"2024-05-03T21:19:35.208007+0000","last_undegraded":"2024-05-03T21:19:35.208007+0000","last_fullsized":"2024-05-03T21:19:35.208007+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_clean_scrub_stamp":"2024-05-03T21:18:41.788726+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T06:52:46.731543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":20,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-05-03T21:19:34.971544+0000","last_change":"2024-05-03T21:18:42.812413+0000","last_active":"2024-05-03T21:19:34.971544+0000","last_peered":"2024-05-03T21:19:34.971544+0000","last_clean":"2024-05-03T21:19:34.971544+0000","last_became_active":"2024-05-03T21:18:42.812065+0000","last_became_peered":"2024-05-03T21:18:42.812065+0000","last_unstale":"2024-05-03T21:19:34.971544+0000","last_undegraded":"2024-05-03T21:19:34.971544+0000","last_fullsized":"2024-05-03T21:19:34.971544+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_clean_scrub_stamp":"2024-05-03T21:18:41.788726+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T07:52:44.489097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"82'4","reported_seq":121,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-05-03T21:20:31.023334+0000","last_change":"2024-05-03T21:18:39.813982+0000","last_active":"2024-05-03T21:20:31.023334+0000","last_peered":"2024-05-03T21:20:31.023334+0000","last_clean":"2024-05-03T21:20:31.023334+0000","last_became_active":"2024-05-03T21:18:39.813590+0000","last_became_peered":"2024-05-03T21:18:39.813590+0000","last_unstale":"2024-05-03T21:20:31.023334+0000","last_undegraded":"2024-05-03T21:20:31.023334+0000","last_fullsized":"2024-05-03T21:20:31.023334+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T21:18:38.766727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T21:18:38.766727+0000","last_clean_scrub_stamp":"2024-05-03T21:18:38.766727+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T02:30:24.733268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":92,"num_read_kb":92,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":20,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-05-03T21:19:34.978749+0000","last_change":"2024-05-03T21:18:42.808357+0000","last_active":"2024-05-03T21:19:34.978749+0000","last_peered":"2024-05-03T21:19:34.978749+0000","last_clean":"2024-05-03T21:19:34.978749+0000","last_became_active":"2024-05-03T21:18:42.808023+0000","last_became_peered":"2024-05-03T21:18:42.808023+0000","last_unstale":"2024-05-03T21:19:34.978749+0000","last_undegraded":"2024-05-03T21:19:34.978749+0000","last_fullsized":"2024-05-03T21:19:34.978749+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_clean_scrub_stamp":"2024-05-03T21:18:41.788726+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T08:33:11.713491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"82'3","reported_seq":31,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-05-03T21:19:35.208863+0000","last_change":"2024-05-03T21:18:39.787052+0000","last_active":"2024-05-03T21:19:35.208863+0000","last_peered":"2024-05-03T21:19:35.208863+0000","last_clean":"2024-05-03T21:19:35.208863+0000","last_became_active":"2024-05-03T21:18:39.786390+0000","last_became_peered":"2024-05-03T21:18:39.786390+0000","last_unstale":"2024-05-03T21:19:35.208863+0000","last_undegraded":"2024-05-03T21:19:35.208863+0000","last_fullsized":"2024-05-03T21:19:35.208863+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T21:18:38.766727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T21:18:38.766727+0000","last_clean_scrub_stamp":"2024-05-03T21:18:38.766727+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T08:18:27.832378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"21'92","reported_seq":100,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-05-03T21:19:34.974544+0000","last_change":"2024-05-03T21:17:17.301485+0000","last_active":"2024-05-03T21:19:34.974544+0000","last_peered":"2024-05-03T21:19:34.974544+0000","last_clean":"2024-05-03T21:19:34.974544+0000","last_became_active":"2024-05-03T21:17:17.296458+0000","last_became_peered":"2024-05-03T21:17:17.296458+0000","last_unstale":"2024-05-03T21:19:34.974544+0000","last_undegraded":"2024-05-03T21:19:34.974544+0000","last_fullsized":"2024-05-03T21:19:34.974544+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T21:14:00.058823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T21:14:00.058823+0000","last_clean_scrub_stamp":"2024-05-03T21:14:00.058823+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T03:13:14.815888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":26,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-05-03T21:19:34.982618+0000","last_change":"2024-05-03T21:19:34.982618+0000","last_active":"2024-05-03T21:19:34.982618+0000","last_peered":"2024-05-03T21:19:34.982618+0000","last_clean":"2024-05-03T21:19:34.982618+0000","last_became_active":"2024-05-03T21:19:34.982259+0000","last_became_peered":"2024-05-03T21:19:34.982259+0000","last_unstale":"2024-05-03T21:19:34.982618+0000","last_undegraded":"2024-05-03T21:19:34.982618+0000","last_fullsized":"2024-05-03T21:19:34.982618+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_clean_scrub_stamp":"2024-05-03T21:18:41.788726+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-04T23:33:44.083386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,9,2],"acting":[6,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"82'3","reported_seq":38,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-05-03T21:19:35.213111+0000","last_change":"2024-05-03T21:19:35.213111+0000","last_active":"2024-05-03T21:19:35.213111+0000","last_peered":"2024-05-03T21:19:35.213111+0000","last_clean":"2024-05-03T21:19:35.213111+0000","last_became_active":"2024-05-03T21:19:35.210041+0000","last_became_peered":"2024-05-03T21:19:35.210041+0000","last_unstale":"2024-05-03T21:19:35.213111+0000","last_undegraded":"2024-05-03T21:19:35.213111+0000","last_fullsized":"2024-05-03T21:19:35.213111+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T21:18:38.766727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T21:18:38.766727+0000","last_clean_scrub_stamp":"2024-05-03T21:18:38.766727+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T02:40:22.026186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,4],"acting":[7,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"82'1","reported_seq":28,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-05-03T21:19:34.972907+0000","last_change":"2024-05-03T21:18:39.814671+0000","last_active":"2024-05-03T21:19:34.972907+0000","last_peered":"2024-05-03T21:19:34.972907+0000","last_clean":"2024-05-03T21:19:34.972907+0000","last_became_active":"2024-05-03T21:18:39.814326+0000","last_became_peered":"2024-05-03T21:18:39.814326+0000","last_unstale":"2024-05-03T21:19:34.972907+0000","last_undegraded":"2024-05-03T21:19:34.972907+0000","last_fullsized":"2024-05-03T21:19:34.972907+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T21:18:38.766727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T21:18:38.766727+0000","last_clean_scrub_stamp":"2024-05-03T21:18:38.766727+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T01:21:53.058203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":20,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-05-03T21:19:34.973632+0000","last_change":"2024-05-03T21:18:42.815525+0000","last_active":"2024-05-03T21:19:34.973632+0000","last_peered":"2024-05-03T21:19:34.973632+0000","last_clean":"2024-05-03T21:19:34.973632+0000","last_became_active":"2024-05-03T21:18:42.815209+0000","last_became_peered":"2024-05-03T21:18:42.815209+0000","last_unstale":"2024-05-03T21:19:34.973632+0000","last_undegraded":"2024-05-03T21:19:34.973632+0000","last_fullsized":"2024-05-03T21:19:34.973632+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_clean_scrub_stamp":"2024-05-03T21:18:41.788726+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T01:02:51.545090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"82'9","reported_seq":34,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-05-03T21:19:34.973901+0000","last_change":"2024-05-03T21:18:39.784592+0000","last_active":"2024-05-03T21:19:34.973901+0000","last_peered":"2024-05-03T21:19:34.973901+0000","last_clean":"2024-05-03T21:19:34.973901+0000","last_became_active":"2024-05-03T21:18:39.784242+0000","last_became_peered":"2024-05-03T21:18:39.784242+0000","last_unstale":"2024-05-03T21:19:34.973901+0000","last_undegraded":"2024-05-03T21:19:34.973901+0000","last_fullsized":"2024-05-03T21:19:34.973901+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T21:18:38.766727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T21:18:38.766727+0000","last_clean_scrub_stamp":"2024-05-03T21:18:38.766727+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-04T22:23:01.914553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":20,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-05-03T21:19:34.979646+0000","last_change":"2024-05-03T21:18:42.814361+0000","last_active":"2024-05-03T21:19:34.979646+0000","last_peered":"2024-05-03T21:19:34.979646+0000","last_clean":"2024-05-03T21:19:34.979646+0000","last_became_active":"2024-05-03T21:18:42.813448+0000","last_became_peered":"2024-05-03T21:18:42.813448+0000","last_unstale":"2024-05-03T21:19:34.979646+0000","last_undegraded":"2024-05-03T21:19:34.979646+0000","last_fullsized":"2024-05-03T21:19:34.979646+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_clean_scrub_stamp":"2024-05-03T21:18:41.788726+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T06:21:11.974428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"82'4","reported_seq":30,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-05-03T21:19:35.209116+0000","last_change":"2024-05-03T21:18:39.905533+0000","last_active":"2024-05-03T21:19:35.209116+0000","last_peered":"2024-05-03T21:19:35.209116+0000","last_clean":"2024-05-03T21:19:35.209116+0000","last_became_active":"2024-05-03T21:18:39.905256+0000","last_became_peered":"2024-05-03T21:18:39.905256+0000","last_unstale":"2024-05-03T21:19:35.209116+0000","last_undegraded":"2024-05-03T21:19:35.209116+0000","last_fullsized":"2024-05-03T21:19:35.209116+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T21:18:38.766727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T21:18:38.766727+0000","last_clean_scrub_stamp":"2024-05-03T21:18:38.766727+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T07:07:12.162628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":20,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-05-03T21:19:34.979935+0000","last_change":"2024-05-03T21:18:42.821052+0000","last_active":"2024-05-03T21:19:34.979935+0000","last_peered":"2024-05-03T21:19:34.979935+0000","last_clean":"2024-05-03T21:19:34.979935+0000","last_became_active":"2024-05-03T21:18:42.820751+0000","last_became_peered":"2024-05-03T21:18:42.820751+0000","last_unstale":"2024-05-03T21:19:34.979935+0000","last_undegraded":"2024-05-03T21:19:34.979935+0000","last_fullsized":"2024-05-03T21:19:34.979935+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_clean_scrub_stamp":"2024-05-03T21:18:41.788726+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T02:14:45.829778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"82'2","reported_seq":28,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-05-03T21:19:34.971736+0000","last_change":"2024-05-03T21:18:39.814793+0000","last_active":"2024-05-03T21:19:34.971736+0000","last_peered":"2024-05-03T21:19:34.971736+0000","last_clean":"2024-05-03T21:19:34.971736+0000","last_became_active":"2024-05-03T21:18:39.814427+0000","last_became_peered":"2024-05-03T21:18:39.814427+0000","last_unstale":"2024-05-03T21:19:34.971736+0000","last_undegraded":"2024-05-03T21:19:34.971736+0000","last_fullsized":"2024-05-03T21:19:34.971736+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T21:18:38.766727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T21:18:38.766727+0000","last_clean_scrub_stamp":"2024-05-03T21:18:38.766727+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T01:01:34.673401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":20,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-05-03T21:19:34.972106+0000","last_change":"2024-05-03T21:18:42.812123+0000","last_active":"2024-05-03T21:19:34.972106+0000","last_peered":"2024-05-03T21:19:34.972106+0000","last_clean":"2024-05-03T21:19:34.972106+0000","last_became_active":"2024-05-03T21:18:42.811799+0000","last_became_peered":"2024-05-03T21:18:42.811799+0000","last_unstale":"2024-05-03T21:19:34.972106+0000","last_undegraded":"2024-05-03T21:19:34.972106+0000","last_fullsized":"2024-05-03T21:19:34.972106+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_clean_scrub_stamp":"2024-05-03T21:18:41.788726+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T05:47:42.995000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"82'3","reported_seq":28,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-05-03T21:19:34.978710+0000","last_change":"2024-05-03T21:18:39.786729+0000","last_active":"2024-05-03T21:19:34.978710+0000","last_peered":"2024-05-03T21:19:34.978710+0000","last_clean":"2024-05-03T21:19:34.978710+0000","last_became_active":"2024-05-03T21:18:39.786276+0000","last_became_peered":"2024-05-03T21:18:39.786276+0000","last_unstale":"2024-05-03T21:19:34.978710+0000","last_undegraded":"2024-05-03T21:19:34.978710+0000","last_fullsized":"2024-05-03T21:19:34.978710+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T21:18:38.766727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T21:18:38.766727+0000","last_clean_scrub_stamp":"2024-05-03T21:18:38.766727+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T00:47:34.921468+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":20,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-05-03T21:19:34.975309+0000","last_change":"2024-05-03T21:18:42.818135+0000","last_active":"2024-05-03T21:19:34.975309+0000","last_peered":"2024-05-03T21:19:34.975309+0000","last_clean":"2024-05-03T21:19:34.975309+0000","last_became_active":"2024-05-03T21:18:42.816868+0000","last_became_peered":"2024-05-03T21:18:42.816868+0000","last_unstale":"2024-05-03T21:19:34.975309+0000","last_undegraded":"2024-05-03T21:19:34.975309+0000","last_fullsized":"2024-05-03T21:19:34.975309+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_clean_scrub_stamp":"2024-05-03T21:18:41.788726+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T00:36:26.783820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"82'1","reported_seq":15,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-05-03T21:19:34.984158+0000","last_change":"2024-05-03T21:19:34.983584+0000","last_active":"2024-05-03T21:19:34.984158+0000","last_peered":"2024-05-03T21:19:34.984158+0000","last_clean":"2024-05-03T21:19:34.984158+0000","last_became_active":"2024-05-03T21:19:34.979470+0000","last_became_peered":"2024-05-03T21:19:34.979470+0000","last_unstale":"2024-05-03T21:19:34.984158+0000","last_undegraded":"2024-05-03T21:19:34.984158+0000","last_fullsized":"2024-05-03T21:19:34.984158+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T21:18:38.766727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T21:18:38.766727+0000","last_clean_scrub_stamp":"2024-05-03T21:18:38.766727+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T06:10:19.528492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,3,8],"acting":[11,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":20,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-05-03T21:19:34.979444+0000","last_change":"2024-05-03T21:18:42.819671+0000","last_active":"2024-05-03T21:19:34.979444+0000","last_peered":"2024-05-03T21:19:34.979444+0000","last_clean":"2024-05-03T21:19:34.979444+0000","last_became_active":"2024-05-03T21:18:42.819336+0000","last_became_peered":"2024-05-03T21:18:42.819336+0000","last_unstale":"2024-05-03T21:19:34.979444+0000","last_undegraded":"2024-05-03T21:19:34.979444+0000","last_fullsized":"2024-05-03T21:19:34.979444+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_clean_scrub_stamp":"2024-05-03T21:18:41.788726+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T04:54:03.726522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"82'4","reported_seq":43,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-05-03T21:19:35.216509+0000","last_change":"2024-05-03T21:19:35.216509+0000","last_active":"2024-05-03T21:19:35.216509+0000","last_peered":"2024-05-03T21:19:35.216509+0000","last_clean":"2024-05-03T21:19:35.216509+0000","last_became_active":"2024-05-03T21:19:35.210095+0000","last_became_peered":"2024-05-03T21:19:35.210095+0000","last_unstale":"2024-05-03T21:19:35.216509+0000","last_undegraded":"2024-05-03T21:19:35.216509+0000","last_fullsized":"2024-05-03T21:19:35.216509+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T21:18:38.766727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T21:18:38.766727+0000","last_clean_scrub_stamp":"2024-05-03T21:18:38.766727+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T00:42:29.519443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,4,7],"acting":[8,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":20,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-05-03T21:19:34.974077+0000","last_change":"2024-05-03T21:18:42.810587+0000","last_active":"2024-05-03T21:19:34.974077+0000","last_peered":"2024-05-03T21:19:34.974077+0000","last_clean":"2024-05-03T21:19:34.974077+0000","last_became_active":"2024-05-03T21:18:42.810274+0000","last_became_peered":"2024-05-03T21:18:42.810274+0000","last_unstale":"2024-05-03T21:19:34.974077+0000","last_undegraded":"2024-05-03T21:19:34.974077+0000","last_fullsized":"2024-05-03T21:19:34.974077+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_clean_scrub_stamp":"2024-05-03T21:18:41.788726+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T02:45:20.891119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"82'3","reported_seq":29,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-05-03T21:19:34.973775+0000","last_change":"2024-05-03T21:18:39.905882+0000","last_active":"2024-05-03T21:19:34.973775+0000","last_peered":"2024-05-03T21:19:34.973775+0000","last_clean":"2024-05-03T21:19:34.973775+0000","last_became_active":"2024-05-03T21:18:39.905488+0000","last_became_peered":"2024-05-03T21:18:39.905488+0000","last_unstale":"2024-05-03T21:19:34.973775+0000","last_undegraded":"2024-05-03T21:19:34.973775+0000","last_fullsized":"2024-05-03T21:19:34.973775+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T21:18:38.766727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T21:18:38.766727+0000","last_clean_scrub_stamp":"2024-05-03T21:18:38.766727+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T02:02:17.913931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":20,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-05-03T21:19:34.974175+0000","last_change":"2024-05-03T21:18:42.818693+0000","last_active":"2024-05-03T21:19:34.974175+0000","last_peered":"2024-05-03T21:19:34.974175+0000","last_clean":"2024-05-03T21:19:34.974175+0000","last_became_active":"2024-05-03T21:18:42.818177+0000","last_became_peered":"2024-05-03T21:18:42.818177+0000","last_unstale":"2024-05-03T21:19:34.974175+0000","last_undegraded":"2024-05-03T21:19:34.974175+0000","last_fullsized":"2024-05-03T21:19:34.974175+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_clean_scrub_stamp":"2024-05-03T21:18:41.788726+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T03:05:10.708866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"82'1","reported_seq":28,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-05-03T21:19:34.971519+0000","last_change":"2024-05-03T21:18:39.792399+0000","last_active":"2024-05-03T21:19:34.971519+0000","last_peered":"2024-05-03T21:19:34.971519+0000","last_clean":"2024-05-03T21:19:34.971519+0000","last_became_active":"2024-05-03T21:18:39.791462+0000","last_became_peered":"2024-05-03T21:18:39.791462+0000","last_unstale":"2024-05-03T21:19:34.971519+0000","last_undegraded":"2024-05-03T21:19:34.971519+0000","last_fullsized":"2024-05-03T21:19:34.971519+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T21:18:38.766727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T21:18:38.766727+0000","last_clean_scrub_stamp":"2024-05-03T21:18:38.766727+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T08:24:23.045438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"82'1","reported_seq":21,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-05-03T21:19:34.973967+0000","last_change":"2024-05-03T21:18:42.813275+0000","last_active":"2024-05-03T21:19:34.973967+0000","last_peered":"2024-05-03T21:19:34.973967+0000","last_clean":"2024-05-03T21:19:34.973967+0000","last_became_active":"2024-05-03T21:18:42.812640+0000","last_became_peered":"2024-05-03T21:18:42.812640+0000","last_unstale":"2024-05-03T21:19:34.973967+0000","last_undegraded":"2024-05-03T21:19:34.973967+0000","last_fullsized":"2024-05-03T21:19:34.973967+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_clean_scrub_stamp":"2024-05-03T21:18:41.788726+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T01:34:27.071382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"82'2","reported_seq":27,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-05-03T21:19:34.973896+0000","last_change":"2024-05-03T21:18:39.787487+0000","last_active":"2024-05-03T21:19:34.973896+0000","last_peered":"2024-05-03T21:19:34.973896+0000","last_clean":"2024-05-03T21:19:34.973896+0000","last_became_active":"2024-05-03T21:18:39.787135+0000","last_became_peered":"2024-05-03T21:18:39.787135+0000","last_unstale":"2024-05-03T21:19:34.973896+0000","last_undegraded":"2024-05-03T21:19:34.973896+0000","last_fullsized":"2024-05-03T21:19:34.973896+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T21:18:38.766727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T21:18:38.766727+0000","last_clean_scrub_stamp":"2024-05-03T21:18:38.766727+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T05:21:13.304600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":20,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-05-03T21:19:34.974534+0000","last_change":"2024-05-03T21:18:42.815682+0000","last_active":"2024-05-03T21:19:34.974534+0000","last_peered":"2024-05-03T21:19:34.974534+0000","last_clean":"2024-05-03T21:19:34.974534+0000","last_became_active":"2024-05-03T21:18:42.813011+0000","last_became_peered":"2024-05-03T21:18:42.813011+0000","last_unstale":"2024-05-03T21:19:34.974534+0000","last_undegraded":"2024-05-03T21:19:34.974534+0000","last_fullsized":"2024-05-03T21:19:34.974534+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_clean_scrub_stamp":"2024-05-03T21:18:41.788726+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T03:47:21.680152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":20,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-05-03T21:19:34.975419+0000","last_change":"2024-05-03T21:18:42.817574+0000","last_active":"2024-05-03T21:19:34.975419+0000","last_peered":"2024-05-03T21:19:34.975419+0000","last_clean":"2024-05-03T21:19:34.975419+0000","last_became_active":"2024-05-03T21:18:42.817227+0000","last_became_peered":"2024-05-03T21:18:42.817227+0000","last_unstale":"2024-05-03T21:19:34.975419+0000","last_undegraded":"2024-05-03T21:19:34.975419+0000","last_fullsized":"2024-05-03T21:19:34.975419+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_clean_scrub_stamp":"2024-05-03T21:18:41.788726+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-04T23:15:34.784285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":26,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-05-03T21:19:34.982916+0000","last_change":"2024-05-03T21:19:34.982916+0000","last_active":"2024-05-03T21:19:34.982916+0000","last_peered":"2024-05-03T21:19:34.982916+0000","last_clean":"2024-05-03T21:19:34.982916+0000","last_became_active":"2024-05-03T21:19:34.982343+0000","last_became_peered":"2024-05-03T21:19:34.982343+0000","last_unstale":"2024-05-03T21:19:34.982916+0000","last_undegraded":"2024-05-03T21:19:34.982916+0000","last_fullsized":"2024-05-03T21:19:34.982916+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_clean_scrub_stamp":"2024-05-03T21:18:41.788726+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T07:39:56.301149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,8],"acting":[2,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":20,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-05-03T21:19:34.973231+0000","last_change":"2024-05-03T21:18:42.823980+0000","last_active":"2024-05-03T21:19:34.973231+0000","last_peered":"2024-05-03T21:19:34.973231+0000","last_clean":"2024-05-03T21:19:34.973231+0000","last_became_active":"2024-05-03T21:18:42.823619+0000","last_became_peered":"2024-05-03T21:18:42.823619+0000","last_unstale":"2024-05-03T21:19:34.973231+0000","last_undegraded":"2024-05-03T21:19:34.973231+0000","last_fullsized":"2024-05-03T21:19:34.973231+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_clean_scrub_stamp":"2024-05-03T21:18:41.788726+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T08:13:53.104446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":20,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-05-03T21:19:34.979401+0000","last_change":"2024-05-03T21:18:42.813596+0000","last_active":"2024-05-03T21:19:34.979401+0000","last_peered":"2024-05-03T21:19:34.979401+0000","last_clean":"2024-05-03T21:19:34.979401+0000","last_became_active":"2024-05-03T21:18:42.813238+0000","last_became_peered":"2024-05-03T21:18:42.813238+0000","last_unstale":"2024-05-03T21:19:34.979401+0000","last_undegraded":"2024-05-03T21:19:34.979401+0000","last_fullsized":"2024-05-03T21:19:34.979401+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_clean_scrub_stamp":"2024-05-03T21:18:41.788726+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T05:14:52.712566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":20,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-05-03T21:19:34.973476+0000","last_change":"2024-05-03T21:18:42.812252+0000","last_active":"2024-05-03T21:19:34.973476+0000","last_peered":"2024-05-03T21:19:34.973476+0000","last_clean":"2024-05-03T21:19:34.973476+0000","last_became_active":"2024-05-03T21:18:42.811917+0000","last_became_peered":"2024-05-03T21:18:42.811917+0000","last_unstale":"2024-05-03T21:19:34.973476+0000","last_undegraded":"2024-05-03T21:19:34.973476+0000","last_fullsized":"2024-05-03T21:19:34.973476+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_clean_scrub_stamp":"2024-05-03T21:18:41.788726+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T00:55:09.512412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":20,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-05-03T21:19:34.974830+0000","last_change":"2024-05-03T21:18:42.819345+0000","last_active":"2024-05-03T21:19:34.974830+0000","last_peered":"2024-05-03T21:19:34.974830+0000","last_clean":"2024-05-03T21:19:34.974830+0000","last_became_active":"2024-05-03T21:18:42.817901+0000","last_became_peered":"2024-05-03T21:18:42.817901+0000","last_unstale":"2024-05-03T21:19:34.974830+0000","last_undegraded":"2024-05-03T21:19:34.974830+0000","last_fullsized":"2024-05-03T21:19:34.974830+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_clean_scrub_stamp":"2024-05-03T21:18:41.788726+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T07:25:30.498261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":20,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-05-03T21:19:34.973679+0000","last_change":"2024-05-03T21:18:42.825009+0000","last_active":"2024-05-03T21:19:34.973679+0000","last_peered":"2024-05-03T21:19:34.973679+0000","last_clean":"2024-05-03T21:19:34.973679+0000","last_became_active":"2024-05-03T21:18:42.824626+0000","last_became_peered":"2024-05-03T21:18:42.824626+0000","last_unstale":"2024-05-03T21:19:34.973679+0000","last_undegraded":"2024-05-03T21:19:34.973679+0000","last_fullsized":"2024-05-03T21:19:34.973679+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_clean_scrub_stamp":"2024-05-03T21:18:41.788726+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T06:48:17.760340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":20,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-05-03T21:19:34.978336+0000","last_change":"2024-05-03T21:18:42.812305+0000","last_active":"2024-05-03T21:19:34.978336+0000","last_peered":"2024-05-03T21:19:34.978336+0000","last_clean":"2024-05-03T21:19:34.978336+0000","last_became_active":"2024-05-03T21:18:42.811738+0000","last_became_peered":"2024-05-03T21:18:42.811738+0000","last_unstale":"2024-05-03T21:19:34.978336+0000","last_undegraded":"2024-05-03T21:19:34.978336+0000","last_fullsized":"2024-05-03T21:19:34.978336+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_clean_scrub_stamp":"2024-05-03T21:18:41.788726+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-04T21:41:24.312517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":20,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-05-03T21:19:34.971389+0000","last_change":"2024-05-03T21:18:42.817306+0000","last_active":"2024-05-03T21:19:34.971389+0000","last_peered":"2024-05-03T21:19:34.971389+0000","last_clean":"2024-05-03T21:19:34.971389+0000","last_became_active":"2024-05-03T21:18:42.816768+0000","last_became_peered":"2024-05-03T21:18:42.816768+0000","last_unstale":"2024-05-03T21:19:34.971389+0000","last_undegraded":"2024-05-03T21:19:34.971389+0000","last_fullsized":"2024-05-03T21:19:34.971389+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_clean_scrub_stamp":"2024-05-03T21:18:41.788726+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-04T23:24:42.055568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":20,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-05-03T21:19:34.978607+0000","last_change":"2024-05-03T21:18:42.816008+0000","last_active":"2024-05-03T21:19:34.978607+0000","last_peered":"2024-05-03T21:19:34.978607+0000","last_clean":"2024-05-03T21:19:34.978607+0000","last_became_active":"2024-05-03T21:18:42.815653+0000","last_became_peered":"2024-05-03T21:18:42.815653+0000","last_unstale":"2024-05-03T21:19:34.978607+0000","last_undegraded":"2024-05-03T21:19:34.978607+0000","last_fullsized":"2024-05-03T21:19:34.978607+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_clean_scrub_stamp":"2024-05-03T21:18:41.788726+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-04T23:15:22.986562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":20,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-05-03T21:19:34.971889+0000","last_change":"2024-05-03T21:18:42.814820+0000","last_active":"2024-05-03T21:19:34.971889+0000","last_peered":"2024-05-03T21:19:34.971889+0000","last_clean":"2024-05-03T21:19:34.971889+0000","last_became_active":"2024-05-03T21:18:42.814498+0000","last_became_peered":"2024-05-03T21:18:42.814498+0000","last_unstale":"2024-05-03T21:19:34.971889+0000","last_undegraded":"2024-05-03T21:19:34.971889+0000","last_fullsized":"2024-05-03T21:19:34.971889+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_clean_scrub_stamp":"2024-05-03T21:18:41.788726+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T01:35:15.727585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":20,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-05-03T21:19:34.973615+0000","last_change":"2024-05-03T21:18:42.814701+0000","last_active":"2024-05-03T21:19:34.973615+0000","last_peered":"2024-05-03T21:19:34.973615+0000","last_clean":"2024-05-03T21:19:34.973615+0000","last_became_active":"2024-05-03T21:18:42.814394+0000","last_became_peered":"2024-05-03T21:18:42.814394+0000","last_unstale":"2024-05-03T21:19:34.973615+0000","last_undegraded":"2024-05-03T21:19:34.973615+0000","last_fullsized":"2024-05-03T21:19:34.973615+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_clean_scrub_stamp":"2024-05-03T21:18:41.788726+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T04:50:05.941873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":20,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-05-03T21:19:34.974476+0000","last_change":"2024-05-03T21:18:42.815987+0000","last_active":"2024-05-03T21:19:34.974476+0000","last_peered":"2024-05-03T21:19:34.974476+0000","last_clean":"2024-05-03T21:19:34.974476+0000","last_became_active":"2024-05-03T21:18:42.815613+0000","last_became_peered":"2024-05-03T21:18:42.815613+0000","last_unstale":"2024-05-03T21:19:34.974476+0000","last_undegraded":"2024-05-03T21:19:34.974476+0000","last_fullsized":"2024-05-03T21:19:34.974476+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_clean_scrub_stamp":"2024-05-03T21:18:41.788726+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T03:57:01.997504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4751360,"data_stored":4722944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":92,"ondisk_log_size":92,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":56154,"num_objects":60,"num_object_clones":0,"num_object_copies":180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":0,"num_read":587,"num_read_kb":354,"num_write":158,"num_write_kb":100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":112,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":393216,"data_stored":168462,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":136,"ondisk_log_size":136,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":74,"seq":317827579942,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27912,"kb_used_data":892,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717240,"statfs":{"total":95995035648,"available":95966453760,"internally_reserved":0,"allocated":913408,"data_stored":442207,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":68,"seq":292057776171,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28580,"kb_used_data":1556,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716572,"statfs":{"total":95995035648,"available":95965769728,"internally_reserved":0,"allocated":1593344,"data_stored":1085729,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":63,"seq":270582939696,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32084,"kb_used_data":900,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713068,"statfs":{"total":95995035648,"available":95962181632,"internally_reserved":0,"allocated":921600,"data_stored":442331,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":57,"seq":244813135926,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32076,"kb_used_data":892,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713076,"statfs":{"total":95995035648,"available":95962189824,"internally_reserved":0,"allocated":913408,"data_stored":442139,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":50,"seq":214748364858,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28500,"kb_used_data":1480,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716652,"statfs":{"total":95995035648,"available":95965851648,"internally_reserved":0,"allocated":1515520,"data_stored":1034572,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":45,"seq":193273528383,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27912,"kb_used_data":888,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717240,"statfs":{"total":95995035648,"available":95966453760,"internally_reserved":0,"allocated":909312,"data_stored":441720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574937,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36284,"kb_used_data":940,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708868,"statfs":{"total":95995035648,"available":95957880832,"internally_reserved":0,"allocated":962560,"data_stored":493275,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738461,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36252,"kb_used_data":908,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708900,"statfs":{"total":95995035648,"available":95957913600,"internally_reserved":0,"allocated":929792,"data_stored":442991,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411411,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36308,"kb_used_data":964,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708844,"statfs":{"total":95995035648,"available":95957856256,"internally_reserved":0,"allocated":987136,"data_stored":495016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":27,"seq":115964117070,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36244,"kb_used_data":904,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708908,"statfs":{"total":95995035648,"available":95957921792,"internally_reserved":0,"allocated":925696,"data_stored":442031,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920840,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27912,"kb_used_data":892,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717240,"statfs":{"total":95995035648,"available":95966453760,"internally_reserved":0,"allocated":913408,"data_stored":442548,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724613,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32672,"kb_used_data":1492,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712480,"statfs":{"total":95995035648,"available":95961579520,"internally_reserved":0,"allocated":1527808,"data_stored":1033865,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":51762,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":53503,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1035,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1867,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2574,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":626,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":94208,"data_stored":53731,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-05-03T21:20:37.479 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9a047fe7599a549042d06a3fb5b6813a203eabea shell --fsid 713bc57a-0991-11ef-bc97-c7b262605968 -- ceph pg dump --format=json 2024-05-03T21:20:37.495 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:20:37 smithi082 ceph-mon[32933]: pgmap v345: 97 pgs: 97 active+clean; 632 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:20:37.574 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:20:37 smithi138 ceph-mon[33110]: pgmap v345: 97 pgs: 97 active+clean; 632 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:20:37.732 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/713bc57a-0991-11ef-bc97-c7b262605968/mon.a/config 2024-05-03T21:20:38.471 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:20:38 smithi067 ceph-mon[29730]: from='client.14916 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-03T21:20:38.495 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:20:38 smithi082 ceph-mon[32933]: from='client.14916 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-03T21:20:38.574 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:20:38 smithi138 ceph-mon[33110]: from='client.14916 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-03T21:20:39.340 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:39.337+0000 7f1c8b9a9700 1 -- 172.21.15.67:0/3061719104 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1c84108850 msgr2=0x7f1c841103d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:20:39.340 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:39.337+0000 7f1c8b9a9700 1 --2- 172.21.15.67:0/3061719104 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1c84108850 0x7f1c841103d0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f1c7400b0d0 tx=0x7f1c7400b3e0 comp rx=0 tx=0).stop 2024-05-03T21:20:39.340 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:39.337+0000 7f1c8b9a9700 1 -- 172.21.15.67:0/3061719104 shutdown_connections 2024-05-03T21:20:39.340 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:39.337+0000 7f1c8b9a9700 1 --2- 172.21.15.67:0/3061719104 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1c84108850 0x7f1c841103d0 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:20:39.340 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:39.337+0000 7f1c8b9a9700 1 --2- 172.21.15.67:0/3061719104 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f1c84103750 0x7f1c84103b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:20:39.340 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:39.337+0000 7f1c8b9a9700 1 --2- 172.21.15.67:0/3061719104 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f1c84104100 0x7f1c84108120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:20:39.340 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:39.337+0000 7f1c8b9a9700 1 -- 172.21.15.67:0/3061719104 >> 172.21.15.67:0/3061719104 conn(0x7f1c8407a600 msgr2=0x7f1c8407aa10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T21:20:39.340 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:39.337+0000 7f1c8b9a9700 1 -- 172.21.15.67:0/3061719104 shutdown_connections 2024-05-03T21:20:39.340 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:39.337+0000 7f1c8b9a9700 1 -- 172.21.15.67:0/3061719104 wait complete. 2024-05-03T21:20:39.342 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:39.338+0000 7f1c8b9a9700 1 Processor -- start 2024-05-03T21:20:39.342 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:39.338+0000 7f1c8b9a9700 1 -- start start 2024-05-03T21:20:39.342 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:39.338+0000 7f1c8b9a9700 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f1c84103750 0x7f1c8419beb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:20:39.342 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:39.338+0000 7f1c8b9a9700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f1c84104100 0x7f1c8419c3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:20:39.342 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:39.338+0000 7f1c8b9a9700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1c84108850 0x7f1c841a0780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:20:39.342 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:39.338+0000 7f1c8b9a9700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f1c84113920 con 0x7f1c84108850 2024-05-03T21:20:39.343 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:39.338+0000 7f1c8b9a9700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f1c841137a0 con 0x7f1c84104100 2024-05-03T21:20:39.343 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:39.338+0000 7f1c8b9a9700 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f1c84113aa0 con 0x7f1c84103750 2024-05-03T21:20:39.343 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:39.338+0000 7f1c89f46700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1c84108850 0x7f1c841a0780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:20:39.343 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:39.338+0000 7f1c89f46700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1c84108850 0x7f1c841a0780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:51398/0 (socket says 172.21.15.67:51398) 2024-05-03T21:20:39.343 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:39.338+0000 7f1c89f46700 1 -- 172.21.15.67:0/1575810993 learned_addr learned my addr 172.21.15.67:0/1575810993 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-03T21:20:39.343 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:39.338+0000 7f1c89745700 1 --2- 172.21.15.67:0/1575810993 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f1c84103750 0x7f1c8419beb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:20:39.343 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:39.338+0000 7f1c88f44700 1 --2- 172.21.15.67:0/1575810993 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f1c84104100 0x7f1c8419c3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:20:39.343 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:39.339+0000 7f1c89f46700 1 -- 172.21.15.67:0/1575810993 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f1c84103750 msgr2=0x7f1c8419beb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:20:39.343 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:39.339+0000 7f1c89f46700 1 --2- 172.21.15.67:0/1575810993 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f1c84103750 0x7f1c8419beb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:20:39.343 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:39.339+0000 7f1c89f46700 1 -- 172.21.15.67:0/1575810993 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f1c84104100 msgr2=0x7f1c8419c3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:20:39.343 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:39.339+0000 7f1c89f46700 1 --2- 172.21.15.67:0/1575810993 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f1c84104100 0x7f1c8419c3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:20:39.343 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:39.339+0000 7f1c89f46700 1 -- 172.21.15.67:0/1575810993 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1c841a0ec0 con 0x7f1c84108850 2024-05-03T21:20:39.343 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:39.339+0000 7f1c89745700 1 --2- 172.21.15.67:0/1575810993 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f1c84103750 0x7f1c8419beb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-03T21:20:39.343 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:39.339+0000 7f1c88f44700 1 --2- 172.21.15.67:0/1575810993 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f1c84104100 0x7f1c8419c3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-03T21:20:39.344 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:39.339+0000 7f1c89f46700 1 --2- 172.21.15.67:0/1575810993 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1c84108850 0x7f1c841a0780 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f1c7400d010 tx=0x7f1c74007ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T21:20:39.344 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:39.339+0000 7f1c827fc700 1 -- 172.21.15.67:0/1575810993 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c7401f070 con 0x7f1c84108850 2024-05-03T21:20:39.344 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:39.339+0000 7f1c8b9a9700 1 -- 172.21.15.67:0/1575810993 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1c841a1150 con 0x7f1c84108850 2024-05-03T21:20:39.344 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:39.339+0000 7f1c827fc700 1 -- 172.21.15.67:0/1575810993 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(28 keys) ==== 1170+0+0 (secure 0 0 0) 0x7f1c74003c00 con 0x7f1c84108850 2024-05-03T21:20:39.344 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:39.339+0000 7f1c827fc700 1 -- 172.21.15.67:0/1575810993 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c7401b840 con 0x7f1c84108850 2024-05-03T21:20:39.344 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:39.339+0000 7f1c8b9a9700 1 -- 172.21.15.67:0/1575810993 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1c841a8a30 con 0x7f1c84108850 2024-05-03T21:20:39.344 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:39.340+0000 7f1c8b9a9700 1 -- 172.21.15.67:0/1575810993 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1c8406a190 con 0x7f1c84108850 2024-05-03T21:20:39.345 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:39.341+0000 7f1c827fc700 1 -- 172.21.15.67:0/1575810993 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 17) ==== 149703+0+0 (secure 0 0 0) 0x7f1c74022070 con 0x7f1c84108850 2024-05-03T21:20:39.345 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:39.342+0000 7f1c827fc700 1 --2- 172.21.15.67:0/1575810993 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f1c6c0b2910 0x7f1c6c0b4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:20:39.345 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:39.342+0000 7f1c89745700 1 --2- 172.21.15.67:0/1575810993 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f1c6c0b2910 0x7f1c6c0b4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:20:39.345 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:39.342+0000 7f1c827fc700 1 -- 172.21.15.67:0/1575810993 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 8211+0+0 (secure 0 0 0) 0x7f1c7400f020 con 0x7f1c84108850 2024-05-03T21:20:39.345 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:39.343+0000 7f1c89745700 1 --2- 172.21.15.67:0/1575810993 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f1c6c0b2910 0x7f1c6c0b4dd0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f1c7c005950 tx=0x7f1c7c0058e0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T21:20:39.348 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:39.346+0000 7f1c827fc700 1 -- 172.21.15.67:0/1575810993 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191839 (secure 0 0 0) 0x7f1c74029020 con 0x7f1c84108850 2024-05-03T21:20:39.471 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:20:39 smithi067 ceph-mon[29730]: pgmap v346: 97 pgs: 97 active+clean; 632 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 85 B/s wr, 6 op/s 2024-05-03T21:20:39.495 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:20:39 smithi082 ceph-mon[32933]: pgmap v346: 97 pgs: 97 active+clean; 632 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 85 B/s wr, 6 op/s 2024-05-03T21:20:39.556 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:39.554+0000 7f1c8b9a9700 1 -- 172.21.15.67:0/1575810993 --> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f1c8410f730 con 0x7f1c6c0b2910 2024-05-03T21:20:39.563 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:39.561+0000 7f1c827fc700 1 -- 172.21.15.67:0/1575810993 <== mgr.14150 v2:172.21.15.67:6800/604451592 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+258716 (secure 0 0 0) 0x7f1c8410f730 con 0x7f1c6c0b2910 2024-05-03T21:20:39.567 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-05-03T21:20:39.569 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:39.566+0000 7f1c8b9a9700 1 -- 172.21.15.67:0/1575810993 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f1c6c0b2910 msgr2=0x7f1c6c0b4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:20:39.569 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:39.566+0000 7f1c8b9a9700 1 --2- 172.21.15.67:0/1575810993 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f1c6c0b2910 0x7f1c6c0b4dd0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f1c7c005950 tx=0x7f1c7c0058e0 comp rx=0 tx=0).stop 2024-05-03T21:20:39.569 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:39.566+0000 7f1c8b9a9700 1 -- 172.21.15.67:0/1575810993 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1c84108850 msgr2=0x7f1c841a0780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:20:39.569 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:39.566+0000 7f1c8b9a9700 1 --2- 172.21.15.67:0/1575810993 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1c84108850 0x7f1c841a0780 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f1c7400d010 tx=0x7f1c74007ad0 comp rx=0 tx=0).stop 2024-05-03T21:20:39.569 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:39.566+0000 7f1c8b9a9700 1 -- 172.21.15.67:0/1575810993 shutdown_connections 2024-05-03T21:20:39.569 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:39.566+0000 7f1c8b9a9700 1 --2- 172.21.15.67:0/1575810993 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f1c6c0b2910 0x7f1c6c0b4dd0 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:20:39.569 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:39.566+0000 7f1c8b9a9700 1 --2- 172.21.15.67:0/1575810993 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1c84108850 0x7f1c841a0780 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:20:39.570 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:39.566+0000 7f1c8b9a9700 1 --2- 172.21.15.67:0/1575810993 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f1c84103750 0x7f1c8419beb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:20:39.570 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:39.566+0000 7f1c8b9a9700 1 --2- 172.21.15.67:0/1575810993 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f1c84104100 0x7f1c8419c3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:20:39.570 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:39.566+0000 7f1c8b9a9700 1 -- 172.21.15.67:0/1575810993 >> 172.21.15.67:0/1575810993 conn(0x7f1c8407a600 msgr2=0x7f1c841016a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T21:20:39.570 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:39.566+0000 7f1c8b9a9700 1 -- 172.21.15.67:0/1575810993 shutdown_connections 2024-05-03T21:20:39.570 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:39.566+0000 7f1c8b9a9700 1 -- 172.21.15.67:0/1575810993 wait complete. 2024-05-03T21:20:39.570 INFO:teuthology.orchestra.run.smithi067.stderr:dumped all 2024-05-03T21:20:39.574 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:20:39 smithi138 ceph-mon[33110]: pgmap v346: 97 pgs: 97 active+clean; 632 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 85 B/s wr, 6 op/s 2024-05-03T21:20:40.230 INFO:teuthology.orchestra.run.smithi067.stdout:{"pg_ready":true,"pg_map":{"version":346,"stamp":"2024-05-03T21:20:37.834291+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":646756,"num_objects":64,"num_object_clones":0,"num_object_copies":192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":64,"num_whiteouts":0,"num_read":678,"num_read_kb":423,"num_write":286,"num_write_kb":2253,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":2951952,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":231,"ondisk_log_size":231,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":291,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":382736,"kb_used_data":12708,"kb_used_omap":18,"kb_used_meta":369837,"kb_avail":1124559088,"statfs":{"total":1151940427776,"available":1151548506112,"internally_reserved":0,"allocated":13012992,"data_stored":7238424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19143,"internal_metadata":378713401},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":75,"num_read_kb":35,"num_write":3,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003146"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":20,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-05-03T21:19:34.975095+0000","last_change":"2024-05-03T21:18:42.819790+0000","last_active":"2024-05-03T21:19:34.975095+0000","last_peered":"2024-05-03T21:19:34.975095+0000","last_clean":"2024-05-03T21:19:34.975095+0000","last_became_active":"2024-05-03T21:18:42.818989+0000","last_became_peered":"2024-05-03T21:18:42.818989+0000","last_unstale":"2024-05-03T21:19:34.975095+0000","last_undegraded":"2024-05-03T21:19:34.975095+0000","last_fullsized":"2024-05-03T21:19:34.975095+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_clean_scrub_stamp":"2024-05-03T21:18:41.788726+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T08:38:43.699198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":20,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-05-03T21:19:34.979895+0000","last_change":"2024-05-03T21:18:42.812454+0000","last_active":"2024-05-03T21:19:34.979895+0000","last_peered":"2024-05-03T21:19:34.979895+0000","last_clean":"2024-05-03T21:19:34.979895+0000","last_became_active":"2024-05-03T21:18:42.812140+0000","last_became_peered":"2024-05-03T21:18:42.812140+0000","last_unstale":"2024-05-03T21:19:34.979895+0000","last_undegraded":"2024-05-03T21:19:34.979895+0000","last_fullsized":"2024-05-03T21:19:34.979895+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_clean_scrub_stamp":"2024-05-03T21:18:41.788726+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-04T22:30:36.057079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":20,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-05-03T21:19:34.979632+0000","last_change":"2024-05-03T21:18:42.815547+0000","last_active":"2024-05-03T21:19:34.979632+0000","last_peered":"2024-05-03T21:19:34.979632+0000","last_clean":"2024-05-03T21:19:34.979632+0000","last_became_active":"2024-05-03T21:18:42.815192+0000","last_became_peered":"2024-05-03T21:18:42.815192+0000","last_unstale":"2024-05-03T21:19:34.979632+0000","last_undegraded":"2024-05-03T21:19:34.979632+0000","last_fullsized":"2024-05-03T21:19:34.979632+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_clean_scrub_stamp":"2024-05-03T21:18:41.788726+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T04:15:05.786275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":11,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-05-03T21:19:34.983212+0000","last_change":"2024-05-03T21:19:34.982658+0000","last_active":"2024-05-03T21:19:34.983212+0000","last_peered":"2024-05-03T21:19:34.983212+0000","last_clean":"2024-05-03T21:19:34.983212+0000","last_became_active":"2024-05-03T21:19:34.982225+0000","last_became_peered":"2024-05-03T21:19:34.982225+0000","last_unstale":"2024-05-03T21:19:34.983212+0000","last_undegraded":"2024-05-03T21:19:34.983212+0000","last_fullsized":"2024-05-03T21:19:34.983212+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_clean_scrub_stamp":"2024-05-03T21:18:41.788726+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T00:11:37.702317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,2],"acting":[9,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":20,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-05-03T21:19:34.972150+0000","last_change":"2024-05-03T21:18:42.807225+0000","last_active":"2024-05-03T21:19:34.972150+0000","last_peered":"2024-05-03T21:19:34.972150+0000","last_clean":"2024-05-03T21:19:34.972150+0000","last_became_active":"2024-05-03T21:18:42.806856+0000","last_became_peered":"2024-05-03T21:18:42.806856+0000","last_unstale":"2024-05-03T21:19:34.972150+0000","last_undegraded":"2024-05-03T21:19:34.972150+0000","last_fullsized":"2024-05-03T21:19:34.972150+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_clean_scrub_stamp":"2024-05-03T21:18:41.788726+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-04T23:59:15.221592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":20,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-05-03T21:19:35.208158+0000","last_change":"2024-05-03T21:18:42.819099+0000","last_active":"2024-05-03T21:19:35.208158+0000","last_peered":"2024-05-03T21:19:35.208158+0000","last_clean":"2024-05-03T21:19:35.208158+0000","last_became_active":"2024-05-03T21:18:42.818775+0000","last_became_peered":"2024-05-03T21:18:42.818775+0000","last_unstale":"2024-05-03T21:19:35.208158+0000","last_undegraded":"2024-05-03T21:19:35.208158+0000","last_fullsized":"2024-05-03T21:19:35.208158+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_clean_scrub_stamp":"2024-05-03T21:18:41.788726+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T03:46:43.762816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":20,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-05-03T21:19:34.973759+0000","last_change":"2024-05-03T21:18:42.811779+0000","last_active":"2024-05-03T21:19:34.973759+0000","last_peered":"2024-05-03T21:19:34.973759+0000","last_clean":"2024-05-03T21:19:34.973759+0000","last_became_active":"2024-05-03T21:18:42.811376+0000","last_became_peered":"2024-05-03T21:18:42.811376+0000","last_unstale":"2024-05-03T21:19:34.973759+0000","last_undegraded":"2024-05-03T21:19:34.973759+0000","last_fullsized":"2024-05-03T21:19:34.973759+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_clean_scrub_stamp":"2024-05-03T21:18:41.788726+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-04T22:08:00.629106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":20,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-05-03T21:19:34.978376+0000","last_change":"2024-05-03T21:18:42.820643+0000","last_active":"2024-05-03T21:19:34.978376+0000","last_peered":"2024-05-03T21:19:34.978376+0000","last_clean":"2024-05-03T21:19:34.978376+0000","last_became_active":"2024-05-03T21:18:42.820317+0000","last_became_peered":"2024-05-03T21:18:42.820317+0000","last_unstale":"2024-05-03T21:19:34.978376+0000","last_undegraded":"2024-05-03T21:19:34.978376+0000","last_fullsized":"2024-05-03T21:19:34.978376+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_clean_scrub_stamp":"2024-05-03T21:18:41.788726+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-04T21:39:44.319396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":20,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-05-03T21:19:34.972315+0000","last_change":"2024-05-03T21:18:42.819895+0000","last_active":"2024-05-03T21:19:34.972315+0000","last_peered":"2024-05-03T21:19:34.972315+0000","last_clean":"2024-05-03T21:19:34.972315+0000","last_became_active":"2024-05-03T21:18:42.818360+0000","last_became_peered":"2024-05-03T21:18:42.818360+0000","last_unstale":"2024-05-03T21:19:34.972315+0000","last_undegraded":"2024-05-03T21:19:34.972315+0000","last_fullsized":"2024-05-03T21:19:34.972315+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_clean_scrub_stamp":"2024-05-03T21:18:41.788726+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T06:49:09.368610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":20,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-05-03T21:19:35.208821+0000","last_change":"2024-05-03T21:18:42.817064+0000","last_active":"2024-05-03T21:19:35.208821+0000","last_peered":"2024-05-03T21:19:35.208821+0000","last_clean":"2024-05-03T21:19:35.208821+0000","last_became_active":"2024-05-03T21:18:42.816727+0000","last_became_peered":"2024-05-03T21:18:42.816727+0000","last_unstale":"2024-05-03T21:19:35.208821+0000","last_undegraded":"2024-05-03T21:19:35.208821+0000","last_fullsized":"2024-05-03T21:19:35.208821+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_clean_scrub_stamp":"2024-05-03T21:18:41.788726+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T06:30:20.093765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":20,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-05-03T21:19:34.974044+0000","last_change":"2024-05-03T21:18:42.818728+0000","last_active":"2024-05-03T21:19:34.974044+0000","last_peered":"2024-05-03T21:19:34.974044+0000","last_clean":"2024-05-03T21:19:34.974044+0000","last_became_active":"2024-05-03T21:18:42.818250+0000","last_became_peered":"2024-05-03T21:18:42.818250+0000","last_unstale":"2024-05-03T21:19:34.974044+0000","last_undegraded":"2024-05-03T21:19:34.974044+0000","last_fullsized":"2024-05-03T21:19:34.974044+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_clean_scrub_stamp":"2024-05-03T21:18:41.788726+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T03:52:04.613842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":20,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-05-03T21:19:35.208942+0000","last_change":"2024-05-03T21:18:42.816897+0000","last_active":"2024-05-03T21:19:35.208942+0000","last_peered":"2024-05-03T21:19:35.208942+0000","last_clean":"2024-05-03T21:19:35.208942+0000","last_became_active":"2024-05-03T21:18:42.816274+0000","last_became_peered":"2024-05-03T21:18:42.816274+0000","last_unstale":"2024-05-03T21:19:35.208942+0000","last_undegraded":"2024-05-03T21:19:35.208942+0000","last_fullsized":"2024-05-03T21:19:35.208942+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_clean_scrub_stamp":"2024-05-03T21:18:41.788726+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-04T21:48:17.744413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":20,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-05-03T21:19:34.973435+0000","last_change":"2024-05-03T21:18:42.820523+0000","last_active":"2024-05-03T21:19:34.973435+0000","last_peered":"2024-05-03T21:19:34.973435+0000","last_clean":"2024-05-03T21:19:34.973435+0000","last_became_active":"2024-05-03T21:18:42.820133+0000","last_became_peered":"2024-05-03T21:18:42.820133+0000","last_unstale":"2024-05-03T21:19:34.973435+0000","last_undegraded":"2024-05-03T21:19:34.973435+0000","last_fullsized":"2024-05-03T21:19:34.973435+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_clean_scrub_stamp":"2024-05-03T21:18:41.788726+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-04T23:25:34.304369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":20,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-05-03T21:19:34.978724+0000","last_change":"2024-05-03T21:18:42.814467+0000","last_active":"2024-05-03T21:19:34.978724+0000","last_peered":"2024-05-03T21:19:34.978724+0000","last_clean":"2024-05-03T21:19:34.978724+0000","last_became_active":"2024-05-03T21:18:42.814124+0000","last_became_peered":"2024-05-03T21:18:42.814124+0000","last_unstale":"2024-05-03T21:19:34.978724+0000","last_undegraded":"2024-05-03T21:19:34.978724+0000","last_fullsized":"2024-05-03T21:19:34.978724+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_clean_scrub_stamp":"2024-05-03T21:18:41.788726+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T08:41:03.567620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":20,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-05-03T21:19:34.978587+0000","last_change":"2024-05-03T21:18:42.822158+0000","last_active":"2024-05-03T21:19:34.978587+0000","last_peered":"2024-05-03T21:19:34.978587+0000","last_clean":"2024-05-03T21:19:34.978587+0000","last_became_active":"2024-05-03T21:18:42.821793+0000","last_became_peered":"2024-05-03T21:18:42.821793+0000","last_unstale":"2024-05-03T21:19:34.978587+0000","last_undegraded":"2024-05-03T21:19:34.978587+0000","last_fullsized":"2024-05-03T21:19:34.978587+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_clean_scrub_stamp":"2024-05-03T21:18:41.788726+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T06:44:09.269513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":20,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-05-03T21:19:34.971720+0000","last_change":"2024-05-03T21:18:42.813904+0000","last_active":"2024-05-03T21:19:34.971720+0000","last_peered":"2024-05-03T21:19:34.971720+0000","last_clean":"2024-05-03T21:19:34.971720+0000","last_became_active":"2024-05-03T21:18:42.813488+0000","last_became_peered":"2024-05-03T21:18:42.813488+0000","last_unstale":"2024-05-03T21:19:34.971720+0000","last_undegraded":"2024-05-03T21:19:34.971720+0000","last_fullsized":"2024-05-03T21:19:34.971720+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_clean_scrub_stamp":"2024-05-03T21:18:41.788726+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T00:52:20.288003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":20,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-05-03T21:19:34.975229+0000","last_change":"2024-05-03T21:18:42.820157+0000","last_active":"2024-05-03T21:19:34.975229+0000","last_peered":"2024-05-03T21:19:34.975229+0000","last_clean":"2024-05-03T21:19:34.975229+0000","last_became_active":"2024-05-03T21:18:42.819450+0000","last_became_peered":"2024-05-03T21:18:42.819450+0000","last_unstale":"2024-05-03T21:19:34.975229+0000","last_undegraded":"2024-05-03T21:19:34.975229+0000","last_fullsized":"2024-05-03T21:19:34.975229+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_clean_scrub_stamp":"2024-05-03T21:18:41.788726+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T04:34:19.252499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":20,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-05-03T21:19:34.973800+0000","last_change":"2024-05-03T21:18:42.816582+0000","last_active":"2024-05-03T21:19:34.973800+0000","last_peered":"2024-05-03T21:19:34.973800+0000","last_clean":"2024-05-03T21:19:34.973800+0000","last_became_active":"2024-05-03T21:18:42.816247+0000","last_became_peered":"2024-05-03T21:18:42.816247+0000","last_unstale":"2024-05-03T21:19:34.973800+0000","last_undegraded":"2024-05-03T21:19:34.973800+0000","last_fullsized":"2024-05-03T21:19:34.973800+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_clean_scrub_stamp":"2024-05-03T21:18:41.788726+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T00:09:02.620176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":20,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-05-03T21:19:34.973840+0000","last_change":"2024-05-03T21:18:42.821587+0000","last_active":"2024-05-03T21:19:34.973840+0000","last_peered":"2024-05-03T21:19:34.973840+0000","last_clean":"2024-05-03T21:19:34.973840+0000","last_became_active":"2024-05-03T21:18:42.821236+0000","last_became_peered":"2024-05-03T21:18:42.821236+0000","last_unstale":"2024-05-03T21:19:34.973840+0000","last_undegraded":"2024-05-03T21:19:34.973840+0000","last_fullsized":"2024-05-03T21:19:34.973840+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_clean_scrub_stamp":"2024-05-03T21:18:41.788726+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-04T22:23:46.746787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"82'9","reported_seq":35,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-05-03T21:19:34.974229+0000","last_change":"2024-05-03T21:18:39.792001+0000","last_active":"2024-05-03T21:19:34.974229+0000","last_peered":"2024-05-03T21:19:34.974229+0000","last_clean":"2024-05-03T21:19:34.974229+0000","last_became_active":"2024-05-03T21:18:39.791644+0000","last_became_peered":"2024-05-03T21:18:39.791644+0000","last_unstale":"2024-05-03T21:19:34.974229+0000","last_undegraded":"2024-05-03T21:19:34.974229+0000","last_fullsized":"2024-05-03T21:19:34.974229+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T21:18:38.766727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T21:18:38.766727+0000","last_clean_scrub_stamp":"2024-05-03T21:18:38.766727+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-04T23:15:36.169243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":20,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-05-03T21:19:34.979208+0000","last_change":"2024-05-03T21:18:42.815115+0000","last_active":"2024-05-03T21:19:34.979208+0000","last_peered":"2024-05-03T21:19:34.979208+0000","last_clean":"2024-05-03T21:19:34.979208+0000","last_became_active":"2024-05-03T21:18:42.814734+0000","last_became_peered":"2024-05-03T21:18:42.814734+0000","last_unstale":"2024-05-03T21:19:34.979208+0000","last_undegraded":"2024-05-03T21:19:34.979208+0000","last_fullsized":"2024-05-03T21:19:34.979208+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_clean_scrub_stamp":"2024-05-03T21:18:41.788726+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-04T23:25:18.232982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"82'2","reported_seq":27,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-05-03T21:19:34.979497+0000","last_change":"2024-05-03T21:18:39.814107+0000","last_active":"2024-05-03T21:19:34.979497+0000","last_peered":"2024-05-03T21:19:34.979497+0000","last_clean":"2024-05-03T21:19:34.979497+0000","last_became_active":"2024-05-03T21:18:39.813615+0000","last_became_peered":"2024-05-03T21:18:39.813615+0000","last_unstale":"2024-05-03T21:19:34.979497+0000","last_undegraded":"2024-05-03T21:19:34.979497+0000","last_fullsized":"2024-05-03T21:19:34.979497+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T21:18:38.766727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T21:18:38.766727+0000","last_clean_scrub_stamp":"2024-05-03T21:18:38.766727+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-04T23:24:00.914805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":20,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-05-03T21:19:34.978900+0000","last_change":"2024-05-03T21:18:42.816925+0000","last_active":"2024-05-03T21:19:34.978900+0000","last_peered":"2024-05-03T21:19:34.978900+0000","last_clean":"2024-05-03T21:19:34.978900+0000","last_became_active":"2024-05-03T21:18:42.814731+0000","last_became_peered":"2024-05-03T21:18:42.814731+0000","last_unstale":"2024-05-03T21:19:34.978900+0000","last_undegraded":"2024-05-03T21:19:34.978900+0000","last_fullsized":"2024-05-03T21:19:34.978900+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_clean_scrub_stamp":"2024-05-03T21:18:41.788726+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T00:18:26.859784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"82'4","reported_seq":30,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-05-03T21:19:34.973033+0000","last_change":"2024-05-03T21:18:39.792101+0000","last_active":"2024-05-03T21:19:34.973033+0000","last_peered":"2024-05-03T21:19:34.973033+0000","last_clean":"2024-05-03T21:19:34.973033+0000","last_became_active":"2024-05-03T21:18:39.791725+0000","last_became_peered":"2024-05-03T21:18:39.791725+0000","last_unstale":"2024-05-03T21:19:34.973033+0000","last_undegraded":"2024-05-03T21:19:34.973033+0000","last_fullsized":"2024-05-03T21:19:34.973033+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T21:18:38.766727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T21:18:38.766727+0000","last_clean_scrub_stamp":"2024-05-03T21:18:38.766727+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T05:09:42.900318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":20,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-05-03T21:19:34.971846+0000","last_change":"2024-05-03T21:18:42.823318+0000","last_active":"2024-05-03T21:19:34.971846+0000","last_peered":"2024-05-03T21:19:34.971846+0000","last_clean":"2024-05-03T21:19:34.971846+0000","last_became_active":"2024-05-03T21:18:42.823011+0000","last_became_peered":"2024-05-03T21:18:42.823011+0000","last_unstale":"2024-05-03T21:19:34.971846+0000","last_undegraded":"2024-05-03T21:19:34.971846+0000","last_fullsized":"2024-05-03T21:19:34.971846+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_clean_scrub_stamp":"2024-05-03T21:18:41.788726+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T01:04:51.967269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"82'2","reported_seq":28,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-05-03T21:19:34.973609+0000","last_change":"2024-05-03T21:18:39.787730+0000","last_active":"2024-05-03T21:19:34.973609+0000","last_peered":"2024-05-03T21:19:34.973609+0000","last_clean":"2024-05-03T21:19:34.973609+0000","last_became_active":"2024-05-03T21:18:39.787403+0000","last_became_peered":"2024-05-03T21:18:39.787403+0000","last_unstale":"2024-05-03T21:19:34.973609+0000","last_undegraded":"2024-05-03T21:19:34.973609+0000","last_fullsized":"2024-05-03T21:19:34.973609+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T21:18:38.766727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T21:18:38.766727+0000","last_clean_scrub_stamp":"2024-05-03T21:18:38.766727+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T06:28:41.454461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":20,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-05-03T21:19:34.971641+0000","last_change":"2024-05-03T21:18:42.818987+0000","last_active":"2024-05-03T21:19:34.971641+0000","last_peered":"2024-05-03T21:19:34.971641+0000","last_clean":"2024-05-03T21:19:34.971641+0000","last_became_active":"2024-05-03T21:18:42.818645+0000","last_became_peered":"2024-05-03T21:18:42.818645+0000","last_unstale":"2024-05-03T21:19:34.971641+0000","last_undegraded":"2024-05-03T21:19:34.971641+0000","last_fullsized":"2024-05-03T21:19:34.971641+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_clean_scrub_stamp":"2024-05-03T21:18:41.788726+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T02:26:27.470390+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"82'6","reported_seq":132,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-05-03T21:20:36.031462+0000","last_change":"2024-05-03T21:18:39.793807+0000","last_active":"2024-05-03T21:20:36.031462+0000","last_peered":"2024-05-03T21:20:36.031462+0000","last_clean":"2024-05-03T21:20:36.031462+0000","last_became_active":"2024-05-03T21:18:39.793447+0000","last_became_peered":"2024-05-03T21:18:39.793447+0000","last_unstale":"2024-05-03T21:20:36.031462+0000","last_undegraded":"2024-05-03T21:20:36.031462+0000","last_fullsized":"2024-05-03T21:20:36.031462+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T21:18:38.766727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T21:18:38.766727+0000","last_clean_scrub_stamp":"2024-05-03T21:18:38.766727+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T06:44:55.036889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1013,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":100,"num_read_kb":3,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":20,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-05-03T21:19:34.972280+0000","last_change":"2024-05-03T21:18:42.807486+0000","last_active":"2024-05-03T21:19:34.972280+0000","last_peered":"2024-05-03T21:19:34.972280+0000","last_clean":"2024-05-03T21:19:34.972280+0000","last_became_active":"2024-05-03T21:18:42.807120+0000","last_became_peered":"2024-05-03T21:18:42.807120+0000","last_unstale":"2024-05-03T21:19:34.972280+0000","last_undegraded":"2024-05-03T21:19:34.972280+0000","last_fullsized":"2024-05-03T21:19:34.972280+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_clean_scrub_stamp":"2024-05-03T21:18:41.788726+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T05:32:38.621330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1a","version":"82'3","reported_seq":28,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-05-03T21:19:35.208026+0000","last_change":"2024-05-03T21:18:39.791943+0000","last_active":"2024-05-03T21:19:35.208026+0000","last_peered":"2024-05-03T21:19:35.208026+0000","last_clean":"2024-05-03T21:19:35.208026+0000","last_became_active":"2024-05-03T21:18:39.791587+0000","last_became_peered":"2024-05-03T21:18:39.791587+0000","last_unstale":"2024-05-03T21:19:35.208026+0000","last_undegraded":"2024-05-03T21:19:35.208026+0000","last_fullsized":"2024-05-03T21:19:35.208026+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T21:18:38.766727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T21:18:38.766727+0000","last_clean_scrub_stamp":"2024-05-03T21:18:38.766727+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T05:01:39.458019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":11,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-05-03T21:19:35.210398+0000","last_change":"2024-05-03T21:19:35.209904+0000","last_active":"2024-05-03T21:19:35.210398+0000","last_peered":"2024-05-03T21:19:35.210398+0000","last_clean":"2024-05-03T21:19:35.210398+0000","last_became_active":"2024-05-03T21:19:35.209475+0000","last_became_peered":"2024-05-03T21:19:35.209475+0000","last_unstale":"2024-05-03T21:19:35.210398+0000","last_undegraded":"2024-05-03T21:19:35.210398+0000","last_fullsized":"2024-05-03T21:19:35.210398+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_clean_scrub_stamp":"2024-05-03T21:18:41.788726+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T09:14:24.101832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4,3],"acting":[9,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.19","version":"82'2","reported_seq":27,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-05-03T21:19:34.973825+0000","last_change":"2024-05-03T21:18:39.784859+0000","last_active":"2024-05-03T21:19:34.973825+0000","last_peered":"2024-05-03T21:19:34.973825+0000","last_clean":"2024-05-03T21:19:34.973825+0000","last_became_active":"2024-05-03T21:18:39.784428+0000","last_became_peered":"2024-05-03T21:18:39.784428+0000","last_unstale":"2024-05-03T21:19:34.973825+0000","last_undegraded":"2024-05-03T21:19:34.973825+0000","last_fullsized":"2024-05-03T21:19:34.973825+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T21:18:38.766727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T21:18:38.766727+0000","last_clean_scrub_stamp":"2024-05-03T21:18:38.766727+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T05:56:18.368399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":20,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-05-03T21:19:34.978436+0000","last_change":"2024-05-03T21:18:42.813285+0000","last_active":"2024-05-03T21:19:34.978436+0000","last_peered":"2024-05-03T21:19:34.978436+0000","last_clean":"2024-05-03T21:19:34.978436+0000","last_became_active":"2024-05-03T21:18:42.812954+0000","last_became_peered":"2024-05-03T21:18:42.812954+0000","last_unstale":"2024-05-03T21:19:34.978436+0000","last_undegraded":"2024-05-03T21:19:34.978436+0000","last_fullsized":"2024-05-03T21:19:34.978436+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_clean_scrub_stamp":"2024-05-03T21:18:41.788726+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T00:19:26.841514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"82'1","reported_seq":27,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-05-03T21:19:34.971312+0000","last_change":"2024-05-03T21:18:39.793485+0000","last_active":"2024-05-03T21:19:34.971312+0000","last_peered":"2024-05-03T21:19:34.971312+0000","last_clean":"2024-05-03T21:19:34.971312+0000","last_became_active":"2024-05-03T21:18:39.793055+0000","last_became_peered":"2024-05-03T21:18:39.793055+0000","last_unstale":"2024-05-03T21:19:34.971312+0000","last_undegraded":"2024-05-03T21:19:34.971312+0000","last_fullsized":"2024-05-03T21:19:34.971312+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T21:18:38.766727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T21:18:38.766727+0000","last_clean_scrub_stamp":"2024-05-03T21:18:38.766727+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T03:10:59.880330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":20,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-05-03T21:19:34.973551+0000","last_change":"2024-05-03T21:18:42.815721+0000","last_active":"2024-05-03T21:19:34.973551+0000","last_peered":"2024-05-03T21:19:34.973551+0000","last_clean":"2024-05-03T21:19:34.973551+0000","last_became_active":"2024-05-03T21:18:42.815408+0000","last_became_peered":"2024-05-03T21:18:42.815408+0000","last_unstale":"2024-05-03T21:19:34.973551+0000","last_undegraded":"2024-05-03T21:19:34.973551+0000","last_fullsized":"2024-05-03T21:19:34.973551+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_clean_scrub_stamp":"2024-05-03T21:18:41.788726+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T03:03:46.851667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"82'3","reported_seq":123,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-05-03T21:20:34.027618+0000","last_change":"2024-05-03T21:18:39.792060+0000","last_active":"2024-05-03T21:20:34.027618+0000","last_peered":"2024-05-03T21:20:34.027618+0000","last_clean":"2024-05-03T21:20:34.027618+0000","last_became_active":"2024-05-03T21:18:39.791704+0000","last_became_peered":"2024-05-03T21:18:39.791704+0000","last_unstale":"2024-05-03T21:20:34.027618+0000","last_undegraded":"2024-05-03T21:20:34.027618+0000","last_fullsized":"2024-05-03T21:20:34.027618+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T21:18:38.766727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T21:18:38.766727+0000","last_clean_scrub_stamp":"2024-05-03T21:18:38.766727+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T03:46:43.784554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":95,"num_read_kb":95,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":20,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-05-03T21:19:34.972208+0000","last_change":"2024-05-03T21:18:42.823016+0000","last_active":"2024-05-03T21:19:34.972208+0000","last_peered":"2024-05-03T21:19:34.972208+0000","last_clean":"2024-05-03T21:19:34.972208+0000","last_became_active":"2024-05-03T21:18:42.822363+0000","last_became_peered":"2024-05-03T21:18:42.822363+0000","last_unstale":"2024-05-03T21:19:34.972208+0000","last_undegraded":"2024-05-03T21:19:34.972208+0000","last_fullsized":"2024-05-03T21:19:34.972208+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_clean_scrub_stamp":"2024-05-03T21:18:41.788726+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T07:46:36.213746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"82'2","reported_seq":27,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-05-03T21:19:34.974314+0000","last_change":"2024-05-03T21:18:39.791944+0000","last_active":"2024-05-03T21:19:34.974314+0000","last_peered":"2024-05-03T21:19:34.974314+0000","last_clean":"2024-05-03T21:19:34.974314+0000","last_became_active":"2024-05-03T21:18:39.791576+0000","last_became_peered":"2024-05-03T21:18:39.791576+0000","last_unstale":"2024-05-03T21:19:34.974314+0000","last_undegraded":"2024-05-03T21:19:34.974314+0000","last_fullsized":"2024-05-03T21:19:34.974314+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T21:18:38.766727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T21:18:38.766727+0000","last_clean_scrub_stamp":"2024-05-03T21:18:38.766727+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T07:05:58.190107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":20,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-05-03T21:19:34.978861+0000","last_change":"2024-05-03T21:18:42.819587+0000","last_active":"2024-05-03T21:19:34.978861+0000","last_peered":"2024-05-03T21:19:34.978861+0000","last_clean":"2024-05-03T21:19:34.978861+0000","last_became_active":"2024-05-03T21:18:42.819286+0000","last_became_peered":"2024-05-03T21:18:42.819286+0000","last_unstale":"2024-05-03T21:19:34.978861+0000","last_undegraded":"2024-05-03T21:19:34.978861+0000","last_fullsized":"2024-05-03T21:19:34.978861+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_clean_scrub_stamp":"2024-05-03T21:18:41.788726+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T01:31:34.367410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"82'4","reported_seq":31,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-05-03T21:19:34.971743+0000","last_change":"2024-05-03T21:18:39.815394+0000","last_active":"2024-05-03T21:19:34.971743+0000","last_peered":"2024-05-03T21:19:34.971743+0000","last_clean":"2024-05-03T21:19:34.971743+0000","last_became_active":"2024-05-03T21:18:39.814874+0000","last_became_peered":"2024-05-03T21:18:39.814874+0000","last_unstale":"2024-05-03T21:19:34.971743+0000","last_undegraded":"2024-05-03T21:19:34.971743+0000","last_fullsized":"2024-05-03T21:19:34.971743+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T21:18:38.766727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T21:18:38.766727+0000","last_clean_scrub_stamp":"2024-05-03T21:18:38.766727+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T08:42:50.532572+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":20,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-05-03T21:19:34.971885+0000","last_change":"2024-05-03T21:18:42.804483+0000","last_active":"2024-05-03T21:19:34.971885+0000","last_peered":"2024-05-03T21:19:34.971885+0000","last_clean":"2024-05-03T21:19:34.971885+0000","last_became_active":"2024-05-03T21:18:42.804064+0000","last_became_peered":"2024-05-03T21:18:42.804064+0000","last_unstale":"2024-05-03T21:19:34.971885+0000","last_undegraded":"2024-05-03T21:19:34.971885+0000","last_fullsized":"2024-05-03T21:19:34.971885+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_clean_scrub_stamp":"2024-05-03T21:18:41.788726+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T01:02:22.249152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"82'27","reported_seq":249,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-05-03T21:20:35.029363+0000","last_change":"2024-05-03T21:18:39.792864+0000","last_active":"2024-05-03T21:20:35.029363+0000","last_peered":"2024-05-03T21:20:35.029363+0000","last_clean":"2024-05-03T21:20:35.029363+0000","last_became_active":"2024-05-03T21:18:39.792469+0000","last_became_peered":"2024-05-03T21:18:39.792469+0000","last_unstale":"2024-05-03T21:20:35.029363+0000","last_undegraded":"2024-05-03T21:20:35.029363+0000","last_fullsized":"2024-05-03T21:20:35.029363+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T21:18:38.766727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T21:18:38.766727+0000","last_clean_scrub_stamp":"2024-05-03T21:18:38.766727+0000","objects_scrubbed":0,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-04T22:31:09.180649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51762,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":197,"num_read_kb":148,"num_write":28,"num_write_kb":59,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":20,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-05-03T21:19:34.972725+0000","last_change":"2024-05-03T21:18:42.816299+0000","last_active":"2024-05-03T21:19:34.972725+0000","last_peered":"2024-05-03T21:19:34.972725+0000","last_clean":"2024-05-03T21:19:34.972725+0000","last_became_active":"2024-05-03T21:18:42.815947+0000","last_became_peered":"2024-05-03T21:18:42.815947+0000","last_unstale":"2024-05-03T21:19:34.972725+0000","last_undegraded":"2024-05-03T21:19:34.972725+0000","last_fullsized":"2024-05-03T21:19:34.972725+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_clean_scrub_stamp":"2024-05-03T21:18:41.788726+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T07:52:21.620885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"82'6","reported_seq":32,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-05-03T21:19:34.973877+0000","last_change":"2024-05-03T21:18:39.794552+0000","last_active":"2024-05-03T21:19:34.973877+0000","last_peered":"2024-05-03T21:19:34.973877+0000","last_clean":"2024-05-03T21:19:34.973877+0000","last_became_active":"2024-05-03T21:18:39.794183+0000","last_became_peered":"2024-05-03T21:18:39.794183+0000","last_unstale":"2024-05-03T21:19:34.973877+0000","last_undegraded":"2024-05-03T21:19:34.973877+0000","last_fullsized":"2024-05-03T21:19:34.973877+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T21:18:38.766727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T21:18:38.766727+0000","last_clean_scrub_stamp":"2024-05-03T21:18:38.766727+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T00:16:32.820261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":20,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-05-03T21:19:34.973435+0000","last_change":"2024-05-03T21:18:42.811542+0000","last_active":"2024-05-03T21:19:34.973435+0000","last_peered":"2024-05-03T21:19:34.973435+0000","last_clean":"2024-05-03T21:19:34.973435+0000","last_became_active":"2024-05-03T21:18:42.810301+0000","last_became_peered":"2024-05-03T21:18:42.810301+0000","last_unstale":"2024-05-03T21:19:34.973435+0000","last_undegraded":"2024-05-03T21:19:34.973435+0000","last_fullsized":"2024-05-03T21:19:34.973435+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_clean_scrub_stamp":"2024-05-03T21:18:41.788726+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T03:01:30.503201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"82'3","reported_seq":29,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-05-03T21:19:34.971818+0000","last_change":"2024-05-03T21:18:39.796028+0000","last_active":"2024-05-03T21:19:34.971818+0000","last_peered":"2024-05-03T21:19:34.971818+0000","last_clean":"2024-05-03T21:19:34.971818+0000","last_became_active":"2024-05-03T21:18:39.795646+0000","last_became_peered":"2024-05-03T21:18:39.795646+0000","last_unstale":"2024-05-03T21:19:34.971818+0000","last_undegraded":"2024-05-03T21:19:34.971818+0000","last_fullsized":"2024-05-03T21:19:34.971818+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T21:18:38.766727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T21:18:38.766727+0000","last_clean_scrub_stamp":"2024-05-03T21:18:38.766727+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T01:56:31.128184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":20,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-05-03T21:19:34.974135+0000","last_change":"2024-05-03T21:18:42.810690+0000","last_active":"2024-05-03T21:19:34.974135+0000","last_peered":"2024-05-03T21:19:34.974135+0000","last_clean":"2024-05-03T21:19:34.974135+0000","last_became_active":"2024-05-03T21:18:42.810397+0000","last_became_peered":"2024-05-03T21:18:42.810397+0000","last_unstale":"2024-05-03T21:19:34.974135+0000","last_undegraded":"2024-05-03T21:19:34.974135+0000","last_fullsized":"2024-05-03T21:19:34.974135+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_clean_scrub_stamp":"2024-05-03T21:18:41.788726+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T09:14:43.292353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"82'4","reported_seq":130,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-05-03T21:20:36.031764+0000","last_change":"2024-05-03T21:18:39.793102+0000","last_active":"2024-05-03T21:20:36.031764+0000","last_peered":"2024-05-03T21:20:36.031764+0000","last_clean":"2024-05-03T21:20:36.031764+0000","last_became_active":"2024-05-03T21:18:39.792700+0000","last_became_peered":"2024-05-03T21:18:39.792700+0000","last_unstale":"2024-05-03T21:20:36.031764+0000","last_undegraded":"2024-05-03T21:20:36.031764+0000","last_fullsized":"2024-05-03T21:20:36.031764+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T21:18:38.766727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T21:18:38.766727+0000","last_clean_scrub_stamp":"2024-05-03T21:18:38.766727+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-04T21:32:36.761116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":979,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":101,"num_read_kb":4,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":20,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-05-03T21:19:34.974364+0000","last_change":"2024-05-03T21:18:42.815975+0000","last_active":"2024-05-03T21:19:34.974364+0000","last_peered":"2024-05-03T21:19:34.974364+0000","last_clean":"2024-05-03T21:19:34.974364+0000","last_became_active":"2024-05-03T21:18:42.814667+0000","last_became_peered":"2024-05-03T21:18:42.814667+0000","last_unstale":"2024-05-03T21:19:34.974364+0000","last_undegraded":"2024-05-03T21:19:34.974364+0000","last_fullsized":"2024-05-03T21:19:34.974364+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_clean_scrub_stamp":"2024-05-03T21:18:41.788726+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T00:50:47.991863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"82'15","reported_seq":29,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-05-03T21:19:34.995443+0000","last_change":"2024-05-03T21:19:34.994980+0000","last_active":"2024-05-03T21:19:34.995443+0000","last_peered":"2024-05-03T21:19:34.995443+0000","last_clean":"2024-05-03T21:19:34.995443+0000","last_became_active":"2024-05-03T21:19:34.979988+0000","last_became_peered":"2024-05-03T21:19:34.979988+0000","last_unstale":"2024-05-03T21:19:34.995443+0000","last_undegraded":"2024-05-03T21:19:34.995443+0000","last_fullsized":"2024-05-03T21:19:34.995443+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T21:18:38.766727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T21:18:38.766727+0000","last_clean_scrub_stamp":"2024-05-03T21:18:38.766727+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T01:14:44.807458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":112,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":20,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-05-03T21:19:34.971435+0000","last_change":"2024-05-03T21:18:42.810313+0000","last_active":"2024-05-03T21:19:34.971435+0000","last_peered":"2024-05-03T21:19:34.971435+0000","last_clean":"2024-05-03T21:19:34.971435+0000","last_became_active":"2024-05-03T21:18:42.809732+0000","last_became_peered":"2024-05-03T21:18:42.809732+0000","last_unstale":"2024-05-03T21:19:34.971435+0000","last_undegraded":"2024-05-03T21:19:34.971435+0000","last_fullsized":"2024-05-03T21:19:34.971435+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_clean_scrub_stamp":"2024-05-03T21:18:41.788726+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T01:36:54.077645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"82'2","reported_seq":29,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-05-03T21:19:34.972521+0000","last_change":"2024-05-03T21:18:39.784067+0000","last_active":"2024-05-03T21:19:34.972521+0000","last_peered":"2024-05-03T21:19:34.972521+0000","last_clean":"2024-05-03T21:19:34.972521+0000","last_became_active":"2024-05-03T21:18:39.783596+0000","last_became_peered":"2024-05-03T21:18:39.783596+0000","last_unstale":"2024-05-03T21:19:34.972521+0000","last_undegraded":"2024-05-03T21:19:34.972521+0000","last_fullsized":"2024-05-03T21:19:34.972521+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T21:18:38.766727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T21:18:38.766727+0000","last_clean_scrub_stamp":"2024-05-03T21:18:38.766727+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T03:10:25.925757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":20,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-05-03T21:19:34.974825+0000","last_change":"2024-05-03T21:18:42.817617+0000","last_active":"2024-05-03T21:19:34.974825+0000","last_peered":"2024-05-03T21:19:34.974825+0000","last_clean":"2024-05-03T21:19:34.974825+0000","last_became_active":"2024-05-03T21:18:42.817267+0000","last_became_peered":"2024-05-03T21:18:42.817267+0000","last_unstale":"2024-05-03T21:19:34.974825+0000","last_undegraded":"2024-05-03T21:19:34.974825+0000","last_fullsized":"2024-05-03T21:19:34.974825+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_clean_scrub_stamp":"2024-05-03T21:18:41.788726+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T08:00:12.864215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":25,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-05-03T21:19:34.973855+0000","last_change":"2024-05-03T21:18:39.814213+0000","last_active":"2024-05-03T21:19:34.973855+0000","last_peered":"2024-05-03T21:19:34.973855+0000","last_clean":"2024-05-03T21:19:34.973855+0000","last_became_active":"2024-05-03T21:18:39.813787+0000","last_became_peered":"2024-05-03T21:18:39.813787+0000","last_unstale":"2024-05-03T21:19:34.973855+0000","last_undegraded":"2024-05-03T21:19:34.973855+0000","last_fullsized":"2024-05-03T21:19:34.973855+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T21:18:38.766727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T21:18:38.766727+0000","last_clean_scrub_stamp":"2024-05-03T21:18:38.766727+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T08:45:17.963396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"82'1","reported_seq":21,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-05-03T21:19:35.208354+0000","last_change":"2024-05-03T21:18:42.821547+0000","last_active":"2024-05-03T21:19:35.208354+0000","last_peered":"2024-05-03T21:19:35.208354+0000","last_clean":"2024-05-03T21:19:35.208354+0000","last_became_active":"2024-05-03T21:18:42.821168+0000","last_became_peered":"2024-05-03T21:18:42.821168+0000","last_unstale":"2024-05-03T21:19:35.208354+0000","last_undegraded":"2024-05-03T21:19:35.208354+0000","last_fullsized":"2024-05-03T21:19:35.208354+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_clean_scrub_stamp":"2024-05-03T21:18:41.788726+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T07:28:25.037648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"82'1","reported_seq":26,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-05-03T21:19:34.973954+0000","last_change":"2024-05-03T21:18:39.792994+0000","last_active":"2024-05-03T21:19:34.973954+0000","last_peered":"2024-05-03T21:19:34.973954+0000","last_clean":"2024-05-03T21:19:34.973954+0000","last_became_active":"2024-05-03T21:18:39.792125+0000","last_became_peered":"2024-05-03T21:18:39.792125+0000","last_unstale":"2024-05-03T21:19:34.973954+0000","last_undegraded":"2024-05-03T21:19:34.973954+0000","last_fullsized":"2024-05-03T21:19:34.973954+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T21:18:38.766727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T21:18:38.766727+0000","last_clean_scrub_stamp":"2024-05-03T21:18:38.766727+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T03:09:45.917263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":20,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-05-03T21:19:35.208007+0000","last_change":"2024-05-03T21:18:42.814513+0000","last_active":"2024-05-03T21:19:35.208007+0000","last_peered":"2024-05-03T21:19:35.208007+0000","last_clean":"2024-05-03T21:19:35.208007+0000","last_became_active":"2024-05-03T21:18:42.814097+0000","last_became_peered":"2024-05-03T21:18:42.814097+0000","last_unstale":"2024-05-03T21:19:35.208007+0000","last_undegraded":"2024-05-03T21:19:35.208007+0000","last_fullsized":"2024-05-03T21:19:35.208007+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_clean_scrub_stamp":"2024-05-03T21:18:41.788726+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T06:52:46.731543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":20,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-05-03T21:19:34.971544+0000","last_change":"2024-05-03T21:18:42.812413+0000","last_active":"2024-05-03T21:19:34.971544+0000","last_peered":"2024-05-03T21:19:34.971544+0000","last_clean":"2024-05-03T21:19:34.971544+0000","last_became_active":"2024-05-03T21:18:42.812065+0000","last_became_peered":"2024-05-03T21:18:42.812065+0000","last_unstale":"2024-05-03T21:19:34.971544+0000","last_undegraded":"2024-05-03T21:19:34.971544+0000","last_fullsized":"2024-05-03T21:19:34.971544+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_clean_scrub_stamp":"2024-05-03T21:18:41.788726+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T07:52:44.489097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"82'4","reported_seq":126,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-05-03T21:20:36.031139+0000","last_change":"2024-05-03T21:18:39.813982+0000","last_active":"2024-05-03T21:20:36.031139+0000","last_peered":"2024-05-03T21:20:36.031139+0000","last_clean":"2024-05-03T21:20:36.031139+0000","last_became_active":"2024-05-03T21:18:39.813590+0000","last_became_peered":"2024-05-03T21:18:39.813590+0000","last_unstale":"2024-05-03T21:20:36.031139+0000","last_undegraded":"2024-05-03T21:20:36.031139+0000","last_fullsized":"2024-05-03T21:20:36.031139+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T21:18:38.766727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T21:18:38.766727+0000","last_clean_scrub_stamp":"2024-05-03T21:18:38.766727+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T02:30:24.733268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":97,"num_read_kb":97,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":20,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-05-03T21:19:34.978749+0000","last_change":"2024-05-03T21:18:42.808357+0000","last_active":"2024-05-03T21:19:34.978749+0000","last_peered":"2024-05-03T21:19:34.978749+0000","last_clean":"2024-05-03T21:19:34.978749+0000","last_became_active":"2024-05-03T21:18:42.808023+0000","last_became_peered":"2024-05-03T21:18:42.808023+0000","last_unstale":"2024-05-03T21:19:34.978749+0000","last_undegraded":"2024-05-03T21:19:34.978749+0000","last_fullsized":"2024-05-03T21:19:34.978749+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_clean_scrub_stamp":"2024-05-03T21:18:41.788726+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T08:33:11.713491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"82'3","reported_seq":31,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-05-03T21:19:35.208863+0000","last_change":"2024-05-03T21:18:39.787052+0000","last_active":"2024-05-03T21:19:35.208863+0000","last_peered":"2024-05-03T21:19:35.208863+0000","last_clean":"2024-05-03T21:19:35.208863+0000","last_became_active":"2024-05-03T21:18:39.786390+0000","last_became_peered":"2024-05-03T21:18:39.786390+0000","last_unstale":"2024-05-03T21:19:35.208863+0000","last_undegraded":"2024-05-03T21:19:35.208863+0000","last_fullsized":"2024-05-03T21:19:35.208863+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T21:18:38.766727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T21:18:38.766727+0000","last_clean_scrub_stamp":"2024-05-03T21:18:38.766727+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T08:18:27.832378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"21'92","reported_seq":100,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-05-03T21:19:34.974544+0000","last_change":"2024-05-03T21:17:17.301485+0000","last_active":"2024-05-03T21:19:34.974544+0000","last_peered":"2024-05-03T21:19:34.974544+0000","last_clean":"2024-05-03T21:19:34.974544+0000","last_became_active":"2024-05-03T21:17:17.296458+0000","last_became_peered":"2024-05-03T21:17:17.296458+0000","last_unstale":"2024-05-03T21:19:34.974544+0000","last_undegraded":"2024-05-03T21:19:34.974544+0000","last_fullsized":"2024-05-03T21:19:34.974544+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T21:14:00.058823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T21:14:00.058823+0000","last_clean_scrub_stamp":"2024-05-03T21:14:00.058823+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T03:13:14.815888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":26,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-05-03T21:19:34.982618+0000","last_change":"2024-05-03T21:19:34.982618+0000","last_active":"2024-05-03T21:19:34.982618+0000","last_peered":"2024-05-03T21:19:34.982618+0000","last_clean":"2024-05-03T21:19:34.982618+0000","last_became_active":"2024-05-03T21:19:34.982259+0000","last_became_peered":"2024-05-03T21:19:34.982259+0000","last_unstale":"2024-05-03T21:19:34.982618+0000","last_undegraded":"2024-05-03T21:19:34.982618+0000","last_fullsized":"2024-05-03T21:19:34.982618+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_clean_scrub_stamp":"2024-05-03T21:18:41.788726+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-04T23:33:44.083386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,9,2],"acting":[6,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"82'3","reported_seq":38,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-05-03T21:19:35.213111+0000","last_change":"2024-05-03T21:19:35.213111+0000","last_active":"2024-05-03T21:19:35.213111+0000","last_peered":"2024-05-03T21:19:35.213111+0000","last_clean":"2024-05-03T21:19:35.213111+0000","last_became_active":"2024-05-03T21:19:35.210041+0000","last_became_peered":"2024-05-03T21:19:35.210041+0000","last_unstale":"2024-05-03T21:19:35.213111+0000","last_undegraded":"2024-05-03T21:19:35.213111+0000","last_fullsized":"2024-05-03T21:19:35.213111+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T21:18:38.766727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T21:18:38.766727+0000","last_clean_scrub_stamp":"2024-05-03T21:18:38.766727+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T02:40:22.026186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,4],"acting":[7,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"82'1","reported_seq":28,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-05-03T21:19:34.972907+0000","last_change":"2024-05-03T21:18:39.814671+0000","last_active":"2024-05-03T21:19:34.972907+0000","last_peered":"2024-05-03T21:19:34.972907+0000","last_clean":"2024-05-03T21:19:34.972907+0000","last_became_active":"2024-05-03T21:18:39.814326+0000","last_became_peered":"2024-05-03T21:18:39.814326+0000","last_unstale":"2024-05-03T21:19:34.972907+0000","last_undegraded":"2024-05-03T21:19:34.972907+0000","last_fullsized":"2024-05-03T21:19:34.972907+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T21:18:38.766727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T21:18:38.766727+0000","last_clean_scrub_stamp":"2024-05-03T21:18:38.766727+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T01:21:53.058203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":20,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-05-03T21:19:34.973632+0000","last_change":"2024-05-03T21:18:42.815525+0000","last_active":"2024-05-03T21:19:34.973632+0000","last_peered":"2024-05-03T21:19:34.973632+0000","last_clean":"2024-05-03T21:19:34.973632+0000","last_became_active":"2024-05-03T21:18:42.815209+0000","last_became_peered":"2024-05-03T21:18:42.815209+0000","last_unstale":"2024-05-03T21:19:34.973632+0000","last_undegraded":"2024-05-03T21:19:34.973632+0000","last_fullsized":"2024-05-03T21:19:34.973632+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_clean_scrub_stamp":"2024-05-03T21:18:41.788726+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T01:02:51.545090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"84'10","reported_seq":35,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-05-03T21:19:34.973954+0000","last_change":"2024-05-03T21:18:39.784592+0000","last_active":"2024-05-03T21:19:34.973954+0000","last_peered":"2024-05-03T21:19:34.973954+0000","last_clean":"2024-05-03T21:19:34.973954+0000","last_became_active":"2024-05-03T21:18:39.784242+0000","last_became_peered":"2024-05-03T21:18:39.784242+0000","last_unstale":"2024-05-03T21:19:34.973954+0000","last_undegraded":"2024-05-03T21:19:34.973954+0000","last_fullsized":"2024-05-03T21:19:34.973954+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T21:18:38.766727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T21:18:38.766727+0000","last_clean_scrub_stamp":"2024-05-03T21:18:38.766727+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-04T22:23:01.914553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":20,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-05-03T21:19:34.979646+0000","last_change":"2024-05-03T21:18:42.814361+0000","last_active":"2024-05-03T21:19:34.979646+0000","last_peered":"2024-05-03T21:19:34.979646+0000","last_clean":"2024-05-03T21:19:34.979646+0000","last_became_active":"2024-05-03T21:18:42.813448+0000","last_became_peered":"2024-05-03T21:18:42.813448+0000","last_unstale":"2024-05-03T21:19:34.979646+0000","last_undegraded":"2024-05-03T21:19:34.979646+0000","last_fullsized":"2024-05-03T21:19:34.979646+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_clean_scrub_stamp":"2024-05-03T21:18:41.788726+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T06:21:11.974428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"82'4","reported_seq":30,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-05-03T21:19:35.209116+0000","last_change":"2024-05-03T21:18:39.905533+0000","last_active":"2024-05-03T21:19:35.209116+0000","last_peered":"2024-05-03T21:19:35.209116+0000","last_clean":"2024-05-03T21:19:35.209116+0000","last_became_active":"2024-05-03T21:18:39.905256+0000","last_became_peered":"2024-05-03T21:18:39.905256+0000","last_unstale":"2024-05-03T21:19:35.209116+0000","last_undegraded":"2024-05-03T21:19:35.209116+0000","last_fullsized":"2024-05-03T21:19:35.209116+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T21:18:38.766727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T21:18:38.766727+0000","last_clean_scrub_stamp":"2024-05-03T21:18:38.766727+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T07:07:12.162628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":20,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-05-03T21:19:34.979935+0000","last_change":"2024-05-03T21:18:42.821052+0000","last_active":"2024-05-03T21:19:34.979935+0000","last_peered":"2024-05-03T21:19:34.979935+0000","last_clean":"2024-05-03T21:19:34.979935+0000","last_became_active":"2024-05-03T21:18:42.820751+0000","last_became_peered":"2024-05-03T21:18:42.820751+0000","last_unstale":"2024-05-03T21:19:34.979935+0000","last_undegraded":"2024-05-03T21:19:34.979935+0000","last_fullsized":"2024-05-03T21:19:34.979935+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_clean_scrub_stamp":"2024-05-03T21:18:41.788726+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T02:14:45.829778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"82'2","reported_seq":28,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-05-03T21:19:34.971736+0000","last_change":"2024-05-03T21:18:39.814793+0000","last_active":"2024-05-03T21:19:34.971736+0000","last_peered":"2024-05-03T21:19:34.971736+0000","last_clean":"2024-05-03T21:19:34.971736+0000","last_became_active":"2024-05-03T21:18:39.814427+0000","last_became_peered":"2024-05-03T21:18:39.814427+0000","last_unstale":"2024-05-03T21:19:34.971736+0000","last_undegraded":"2024-05-03T21:19:34.971736+0000","last_fullsized":"2024-05-03T21:19:34.971736+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T21:18:38.766727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T21:18:38.766727+0000","last_clean_scrub_stamp":"2024-05-03T21:18:38.766727+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T01:01:34.673401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":20,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-05-03T21:19:34.972106+0000","last_change":"2024-05-03T21:18:42.812123+0000","last_active":"2024-05-03T21:19:34.972106+0000","last_peered":"2024-05-03T21:19:34.972106+0000","last_clean":"2024-05-03T21:19:34.972106+0000","last_became_active":"2024-05-03T21:18:42.811799+0000","last_became_peered":"2024-05-03T21:18:42.811799+0000","last_unstale":"2024-05-03T21:19:34.972106+0000","last_undegraded":"2024-05-03T21:19:34.972106+0000","last_fullsized":"2024-05-03T21:19:34.972106+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_clean_scrub_stamp":"2024-05-03T21:18:41.788726+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T05:47:42.995000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"82'3","reported_seq":28,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-05-03T21:19:34.978710+0000","last_change":"2024-05-03T21:18:39.786729+0000","last_active":"2024-05-03T21:19:34.978710+0000","last_peered":"2024-05-03T21:19:34.978710+0000","last_clean":"2024-05-03T21:19:34.978710+0000","last_became_active":"2024-05-03T21:18:39.786276+0000","last_became_peered":"2024-05-03T21:18:39.786276+0000","last_unstale":"2024-05-03T21:19:34.978710+0000","last_undegraded":"2024-05-03T21:19:34.978710+0000","last_fullsized":"2024-05-03T21:19:34.978710+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T21:18:38.766727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T21:18:38.766727+0000","last_clean_scrub_stamp":"2024-05-03T21:18:38.766727+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T00:47:34.921468+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":20,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-05-03T21:19:34.975309+0000","last_change":"2024-05-03T21:18:42.818135+0000","last_active":"2024-05-03T21:19:34.975309+0000","last_peered":"2024-05-03T21:19:34.975309+0000","last_clean":"2024-05-03T21:19:34.975309+0000","last_became_active":"2024-05-03T21:18:42.816868+0000","last_became_peered":"2024-05-03T21:18:42.816868+0000","last_unstale":"2024-05-03T21:19:34.975309+0000","last_undegraded":"2024-05-03T21:19:34.975309+0000","last_fullsized":"2024-05-03T21:19:34.975309+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_clean_scrub_stamp":"2024-05-03T21:18:41.788726+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T00:36:26.783820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"82'1","reported_seq":15,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-05-03T21:19:34.984158+0000","last_change":"2024-05-03T21:19:34.983584+0000","last_active":"2024-05-03T21:19:34.984158+0000","last_peered":"2024-05-03T21:19:34.984158+0000","last_clean":"2024-05-03T21:19:34.984158+0000","last_became_active":"2024-05-03T21:19:34.979470+0000","last_became_peered":"2024-05-03T21:19:34.979470+0000","last_unstale":"2024-05-03T21:19:34.984158+0000","last_undegraded":"2024-05-03T21:19:34.984158+0000","last_fullsized":"2024-05-03T21:19:34.984158+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T21:18:38.766727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T21:18:38.766727+0000","last_clean_scrub_stamp":"2024-05-03T21:18:38.766727+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T06:10:19.528492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,3,8],"acting":[11,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":20,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-05-03T21:19:34.979444+0000","last_change":"2024-05-03T21:18:42.819671+0000","last_active":"2024-05-03T21:19:34.979444+0000","last_peered":"2024-05-03T21:19:34.979444+0000","last_clean":"2024-05-03T21:19:34.979444+0000","last_became_active":"2024-05-03T21:18:42.819336+0000","last_became_peered":"2024-05-03T21:18:42.819336+0000","last_unstale":"2024-05-03T21:19:34.979444+0000","last_undegraded":"2024-05-03T21:19:34.979444+0000","last_fullsized":"2024-05-03T21:19:34.979444+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_clean_scrub_stamp":"2024-05-03T21:18:41.788726+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T04:54:03.726522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"82'4","reported_seq":43,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-05-03T21:19:35.216509+0000","last_change":"2024-05-03T21:19:35.216509+0000","last_active":"2024-05-03T21:19:35.216509+0000","last_peered":"2024-05-03T21:19:35.216509+0000","last_clean":"2024-05-03T21:19:35.216509+0000","last_became_active":"2024-05-03T21:19:35.210095+0000","last_became_peered":"2024-05-03T21:19:35.210095+0000","last_unstale":"2024-05-03T21:19:35.216509+0000","last_undegraded":"2024-05-03T21:19:35.216509+0000","last_fullsized":"2024-05-03T21:19:35.216509+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T21:18:38.766727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T21:18:38.766727+0000","last_clean_scrub_stamp":"2024-05-03T21:18:38.766727+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T00:42:29.519443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,4,7],"acting":[8,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":20,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-05-03T21:19:34.974077+0000","last_change":"2024-05-03T21:18:42.810587+0000","last_active":"2024-05-03T21:19:34.974077+0000","last_peered":"2024-05-03T21:19:34.974077+0000","last_clean":"2024-05-03T21:19:34.974077+0000","last_became_active":"2024-05-03T21:18:42.810274+0000","last_became_peered":"2024-05-03T21:18:42.810274+0000","last_unstale":"2024-05-03T21:19:34.974077+0000","last_undegraded":"2024-05-03T21:19:34.974077+0000","last_fullsized":"2024-05-03T21:19:34.974077+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_clean_scrub_stamp":"2024-05-03T21:18:41.788726+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T02:45:20.891119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"82'3","reported_seq":29,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-05-03T21:19:34.973775+0000","last_change":"2024-05-03T21:18:39.905882+0000","last_active":"2024-05-03T21:19:34.973775+0000","last_peered":"2024-05-03T21:19:34.973775+0000","last_clean":"2024-05-03T21:19:34.973775+0000","last_became_active":"2024-05-03T21:18:39.905488+0000","last_became_peered":"2024-05-03T21:18:39.905488+0000","last_unstale":"2024-05-03T21:19:34.973775+0000","last_undegraded":"2024-05-03T21:19:34.973775+0000","last_fullsized":"2024-05-03T21:19:34.973775+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T21:18:38.766727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T21:18:38.766727+0000","last_clean_scrub_stamp":"2024-05-03T21:18:38.766727+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T02:02:17.913931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":20,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-05-03T21:19:34.974175+0000","last_change":"2024-05-03T21:18:42.818693+0000","last_active":"2024-05-03T21:19:34.974175+0000","last_peered":"2024-05-03T21:19:34.974175+0000","last_clean":"2024-05-03T21:19:34.974175+0000","last_became_active":"2024-05-03T21:18:42.818177+0000","last_became_peered":"2024-05-03T21:18:42.818177+0000","last_unstale":"2024-05-03T21:19:34.974175+0000","last_undegraded":"2024-05-03T21:19:34.974175+0000","last_fullsized":"2024-05-03T21:19:34.974175+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_clean_scrub_stamp":"2024-05-03T21:18:41.788726+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T03:05:10.708866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"82'1","reported_seq":28,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-05-03T21:19:34.971519+0000","last_change":"2024-05-03T21:18:39.792399+0000","last_active":"2024-05-03T21:19:34.971519+0000","last_peered":"2024-05-03T21:19:34.971519+0000","last_clean":"2024-05-03T21:19:34.971519+0000","last_became_active":"2024-05-03T21:18:39.791462+0000","last_became_peered":"2024-05-03T21:18:39.791462+0000","last_unstale":"2024-05-03T21:19:34.971519+0000","last_undegraded":"2024-05-03T21:19:34.971519+0000","last_fullsized":"2024-05-03T21:19:34.971519+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T21:18:38.766727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T21:18:38.766727+0000","last_clean_scrub_stamp":"2024-05-03T21:18:38.766727+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T08:24:23.045438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"82'1","reported_seq":21,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-05-03T21:19:34.973967+0000","last_change":"2024-05-03T21:18:42.813275+0000","last_active":"2024-05-03T21:19:34.973967+0000","last_peered":"2024-05-03T21:19:34.973967+0000","last_clean":"2024-05-03T21:19:34.973967+0000","last_became_active":"2024-05-03T21:18:42.812640+0000","last_became_peered":"2024-05-03T21:18:42.812640+0000","last_unstale":"2024-05-03T21:19:34.973967+0000","last_undegraded":"2024-05-03T21:19:34.973967+0000","last_fullsized":"2024-05-03T21:19:34.973967+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_clean_scrub_stamp":"2024-05-03T21:18:41.788726+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T01:34:27.071382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"82'2","reported_seq":27,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-05-03T21:19:34.973896+0000","last_change":"2024-05-03T21:18:39.787487+0000","last_active":"2024-05-03T21:19:34.973896+0000","last_peered":"2024-05-03T21:19:34.973896+0000","last_clean":"2024-05-03T21:19:34.973896+0000","last_became_active":"2024-05-03T21:18:39.787135+0000","last_became_peered":"2024-05-03T21:18:39.787135+0000","last_unstale":"2024-05-03T21:19:34.973896+0000","last_undegraded":"2024-05-03T21:19:34.973896+0000","last_fullsized":"2024-05-03T21:19:34.973896+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T21:18:38.766727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T21:18:38.766727+0000","last_clean_scrub_stamp":"2024-05-03T21:18:38.766727+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T05:21:13.304600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":20,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-05-03T21:19:34.974534+0000","last_change":"2024-05-03T21:18:42.815682+0000","last_active":"2024-05-03T21:19:34.974534+0000","last_peered":"2024-05-03T21:19:34.974534+0000","last_clean":"2024-05-03T21:19:34.974534+0000","last_became_active":"2024-05-03T21:18:42.813011+0000","last_became_peered":"2024-05-03T21:18:42.813011+0000","last_unstale":"2024-05-03T21:19:34.974534+0000","last_undegraded":"2024-05-03T21:19:34.974534+0000","last_fullsized":"2024-05-03T21:19:34.974534+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_clean_scrub_stamp":"2024-05-03T21:18:41.788726+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T03:47:21.680152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":20,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-05-03T21:19:34.975419+0000","last_change":"2024-05-03T21:18:42.817574+0000","last_active":"2024-05-03T21:19:34.975419+0000","last_peered":"2024-05-03T21:19:34.975419+0000","last_clean":"2024-05-03T21:19:34.975419+0000","last_became_active":"2024-05-03T21:18:42.817227+0000","last_became_peered":"2024-05-03T21:18:42.817227+0000","last_unstale":"2024-05-03T21:19:34.975419+0000","last_undegraded":"2024-05-03T21:19:34.975419+0000","last_fullsized":"2024-05-03T21:19:34.975419+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_clean_scrub_stamp":"2024-05-03T21:18:41.788726+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-04T23:15:34.784285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":26,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-05-03T21:19:34.982916+0000","last_change":"2024-05-03T21:19:34.982916+0000","last_active":"2024-05-03T21:19:34.982916+0000","last_peered":"2024-05-03T21:19:34.982916+0000","last_clean":"2024-05-03T21:19:34.982916+0000","last_became_active":"2024-05-03T21:19:34.982343+0000","last_became_peered":"2024-05-03T21:19:34.982343+0000","last_unstale":"2024-05-03T21:19:34.982916+0000","last_undegraded":"2024-05-03T21:19:34.982916+0000","last_fullsized":"2024-05-03T21:19:34.982916+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_clean_scrub_stamp":"2024-05-03T21:18:41.788726+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T07:39:56.301149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,8],"acting":[2,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":20,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-05-03T21:19:34.973231+0000","last_change":"2024-05-03T21:18:42.823980+0000","last_active":"2024-05-03T21:19:34.973231+0000","last_peered":"2024-05-03T21:19:34.973231+0000","last_clean":"2024-05-03T21:19:34.973231+0000","last_became_active":"2024-05-03T21:18:42.823619+0000","last_became_peered":"2024-05-03T21:18:42.823619+0000","last_unstale":"2024-05-03T21:19:34.973231+0000","last_undegraded":"2024-05-03T21:19:34.973231+0000","last_fullsized":"2024-05-03T21:19:34.973231+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_clean_scrub_stamp":"2024-05-03T21:18:41.788726+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T08:13:53.104446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":20,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-05-03T21:19:34.979401+0000","last_change":"2024-05-03T21:18:42.813596+0000","last_active":"2024-05-03T21:19:34.979401+0000","last_peered":"2024-05-03T21:19:34.979401+0000","last_clean":"2024-05-03T21:19:34.979401+0000","last_became_active":"2024-05-03T21:18:42.813238+0000","last_became_peered":"2024-05-03T21:18:42.813238+0000","last_unstale":"2024-05-03T21:19:34.979401+0000","last_undegraded":"2024-05-03T21:19:34.979401+0000","last_fullsized":"2024-05-03T21:19:34.979401+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_clean_scrub_stamp":"2024-05-03T21:18:41.788726+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T05:14:52.712566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":20,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-05-03T21:19:34.973476+0000","last_change":"2024-05-03T21:18:42.812252+0000","last_active":"2024-05-03T21:19:34.973476+0000","last_peered":"2024-05-03T21:19:34.973476+0000","last_clean":"2024-05-03T21:19:34.973476+0000","last_became_active":"2024-05-03T21:18:42.811917+0000","last_became_peered":"2024-05-03T21:18:42.811917+0000","last_unstale":"2024-05-03T21:19:34.973476+0000","last_undegraded":"2024-05-03T21:19:34.973476+0000","last_fullsized":"2024-05-03T21:19:34.973476+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_clean_scrub_stamp":"2024-05-03T21:18:41.788726+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T00:55:09.512412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":20,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-05-03T21:19:34.974830+0000","last_change":"2024-05-03T21:18:42.819345+0000","last_active":"2024-05-03T21:19:34.974830+0000","last_peered":"2024-05-03T21:19:34.974830+0000","last_clean":"2024-05-03T21:19:34.974830+0000","last_became_active":"2024-05-03T21:18:42.817901+0000","last_became_peered":"2024-05-03T21:18:42.817901+0000","last_unstale":"2024-05-03T21:19:34.974830+0000","last_undegraded":"2024-05-03T21:19:34.974830+0000","last_fullsized":"2024-05-03T21:19:34.974830+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_clean_scrub_stamp":"2024-05-03T21:18:41.788726+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T07:25:30.498261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":20,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-05-03T21:19:34.973679+0000","last_change":"2024-05-03T21:18:42.825009+0000","last_active":"2024-05-03T21:19:34.973679+0000","last_peered":"2024-05-03T21:19:34.973679+0000","last_clean":"2024-05-03T21:19:34.973679+0000","last_became_active":"2024-05-03T21:18:42.824626+0000","last_became_peered":"2024-05-03T21:18:42.824626+0000","last_unstale":"2024-05-03T21:19:34.973679+0000","last_undegraded":"2024-05-03T21:19:34.973679+0000","last_fullsized":"2024-05-03T21:19:34.973679+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_clean_scrub_stamp":"2024-05-03T21:18:41.788726+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T06:48:17.760340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":20,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-05-03T21:19:34.978336+0000","last_change":"2024-05-03T21:18:42.812305+0000","last_active":"2024-05-03T21:19:34.978336+0000","last_peered":"2024-05-03T21:19:34.978336+0000","last_clean":"2024-05-03T21:19:34.978336+0000","last_became_active":"2024-05-03T21:18:42.811738+0000","last_became_peered":"2024-05-03T21:18:42.811738+0000","last_unstale":"2024-05-03T21:19:34.978336+0000","last_undegraded":"2024-05-03T21:19:34.978336+0000","last_fullsized":"2024-05-03T21:19:34.978336+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_clean_scrub_stamp":"2024-05-03T21:18:41.788726+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-04T21:41:24.312517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":20,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-05-03T21:19:34.971389+0000","last_change":"2024-05-03T21:18:42.817306+0000","last_active":"2024-05-03T21:19:34.971389+0000","last_peered":"2024-05-03T21:19:34.971389+0000","last_clean":"2024-05-03T21:19:34.971389+0000","last_became_active":"2024-05-03T21:18:42.816768+0000","last_became_peered":"2024-05-03T21:18:42.816768+0000","last_unstale":"2024-05-03T21:19:34.971389+0000","last_undegraded":"2024-05-03T21:19:34.971389+0000","last_fullsized":"2024-05-03T21:19:34.971389+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_clean_scrub_stamp":"2024-05-03T21:18:41.788726+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-04T23:24:42.055568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":20,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-05-03T21:19:34.978607+0000","last_change":"2024-05-03T21:18:42.816008+0000","last_active":"2024-05-03T21:19:34.978607+0000","last_peered":"2024-05-03T21:19:34.978607+0000","last_clean":"2024-05-03T21:19:34.978607+0000","last_became_active":"2024-05-03T21:18:42.815653+0000","last_became_peered":"2024-05-03T21:18:42.815653+0000","last_unstale":"2024-05-03T21:19:34.978607+0000","last_undegraded":"2024-05-03T21:19:34.978607+0000","last_fullsized":"2024-05-03T21:19:34.978607+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_clean_scrub_stamp":"2024-05-03T21:18:41.788726+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-04T23:15:22.986562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":20,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-05-03T21:19:34.971889+0000","last_change":"2024-05-03T21:18:42.814820+0000","last_active":"2024-05-03T21:19:34.971889+0000","last_peered":"2024-05-03T21:19:34.971889+0000","last_clean":"2024-05-03T21:19:34.971889+0000","last_became_active":"2024-05-03T21:18:42.814498+0000","last_became_peered":"2024-05-03T21:18:42.814498+0000","last_unstale":"2024-05-03T21:19:34.971889+0000","last_undegraded":"2024-05-03T21:19:34.971889+0000","last_fullsized":"2024-05-03T21:19:34.971889+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_clean_scrub_stamp":"2024-05-03T21:18:41.788726+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T01:35:15.727585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":20,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-05-03T21:19:34.973615+0000","last_change":"2024-05-03T21:18:42.814701+0000","last_active":"2024-05-03T21:19:34.973615+0000","last_peered":"2024-05-03T21:19:34.973615+0000","last_clean":"2024-05-03T21:19:34.973615+0000","last_became_active":"2024-05-03T21:18:42.814394+0000","last_became_peered":"2024-05-03T21:18:42.814394+0000","last_unstale":"2024-05-03T21:19:34.973615+0000","last_undegraded":"2024-05-03T21:19:34.973615+0000","last_fullsized":"2024-05-03T21:19:34.973615+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_clean_scrub_stamp":"2024-05-03T21:18:41.788726+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T04:50:05.941873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":20,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-05-03T21:19:34.974476+0000","last_change":"2024-05-03T21:18:42.815987+0000","last_active":"2024-05-03T21:19:34.974476+0000","last_peered":"2024-05-03T21:19:34.974476+0000","last_clean":"2024-05-03T21:19:34.974476+0000","last_became_active":"2024-05-03T21:18:42.815613+0000","last_became_peered":"2024-05-03T21:18:42.815613+0000","last_unstale":"2024-05-03T21:19:34.974476+0000","last_undegraded":"2024-05-03T21:19:34.974476+0000","last_fullsized":"2024-05-03T21:19:34.974476+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T21:18:41.788726+0000","last_clean_scrub_stamp":"2024-05-03T21:18:41.788726+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T03:57:01.997504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4751360,"data_stored":4722944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":92,"ondisk_log_size":92,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":56154,"num_objects":60,"num_object_clones":0,"num_object_copies":180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":0,"num_read":602,"num_read_kb":359,"num_write":161,"num_write_kb":101,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":112,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":393216,"data_stored":168462,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":74,"seq":317827579943,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27912,"kb_used_data":892,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717240,"statfs":{"total":95995035648,"available":95966453760,"internally_reserved":0,"allocated":913408,"data_stored":442207,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":68,"seq":292057776171,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28580,"kb_used_data":1556,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716572,"statfs":{"total":95995035648,"available":95965769728,"internally_reserved":0,"allocated":1593344,"data_stored":1085729,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":63,"seq":270582939697,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32084,"kb_used_data":900,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713068,"statfs":{"total":95995035648,"available":95962181632,"internally_reserved":0,"allocated":921600,"data_stored":442331,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":57,"seq":244813135927,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32076,"kb_used_data":892,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713076,"statfs":{"total":95995035648,"available":95962189824,"internally_reserved":0,"allocated":913408,"data_stored":442139,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":50,"seq":214748364858,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28500,"kb_used_data":1480,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716652,"statfs":{"total":95995035648,"available":95965851648,"internally_reserved":0,"allocated":1515520,"data_stored":1034572,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":45,"seq":193273528384,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27912,"kb_used_data":888,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717240,"statfs":{"total":95995035648,"available":95966453760,"internally_reserved":0,"allocated":909312,"data_stored":441720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574937,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36284,"kb_used_data":940,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708868,"statfs":{"total":95995035648,"available":95957880832,"internally_reserved":0,"allocated":962560,"data_stored":493275,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738462,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36252,"kb_used_data":908,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708900,"statfs":{"total":95995035648,"available":95957913600,"internally_reserved":0,"allocated":929792,"data_stored":442991,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411411,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36308,"kb_used_data":964,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708844,"statfs":{"total":95995035648,"available":95957856256,"internally_reserved":0,"allocated":987136,"data_stored":495016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":27,"seq":115964117070,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36244,"kb_used_data":904,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708908,"statfs":{"total":95995035648,"available":95957921792,"internally_reserved":0,"allocated":925696,"data_stored":442031,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920841,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27912,"kb_used_data":892,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717240,"statfs":{"total":95995035648,"available":95966453760,"internally_reserved":0,"allocated":913408,"data_stored":442548,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724614,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32672,"kb_used_data":1492,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712480,"statfs":{"total":95995035648,"available":95961579520,"internally_reserved":0,"allocated":1527808,"data_stored":1033865,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":51762,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":53503,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1035,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1867,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2574,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":626,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":94208,"data_stored":53731,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-05-03T21:20:40.234 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-05-03T21:20:40.234 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-05-03T21:20:40.234 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2024-05-03T21:20:40.235 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9a047fe7599a549042d06a3fb5b6813a203eabea shell --fsid 713bc57a-0991-11ef-bc97-c7b262605968 -- ceph health --format=json 2024-05-03T21:20:40.521 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/713bc57a-0991-11ef-bc97-c7b262605968/mon.a/config 2024-05-03T21:20:41.347 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:20:41 smithi067 ceph-mon[29730]: from='client.14922 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-03T21:20:41.348 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:20:41 smithi067 ceph-mon[29730]: pgmap v347: 97 pgs: 97 active+clean; 632 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 85 B/s wr, 5 op/s 2024-05-03T21:20:41.495 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:20:41 smithi082 ceph-mon[32933]: from='client.14922 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-03T21:20:41.495 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:20:41 smithi082 ceph-mon[32933]: pgmap v347: 97 pgs: 97 active+clean; 632 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 85 B/s wr, 5 op/s 2024-05-03T21:20:41.574 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:20:41 smithi138 ceph-mon[33110]: from='client.14922 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-03T21:20:41.574 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:20:41 smithi138 ceph-mon[33110]: pgmap v347: 97 pgs: 97 active+clean; 632 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 85 B/s wr, 5 op/s 2024-05-03T21:20:42.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:42.129+0000 7f0216521700 1 -- 172.21.15.67:0/2379593626 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0210108410 msgr2=0x7f02101087f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:20:42.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:42.129+0000 7f0216521700 1 --2- 172.21.15.67:0/2379593626 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0210108410 0x7f02101087f0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f0200009990 tx=0x7f0200009ca0 comp rx=0 tx=0).stop 2024-05-03T21:20:42.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:42.129+0000 7f0216521700 1 -- 172.21.15.67:0/2379593626 shutdown_connections 2024-05-03T21:20:42.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:42.129+0000 7f0216521700 1 --2- 172.21.15.67:0/2379593626 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0210108410 0x7f02101087f0 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:20:42.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:42.129+0000 7f0216521700 1 --2- 172.21.15.67:0/2379593626 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f0210102d60 0x7f021010f250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:20:42.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:42.129+0000 7f0216521700 1 --2- 172.21.15.67:0/2379593626 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f02101023a0 0x7f0210102820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:20:42.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:42.129+0000 7f0216521700 1 -- 172.21.15.67:0/2379593626 >> 172.21.15.67:0/2379593626 conn(0x7f0210079350 msgr2=0x7f0210079760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T21:20:42.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:42.129+0000 7f0216521700 1 -- 172.21.15.67:0/2379593626 shutdown_connections 2024-05-03T21:20:42.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:42.129+0000 7f0216521700 1 -- 172.21.15.67:0/2379593626 wait complete. 2024-05-03T21:20:42.133 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:42.130+0000 7f0216521700 1 Processor -- start 2024-05-03T21:20:42.133 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:42.130+0000 7f0216521700 1 -- start start 2024-05-03T21:20:42.134 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:42.130+0000 7f0216521700 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f02101023a0 0x7f0210104080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:20:42.134 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:42.130+0000 7f0216521700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f0210102d60 0x7f02101045c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:20:42.134 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:42.130+0000 7f0216521700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0210108410 0x7f0210104b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:20:42.134 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:42.130+0000 7f0216521700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f0210112840 con 0x7f0210108410 2024-05-03T21:20:42.134 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:42.130+0000 7f0216521700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f02101126c0 con 0x7f0210102d60 2024-05-03T21:20:42.134 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:42.130+0000 7f0216521700 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f02101129c0 con 0x7f02101023a0 2024-05-03T21:20:42.135 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:42.130+0000 7f0214abe700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0210108410 0x7f0210104b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:20:42.135 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:42.130+0000 7f0214abe700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0210108410 0x7f0210104b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:51412/0 (socket says 172.21.15.67:51412) 2024-05-03T21:20:42.135 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:42.130+0000 7f0214abe700 1 -- 172.21.15.67:0/1908045422 learned_addr learned my addr 172.21.15.67:0/1908045422 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-03T21:20:42.135 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:42.130+0000 7f020f7fe700 1 --2- 172.21.15.67:0/1908045422 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f0210102d60 0x7f02101045c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:20:42.135 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:42.130+0000 7f020ffff700 1 --2- 172.21.15.67:0/1908045422 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f02101023a0 0x7f0210104080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:20:42.135 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:42.131+0000 7f0214abe700 1 -- 172.21.15.67:0/1908045422 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f02101023a0 msgr2=0x7f0210104080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:20:42.135 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:42.131+0000 7f0214abe700 1 --2- 172.21.15.67:0/1908045422 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f02101023a0 0x7f0210104080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:20:42.135 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:42.131+0000 7f0214abe700 1 -- 172.21.15.67:0/1908045422 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f0210102d60 msgr2=0x7f02101045c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:20:42.135 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:42.131+0000 7f0214abe700 1 --2- 172.21.15.67:0/1908045422 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f0210102d60 0x7f02101045c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:20:42.135 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:42.131+0000 7f0214abe700 1 -- 172.21.15.67:0/1908045422 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f02101a8210 con 0x7f0210108410 2024-05-03T21:20:42.135 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:42.131+0000 7f0214abe700 1 --2- 172.21.15.67:0/1908045422 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0210108410 0x7f0210104b00 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f0208007d70 tx=0x7f020800a340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T21:20:42.135 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:42.131+0000 7f020d7fa700 1 -- 172.21.15.67:0/1908045422 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0208016020 con 0x7f0210108410 2024-05-03T21:20:42.135 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:42.131+0000 7f0216521700 1 -- 172.21.15.67:0/1908045422 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f02101a8500 con 0x7f0210108410 2024-05-03T21:20:42.136 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:42.131+0000 7f020d7fa700 1 -- 172.21.15.67:0/1908045422 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(28 keys) ==== 1170+0+0 (secure 0 0 0) 0x7f020800ac90 con 0x7f0210108410 2024-05-03T21:20:42.136 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:42.131+0000 7f020d7fa700 1 -- 172.21.15.67:0/1908045422 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0208011620 con 0x7f0210108410 2024-05-03T21:20:42.136 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:42.131+0000 7f0216521700 1 -- 172.21.15.67:0/1908045422 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f02101a8a40 con 0x7f0210108410 2024-05-03T21:20:42.136 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:42.132+0000 7f0216521700 1 -- 172.21.15.67:0/1908045422 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f021006a190 con 0x7f0210108410 2024-05-03T21:20:42.136 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:42.134+0000 7f020d7fa700 1 -- 172.21.15.67:0/1908045422 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 17) ==== 149703+0+0 (secure 0 0 0) 0x7f0208019020 con 0x7f0210108410 2024-05-03T21:20:42.137 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:42.135+0000 7f020d7fa700 1 --2- 172.21.15.67:0/1908045422 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f01f80b2910 0x7f01f80b4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:20:42.137 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:42.135+0000 7f020ffff700 1 --2- 172.21.15.67:0/1908045422 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f01f80b2910 0x7f01f80b4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:20:42.137 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:42.135+0000 7f020d7fa700 1 -- 172.21.15.67:0/1908045422 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 8211+0+0 (secure 0 0 0) 0x7f02080e2540 con 0x7f0210108410 2024-05-03T21:20:42.138 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:42.135+0000 7f020ffff700 1 --2- 172.21.15.67:0/1908045422 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f01f80b2910 0x7f01f80b4dd0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f02000098e0 tx=0x7f0200005960 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T21:20:42.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:42.138+0000 7f020d7fa700 1 -- 172.21.15.67:0/1908045422 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191839 (secure 0 0 0) 0x7f0208020020 con 0x7f0210108410 2024-05-03T21:20:42.415 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:42.413+0000 7f0216521700 1 -- 172.21.15.67:0/1908045422 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) -- 0x7f02101a8d00 con 0x7f0210108410 2024-05-03T21:20:42.416 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:42.413+0000 7f020d7fa700 1 -- 172.21.15.67:0/1908045422 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) ==== 72+0+46 (secure 0 0 0) 0x7f02080aa220 con 0x7f0210108410 2024-05-03T21:20:42.416 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-05-03T21:20:42.416 INFO:teuthology.orchestra.run.smithi067.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-05-03T21:20:42.418 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:42.415+0000 7f0216521700 1 -- 172.21.15.67:0/1908045422 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f01f80b2910 msgr2=0x7f01f80b4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:20:42.418 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:42.415+0000 7f0216521700 1 --2- 172.21.15.67:0/1908045422 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f01f80b2910 0x7f01f80b4dd0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f02000098e0 tx=0x7f0200005960 comp rx=0 tx=0).stop 2024-05-03T21:20:42.418 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:42.415+0000 7f0216521700 1 -- 172.21.15.67:0/1908045422 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0210108410 msgr2=0x7f0210104b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:20:42.418 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:42.415+0000 7f0216521700 1 --2- 172.21.15.67:0/1908045422 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0210108410 0x7f0210104b00 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f0208007d70 tx=0x7f020800a340 comp rx=0 tx=0).stop 2024-05-03T21:20:42.418 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:42.416+0000 7f0216521700 1 -- 172.21.15.67:0/1908045422 shutdown_connections 2024-05-03T21:20:42.418 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:42.416+0000 7f0216521700 1 --2- 172.21.15.67:0/1908045422 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f01f80b2910 0x7f01f80b4dd0 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:20:42.418 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:42.416+0000 7f0216521700 1 --2- 172.21.15.67:0/1908045422 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0210108410 0x7f0210104b00 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:20:42.419 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:42.416+0000 7f0216521700 1 --2- 172.21.15.67:0/1908045422 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f02101023a0 0x7f0210104080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:20:42.419 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:42.416+0000 7f0216521700 1 --2- 172.21.15.67:0/1908045422 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f0210102d60 0x7f02101045c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:20:42.419 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:42.416+0000 7f0216521700 1 -- 172.21.15.67:0/1908045422 >> 172.21.15.67:0/1908045422 conn(0x7f0210079350 msgr2=0x7f0210101650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T21:20:42.419 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:42.416+0000 7f0216521700 1 -- 172.21.15.67:0/1908045422 shutdown_connections 2024-05-03T21:20:42.419 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:42.416+0000 7f0216521700 1 -- 172.21.15.67:0/1908045422 wait complete. 2024-05-03T21:20:43.043 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy done 2024-05-03T21:20:43.043 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9a047fe7599a549042d06a3fb5b6813a203eabea shell --fsid 713bc57a-0991-11ef-bc97-c7b262605968 -- ceph fs dump --format=json 2024-05-03T21:20:43.293 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/713bc57a-0991-11ef-bc97-c7b262605968/mon.a/config 2024-05-03T21:20:43.407 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:20:43 smithi067 ceph-mon[29730]: pgmap v348: 97 pgs: 97 active+clean; 632 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 85 B/s wr, 6 op/s 2024-05-03T21:20:43.407 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:20:43 smithi067 ceph-mon[29730]: from='client.? 172.21.15.67:0/1908045422' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-05-03T21:20:43.495 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:20:43 smithi082 ceph-mon[32933]: pgmap v348: 97 pgs: 97 active+clean; 632 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 85 B/s wr, 6 op/s 2024-05-03T21:20:43.495 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:20:43 smithi082 ceph-mon[32933]: from='client.? 172.21.15.67:0/1908045422' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-05-03T21:20:43.575 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:20:43 smithi138 ceph-mon[33110]: pgmap v348: 97 pgs: 97 active+clean; 632 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 85 B/s wr, 6 op/s 2024-05-03T21:20:43.575 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:20:43 smithi138 ceph-mon[33110]: from='client.? 172.21.15.67:0/1908045422' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-05-03T21:20:44.873 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:44.870+0000 7f2b84dde700 1 -- 172.21.15.67:0/1066212458 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2b80103750 msgr2=0x7f2b80103b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:20:44.874 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:44.870+0000 7f2b84dde700 1 --2- 172.21.15.67:0/1066212458 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2b80103750 0x7f2b80103b30 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f2b7000b0d0 tx=0x7f2b7000b3e0 comp rx=0 tx=0).stop 2024-05-03T21:20:44.874 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:44.870+0000 7f2b84dde700 1 -- 172.21.15.67:0/1066212458 shutdown_connections 2024-05-03T21:20:44.874 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:44.870+0000 7f2b84dde700 1 --2- 172.21.15.67:0/1066212458 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2b80103750 0x7f2b80103b30 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:20:44.874 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:44.870+0000 7f2b84dde700 1 --2- 172.21.15.67:0/1066212458 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f2b80108850 0x7f2b801103d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:20:44.874 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:44.870+0000 7f2b84dde700 1 --2- 172.21.15.67:0/1066212458 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f2b80104100 0x7f2b80108120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:20:44.874 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:44.870+0000 7f2b84dde700 1 -- 172.21.15.67:0/1066212458 >> 172.21.15.67:0/1066212458 conn(0x7f2b8007a600 msgr2=0x7f2b8007aa10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T21:20:44.874 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:44.871+0000 7f2b84dde700 1 -- 172.21.15.67:0/1066212458 shutdown_connections 2024-05-03T21:20:44.874 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:44.871+0000 7f2b84dde700 1 -- 172.21.15.67:0/1066212458 wait complete. 2024-05-03T21:20:44.874 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:44.871+0000 7f2b84dde700 1 Processor -- start 2024-05-03T21:20:44.874 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:44.871+0000 7f2b84dde700 1 -- start start 2024-05-03T21:20:44.876 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:44.872+0000 7f2b84dde700 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f2b80103750 0x7f2b800757a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:20:44.876 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:44.872+0000 7f2b84dde700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2b80104100 0x7f2b80075ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:20:44.876 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:44.872+0000 7f2b84dde700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f2b80108850 0x7f2b80076220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:20:44.876 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:44.872+0000 7f2b84dde700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f2b80113920 con 0x7f2b80104100 2024-05-03T21:20:44.876 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:44.872+0000 7f2b84dde700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f2b801137a0 con 0x7f2b80108850 2024-05-03T21:20:44.876 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:44.872+0000 7f2b84dde700 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f2b80113aa0 con 0x7f2b80103750 2024-05-03T21:20:44.877 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:44.872+0000 7f2b7dd9b700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2b80104100 0x7f2b80075ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:20:44.877 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:44.872+0000 7f2b7dd9b700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2b80104100 0x7f2b80075ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:51426/0 (socket says 172.21.15.67:51426) 2024-05-03T21:20:44.877 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:44.872+0000 7f2b7dd9b700 1 -- 172.21.15.67:0/2313745539 learned_addr learned my addr 172.21.15.67:0/2313745539 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-03T21:20:44.877 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:44.872+0000 7f2b7e59c700 1 --2- 172.21.15.67:0/2313745539 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f2b80103750 0x7f2b800757a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:20:44.877 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:44.872+0000 7f2b7ed9d700 1 --2- 172.21.15.67:0/2313745539 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f2b80108850 0x7f2b80076220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:20:44.877 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:44.872+0000 7f2b7dd9b700 1 -- 172.21.15.67:0/2313745539 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f2b80103750 msgr2=0x7f2b800757a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:20:44.877 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:44.872+0000 7f2b7dd9b700 1 --2- 172.21.15.67:0/2313745539 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f2b80103750 0x7f2b800757a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:20:44.877 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:44.872+0000 7f2b7dd9b700 1 -- 172.21.15.67:0/2313745539 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f2b80108850 msgr2=0x7f2b80076220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:20:44.877 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:44.872+0000 7f2b7dd9b700 1 --2- 172.21.15.67:0/2313745539 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f2b80108850 0x7f2b80076220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:20:44.877 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:44.872+0000 7f2b7dd9b700 1 -- 172.21.15.67:0/2313745539 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2b801a8280 con 0x7f2b80104100 2024-05-03T21:20:44.877 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:44.872+0000 7f2b7ed9d700 1 --2- 172.21.15.67:0/2313745539 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f2b80108850 0x7f2b80076220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-03T21:20:44.877 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:44.872+0000 7f2b7dd9b700 1 --2- 172.21.15.67:0/2313745539 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2b80104100 0x7f2b80075ce0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f2b6800b530 tx=0x7f2b6800b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T21:20:44.877 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:44.872+0000 7f2b777fe700 1 -- 172.21.15.67:0/2313745539 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b68015020 con 0x7f2b80104100 2024-05-03T21:20:44.878 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:44.873+0000 7f2b84dde700 1 -- 172.21.15.67:0/2313745539 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2b801a8570 con 0x7f2b80104100 2024-05-03T21:20:44.878 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:44.873+0000 7f2b777fe700 1 -- 172.21.15.67:0/2313745539 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(28 keys) ==== 1170+0+0 (secure 0 0 0) 0x7f2b68011d30 con 0x7f2b80104100 2024-05-03T21:20:44.878 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:44.873+0000 7f2b777fe700 1 -- 172.21.15.67:0/2313745539 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b6800f2e0 con 0x7f2b80104100 2024-05-03T21:20:44.878 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:44.873+0000 7f2b84dde700 1 -- 172.21.15.67:0/2313745539 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2b801a8ab0 con 0x7f2b80104100 2024-05-03T21:20:44.878 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:44.874+0000 7f2b84dde700 1 -- 172.21.15.67:0/2313745539 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2b8010ba90 con 0x7f2b80104100 2024-05-03T21:20:44.879 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:44.875+0000 7f2b777fe700 1 -- 172.21.15.67:0/2313745539 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 17) ==== 149703+0+0 (secure 0 0 0) 0x7f2b68018020 con 0x7f2b80104100 2024-05-03T21:20:44.879 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:44.875+0000 7f2b777fe700 1 --2- 172.21.15.67:0/2313745539 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f2b6c0b2990 0x7f2b6c0b4e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:20:44.879 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:44.876+0000 7f2b7e59c700 1 --2- 172.21.15.67:0/2313745539 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f2b6c0b2990 0x7f2b6c0b4e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:20:44.879 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:44.876+0000 7f2b777fe700 1 -- 172.21.15.67:0/2313745539 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 8211+0+0 (secure 0 0 0) 0x7f2b680e1a70 con 0x7f2b80104100 2024-05-03T21:20:44.879 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:44.876+0000 7f2b7e59c700 1 --2- 172.21.15.67:0/2313745539 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f2b6c0b2990 0x7f2b6c0b4e50 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f2b70009e60 tx=0x7f2b7000b040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T21:20:44.882 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:44.880+0000 7f2b777fe700 1 -- 172.21.15.67:0/2313745539 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191839 (secure 0 0 0) 0x7f2b6801f020 con 0x7f2b80104100 2024-05-03T21:20:45.142 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:45.139+0000 7f2b84dde700 1 -- 172.21.15.67:0/2313745539 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f2b8006a190 con 0x7f2b80104100 2024-05-03T21:20:45.143 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:45.140+0000 7f2b777fe700 1 -- 172.21.15.67:0/2313745539 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+11868 (secure 0 0 0) 0x7f2b680a9750 con 0x7f2b80104100 2024-05-03T21:20:45.143 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-05-03T21:20:45.143 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14541,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6835/4224826546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":4224826546},{"type":"v1","addr":"172.21.15.67:6835","nonce":4224826546}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14553,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6837/2485231621","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6836","nonce":2485231621},{"type":"v1","addr":"172.21.15.67:6837","nonce":2485231621}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14598,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6839/2106078262","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6838","nonce":2106078262},{"type":"v1","addr":"172.21.15.67:6839","nonce":2106078262}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14610,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6841/1559196215","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6840","nonce":1559196215},{"type":"v1","addr":"172.21.15.67:6841","nonce":1559196215}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24376,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.138:6833/3028022602","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6832","nonce":3028022602},{"type":"v1","addr":"172.21.15.138:6833","nonce":3028022602}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24400,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.138:6835/3342864697","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6834","nonce":3342864697},{"type":"v1","addr":"172.21.15.138:6835","nonce":3342864697}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7}],"filesystems":[{"mdsmap":{"epoch":25,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-05-03T21:18:44.665825+0000","modified":"2024-05-03T21:19:03.775546+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24473,"mds_1":24449,"mds_2":24425},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24406":{"gid":24406,"name":"c","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.138:6837/2203273249","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6836","nonce":2203273249},{"type":"v1","addr":"172.21.15.138:6837","nonce":2203273249}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24407":{"gid":24407,"name":"k","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.82:6833/1127287628","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6832","nonce":1127287628},{"type":"v1","addr":"172.21.15.82:6833","nonce":1127287628}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24418":{"gid":24418,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.138:6839/334382443","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6838","nonce":334382443},{"type":"v1","addr":"172.21.15.138:6839","nonce":334382443}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24425":{"gid":24425,"name":"e","rank":2,"incarnation":21,"state":"up:active","state_seq":14,"addr":"172.21.15.82:6835/280518761","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6834","nonce":280518761},{"type":"v1","addr":"172.21.15.82:6835","nonce":280518761}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24449":{"gid":24449,"name":"h","rank":1,"incarnation":19,"state":"up:active","state_seq":11,"addr":"172.21.15.82:6837/1275987852","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6836","nonce":1275987852},{"type":"v1","addr":"172.21.15.82:6837","nonce":1275987852}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24473":{"gid":24473,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":5,"addr":"172.21.15.82:6839/3644499578","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6838","nonce":3644499578},{"type":"v1","addr":"172.21.15.82:6839","nonce":3644499578}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24473,"qdb_cluster":[24449,24473,24425]},"id":1}]} 2024-05-03T21:20:45.145 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:45.142+0000 7f2b84dde700 1 -- 172.21.15.67:0/2313745539 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f2b6c0b2990 msgr2=0x7f2b6c0b4e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:20:45.145 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:45.142+0000 7f2b84dde700 1 --2- 172.21.15.67:0/2313745539 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f2b6c0b2990 0x7f2b6c0b4e50 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f2b70009e60 tx=0x7f2b7000b040 comp rx=0 tx=0).stop 2024-05-03T21:20:45.145 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:45.142+0000 7f2b84dde700 1 -- 172.21.15.67:0/2313745539 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2b80104100 msgr2=0x7f2b80075ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:20:45.145 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:45.142+0000 7f2b84dde700 1 --2- 172.21.15.67:0/2313745539 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2b80104100 0x7f2b80075ce0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f2b6800b530 tx=0x7f2b6800b8f0 comp rx=0 tx=0).stop 2024-05-03T21:20:45.145 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:45.142+0000 7f2b84dde700 1 -- 172.21.15.67:0/2313745539 shutdown_connections 2024-05-03T21:20:45.145 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:45.142+0000 7f2b84dde700 1 --2- 172.21.15.67:0/2313745539 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f2b6c0b2990 0x7f2b6c0b4e50 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:20:45.145 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:45.142+0000 7f2b84dde700 1 --2- 172.21.15.67:0/2313745539 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f2b80108850 0x7f2b80076220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:20:45.145 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:45.142+0000 7f2b84dde700 1 --2- 172.21.15.67:0/2313745539 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f2b80103750 0x7f2b800757a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:20:45.146 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:45.142+0000 7f2b84dde700 1 --2- 172.21.15.67:0/2313745539 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2b80104100 0x7f2b80075ce0 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:20:45.146 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:45.142+0000 7f2b84dde700 1 -- 172.21.15.67:0/2313745539 >> 172.21.15.67:0/2313745539 conn(0x7f2b8007a600 msgr2=0x7f2b80102c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T21:20:45.146 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:45.143+0000 7f2b84dde700 1 -- 172.21.15.67:0/2313745539 shutdown_connections 2024-05-03T21:20:45.146 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:45.143+0000 7f2b84dde700 1 -- 172.21.15.67:0/2313745539 wait complete. 2024-05-03T21:20:45.146 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 25 2024-05-03T21:20:45.453 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:20:45 smithi067 ceph-mon[29730]: pgmap v349: 97 pgs: 97 active+clean; 632 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 85 B/s wr, 6 op/s 2024-05-03T21:20:45.453 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:20:45 smithi067 ceph-mon[29730]: from='client.? 172.21.15.67:0/2313745539' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-03T21:20:45.495 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:20:45 smithi082 ceph-mon[32933]: pgmap v349: 97 pgs: 97 active+clean; 632 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 85 B/s wr, 6 op/s 2024-05-03T21:20:45.495 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:20:45 smithi082 ceph-mon[32933]: from='client.? 172.21.15.67:0/2313745539' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-03T21:20:45.574 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:20:45 smithi138 ceph-mon[33110]: pgmap v349: 97 pgs: 97 active+clean; 632 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 85 B/s wr, 6 op/s 2024-05-03T21:20:45.574 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:20:45 smithi138 ceph-mon[33110]: from='client.? 172.21.15.67:0/2313745539' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-03T21:20:45.724 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9a047fe7599a549042d06a3fb5b6813a203eabea shell --fsid 713bc57a-0991-11ef-bc97-c7b262605968 -- ceph fs dump --format=json 2024-05-03T21:20:45.975 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/713bc57a-0991-11ef-bc97-c7b262605968/mon.a/config 2024-05-03T21:20:47.363 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:20:47 smithi067 ceph-mon[29730]: pgmap v350: 97 pgs: 97 active+clean; 632 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 85 B/s wr, 6 op/s 2024-05-03T21:20:47.363 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:20:47 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T21:20:47.495 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:20:47 smithi082 ceph-mon[32933]: pgmap v350: 97 pgs: 97 active+clean; 632 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 85 B/s wr, 6 op/s 2024-05-03T21:20:47.495 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:20:47 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T21:20:47.572 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:47.557+0000 7fc7f0212700 1 -- 172.21.15.67:0/1480282402 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc7e8108410 msgr2=0x7fc7e81087f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:20:47.572 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:47.557+0000 7fc7f0212700 1 --2- 172.21.15.67:0/1480282402 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc7e8108410 0x7fc7e81087f0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7fc7e4009990 tx=0x7fc7e4009ca0 comp rx=0 tx=0).stop 2024-05-03T21:20:47.572 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:47.557+0000 7fc7f0212700 1 -- 172.21.15.67:0/1480282402 shutdown_connections 2024-05-03T21:20:47.572 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:47.557+0000 7fc7f0212700 1 --2- 172.21.15.67:0/1480282402 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc7e8108410 0x7fc7e81087f0 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:20:47.573 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:47.557+0000 7fc7f0212700 1 --2- 172.21.15.67:0/1480282402 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fc7e8102d60 0x7fc7e810f250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:20:47.573 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:47.557+0000 7fc7f0212700 1 --2- 172.21.15.67:0/1480282402 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fc7e81023a0 0x7fc7e8102820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:20:47.573 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:47.557+0000 7fc7f0212700 1 -- 172.21.15.67:0/1480282402 >> 172.21.15.67:0/1480282402 conn(0x7fc7e8079350 msgr2=0x7fc7e8079760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T21:20:47.573 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:47.557+0000 7fc7f0212700 1 -- 172.21.15.67:0/1480282402 shutdown_connections 2024-05-03T21:20:47.573 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:47.557+0000 7fc7f0212700 1 -- 172.21.15.67:0/1480282402 wait complete. 2024-05-03T21:20:47.573 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:47.558+0000 7fc7f0212700 1 Processor -- start 2024-05-03T21:20:47.573 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:47.558+0000 7fc7f0212700 1 -- start start 2024-05-03T21:20:47.573 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:47.558+0000 7fc7f0212700 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fc7e81023a0 0x7fc7e819abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:20:47.573 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:47.558+0000 7fc7f0212700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fc7e8102d60 0x7fc7e819b100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:20:47.573 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:47.558+0000 7fc7f0212700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc7e8108410 0x7fc7e81a5c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:20:47.573 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:47.558+0000 7fc7f0212700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fc7e8112840 con 0x7fc7e8108410 2024-05-03T21:20:47.573 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:47.558+0000 7fc7f0212700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7fc7e81126c0 con 0x7fc7e8102d60 2024-05-03T21:20:47.574 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:47.558+0000 7fc7f0212700 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7fc7e81129c0 con 0x7fc7e81023a0 2024-05-03T21:20:47.574 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:47.558+0000 7fc7ee7af700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc7e8108410 0x7fc7e81a5c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:20:47.574 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:47.558+0000 7fc7ee7af700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc7e8108410 0x7fc7e81a5c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:59698/0 (socket says 172.21.15.67:59698) 2024-05-03T21:20:47.574 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:47.558+0000 7fc7ee7af700 1 -- 172.21.15.67:0/3322677715 learned_addr learned my addr 172.21.15.67:0/3322677715 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-03T21:20:47.574 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:47.558+0000 7fc7ed7ad700 1 --2- 172.21.15.67:0/3322677715 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fc7e8102d60 0x7fc7e819b100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:20:47.574 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:47.558+0000 7fc7edfae700 1 --2- 172.21.15.67:0/3322677715 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fc7e81023a0 0x7fc7e819abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:20:47.574 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:47.558+0000 7fc7ee7af700 1 -- 172.21.15.67:0/3322677715 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fc7e81023a0 msgr2=0x7fc7e819abc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:20:47.574 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:47.558+0000 7fc7ee7af700 1 --2- 172.21.15.67:0/3322677715 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fc7e81023a0 0x7fc7e819abc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:20:47.574 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:47.558+0000 7fc7ee7af700 1 -- 172.21.15.67:0/3322677715 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fc7e8102d60 msgr2=0x7fc7e819b100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:20:47.574 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:47.558+0000 7fc7ee7af700 1 --2- 172.21.15.67:0/3322677715 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fc7e8102d60 0x7fc7e819b100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:20:47.574 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:47.558+0000 7fc7ee7af700 1 -- 172.21.15.67:0/3322677715 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc7e81a6180 con 0x7fc7e8108410 2024-05-03T21:20:47.574 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:47.558+0000 7fc7edfae700 1 --2- 172.21.15.67:0/3322677715 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fc7e81023a0 0x7fc7e819abc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-03T21:20:47.574 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:47.558+0000 7fc7ee7af700 1 --2- 172.21.15.67:0/3322677715 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc7e8108410 0x7fc7e81a5c40 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7fc7e000ef30 tx=0x7fc7e000c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T21:20:47.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:47.559+0000 7fc7deffd700 1 -- 172.21.15.67:0/3322677715 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7e00092e0 con 0x7fc7e8108410 2024-05-03T21:20:47.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:47.559+0000 7fc7f0212700 1 -- 172.21.15.67:0/3322677715 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc7e81a6350 con 0x7fc7e8108410 2024-05-03T21:20:47.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:47.559+0000 7fc7deffd700 1 -- 172.21.15.67:0/3322677715 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(28 keys) ==== 1170+0+0 (secure 0 0 0) 0x7fc7e0004bb0 con 0x7fc7e8108410 2024-05-03T21:20:47.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:47.559+0000 7fc7deffd700 1 -- 172.21.15.67:0/3322677715 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7e0004280 con 0x7fc7e8108410 2024-05-03T21:20:47.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:47.559+0000 7fc7f0212700 1 -- 172.21.15.67:0/3322677715 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc7e81a68b0 con 0x7fc7e8108410 2024-05-03T21:20:47.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:47.560+0000 7fc7f0212700 1 -- 172.21.15.67:0/3322677715 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc7e806a190 con 0x7fc7e8108410 2024-05-03T21:20:47.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:47.561+0000 7fc7deffd700 1 -- 172.21.15.67:0/3322677715 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 17) ==== 149703+0+0 (secure 0 0 0) 0x7fc7e001b030 con 0x7fc7e8108410 2024-05-03T21:20:47.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:47.562+0000 7fc7deffd700 1 --2- 172.21.15.67:0/3322677715 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7fc7d40b2910 0x7fc7d40b4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:20:47.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:47.562+0000 7fc7edfae700 1 --2- 172.21.15.67:0/3322677715 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7fc7d40b2910 0x7fc7d40b4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:20:47.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:47.562+0000 7fc7deffd700 1 -- 172.21.15.67:0/3322677715 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 8211+0+0 (secure 0 0 0) 0x7fc7e00e2800 con 0x7fc7e8108410 2024-05-03T21:20:47.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:47.562+0000 7fc7edfae700 1 --2- 172.21.15.67:0/3322677715 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7fc7d40b2910 0x7fc7d40b4dd0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fc7e400b2e0 tx=0x7fc7e4009690 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T21:20:47.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:47.565+0000 7fc7deffd700 1 -- 172.21.15.67:0/3322677715 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191839 (secure 0 0 0) 0x7fc7e0022020 con 0x7fc7e8108410 2024-05-03T21:20:47.576 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:20:47 smithi138 ceph-mon[33110]: pgmap v350: 97 pgs: 97 active+clean; 632 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 85 B/s wr, 6 op/s 2024-05-03T21:20:47.576 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:20:47 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T21:20:47.852 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:47.849+0000 7fc7f0212700 1 -- 172.21.15.67:0/3322677715 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc7e802e510 con 0x7fc7e8108410 2024-05-03T21:20:47.852 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:47.850+0000 7fc7deffd700 1 -- 172.21.15.67:0/3322677715 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+11868 (secure 0 0 0) 0x7fc7e0010040 con 0x7fc7e8108410 2024-05-03T21:20:47.853 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-05-03T21:20:47.853 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14541,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6835/4224826546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":4224826546},{"type":"v1","addr":"172.21.15.67:6835","nonce":4224826546}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14553,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6837/2485231621","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6836","nonce":2485231621},{"type":"v1","addr":"172.21.15.67:6837","nonce":2485231621}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14598,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6839/2106078262","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6838","nonce":2106078262},{"type":"v1","addr":"172.21.15.67:6839","nonce":2106078262}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14610,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6841/1559196215","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6840","nonce":1559196215},{"type":"v1","addr":"172.21.15.67:6841","nonce":1559196215}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24376,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.138:6833/3028022602","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6832","nonce":3028022602},{"type":"v1","addr":"172.21.15.138:6833","nonce":3028022602}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24400,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.138:6835/3342864697","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6834","nonce":3342864697},{"type":"v1","addr":"172.21.15.138:6835","nonce":3342864697}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7}],"filesystems":[{"mdsmap":{"epoch":25,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-05-03T21:18:44.665825+0000","modified":"2024-05-03T21:19:03.775546+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24473,"mds_1":24449,"mds_2":24425},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24406":{"gid":24406,"name":"c","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.138:6837/2203273249","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6836","nonce":2203273249},{"type":"v1","addr":"172.21.15.138:6837","nonce":2203273249}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24407":{"gid":24407,"name":"k","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.82:6833/1127287628","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6832","nonce":1127287628},{"type":"v1","addr":"172.21.15.82:6833","nonce":1127287628}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24418":{"gid":24418,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.138:6839/334382443","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6838","nonce":334382443},{"type":"v1","addr":"172.21.15.138:6839","nonce":334382443}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24425":{"gid":24425,"name":"e","rank":2,"incarnation":21,"state":"up:active","state_seq":14,"addr":"172.21.15.82:6835/280518761","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6834","nonce":280518761},{"type":"v1","addr":"172.21.15.82:6835","nonce":280518761}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24449":{"gid":24449,"name":"h","rank":1,"incarnation":19,"state":"up:active","state_seq":11,"addr":"172.21.15.82:6837/1275987852","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6836","nonce":1275987852},{"type":"v1","addr":"172.21.15.82:6837","nonce":1275987852}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24473":{"gid":24473,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":5,"addr":"172.21.15.82:6839/3644499578","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6838","nonce":3644499578},{"type":"v1","addr":"172.21.15.82:6839","nonce":3644499578}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24473,"qdb_cluster":[24449,24473,24425]},"id":1}]} 2024-05-03T21:20:47.854 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:47.852+0000 7fc7f0212700 1 -- 172.21.15.67:0/3322677715 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7fc7d40b2910 msgr2=0x7fc7d40b4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:20:47.854 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:47.852+0000 7fc7f0212700 1 --2- 172.21.15.67:0/3322677715 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7fc7d40b2910 0x7fc7d40b4dd0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fc7e400b2e0 tx=0x7fc7e4009690 comp rx=0 tx=0).stop 2024-05-03T21:20:47.855 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:47.852+0000 7fc7f0212700 1 -- 172.21.15.67:0/3322677715 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc7e8108410 msgr2=0x7fc7e81a5c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:20:47.855 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:47.852+0000 7fc7f0212700 1 --2- 172.21.15.67:0/3322677715 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc7e8108410 0x7fc7e81a5c40 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7fc7e000ef30 tx=0x7fc7e000c3a0 comp rx=0 tx=0).stop 2024-05-03T21:20:47.855 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:47.852+0000 7fc7f0212700 1 -- 172.21.15.67:0/3322677715 shutdown_connections 2024-05-03T21:20:47.855 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:47.852+0000 7fc7f0212700 1 --2- 172.21.15.67:0/3322677715 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7fc7d40b2910 0x7fc7d40b4dd0 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:20:47.855 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:47.852+0000 7fc7f0212700 1 --2- 172.21.15.67:0/3322677715 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc7e8108410 0x7fc7e81a5c40 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:20:47.855 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:47.852+0000 7fc7f0212700 1 --2- 172.21.15.67:0/3322677715 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fc7e81023a0 0x7fc7e819abc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:20:47.855 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:47.852+0000 7fc7f0212700 1 --2- 172.21.15.67:0/3322677715 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fc7e8102d60 0x7fc7e819b100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:20:47.855 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:47.852+0000 7fc7f0212700 1 -- 172.21.15.67:0/3322677715 >> 172.21.15.67:0/3322677715 conn(0x7fc7e8079350 msgr2=0x7fc7e8100e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T21:20:47.855 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:47.852+0000 7fc7f0212700 1 -- 172.21.15.67:0/3322677715 shutdown_connections 2024-05-03T21:20:47.855 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:47.852+0000 7fc7f0212700 1 -- 172.21.15.67:0/3322677715 wait complete. 2024-05-03T21:20:47.856 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 25 2024-05-03T21:20:48.245 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:20:48 smithi082 ceph-mon[32933]: from='client.? 172.21.15.67:0/3322677715' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-03T21:20:48.471 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:20:48 smithi067 ceph-mon[29730]: from='client.? 172.21.15.67:0/3322677715' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-03T21:20:48.574 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:20:48 smithi138 ceph-mon[33110]: from='client.? 172.21.15.67:0/3322677715' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-03T21:20:49.083 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9a047fe7599a549042d06a3fb5b6813a203eabea shell --fsid 713bc57a-0991-11ef-bc97-c7b262605968 -- ceph osd dump --format=json 2024-05-03T21:20:49.357 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/713bc57a-0991-11ef-bc97-c7b262605968/mon.a/config 2024-05-03T21:20:49.471 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:20:49 smithi067 ceph-mon[29730]: pgmap v351: 97 pgs: 97 active+clean; 632 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 85 B/s wr, 6 op/s 2024-05-03T21:20:49.471 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:20:49 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:20:49.471 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:20:49 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:20:49.471 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:20:49 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:20:49.471 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:20:49 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:20:49.495 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:20:49 smithi082 ceph-mon[32933]: pgmap v351: 97 pgs: 97 active+clean; 632 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 85 B/s wr, 6 op/s 2024-05-03T21:20:49.495 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:20:49 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:20:49.495 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:20:49 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:20:49.496 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:20:49 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:20:49.496 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:20:49 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:20:49.574 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:20:49 smithi138 ceph-mon[33110]: pgmap v351: 97 pgs: 97 active+clean; 632 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 85 B/s wr, 6 op/s 2024-05-03T21:20:49.574 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:20:49 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:20:49.574 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:20:49 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:20:49.574 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:20:49 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:20:49.575 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:20:49 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:20:51.038 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:51.033+0000 7f4ca586d700 1 -- 172.21.15.67:0/3981892472 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4ca010ff40 msgr2=0x7f4ca0112330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:20:51.038 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:51.033+0000 7f4ca586d700 1 --2- 172.21.15.67:0/3981892472 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4ca010ff40 0x7f4ca0112330 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f4c9800b0a0 tx=0x7f4c9800b3b0 comp rx=0 tx=0).stop 2024-05-03T21:20:51.038 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:51.033+0000 7f4ca586d700 1 -- 172.21.15.67:0/3981892472 shutdown_connections 2024-05-03T21:20:51.039 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:51.033+0000 7f4ca586d700 1 --2- 172.21.15.67:0/3981892472 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4ca010ff40 0x7f4ca0112330 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:20:51.039 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:51.033+0000 7f4ca586d700 1 --2- 172.21.15.67:0/3981892472 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f4ca006ba40 0x7f4ca006be20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:20:51.039 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:51.033+0000 7f4ca586d700 1 --2- 172.21.15.67:0/3981892472 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f4ca006c360 0x7f4ca010fa00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:20:51.039 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:51.033+0000 7f4ca586d700 1 -- 172.21.15.67:0/3981892472 >> 172.21.15.67:0/3981892472 conn(0x7f4ca0079310 msgr2=0x7f4ca0079720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T21:20:51.039 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:51.033+0000 7f4ca586d700 1 -- 172.21.15.67:0/3981892472 shutdown_connections 2024-05-03T21:20:51.039 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:51.033+0000 7f4ca586d700 1 -- 172.21.15.67:0/3981892472 wait complete. 2024-05-03T21:20:51.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:51.034+0000 7f4ca586d700 1 Processor -- start 2024-05-03T21:20:51.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:51.034+0000 7f4ca586d700 1 -- start start 2024-05-03T21:20:51.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:51.035+0000 7f4ca586d700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4ca006ba40 0x7f4ca019c0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:20:51.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:51.035+0000 7f4ca586d700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f4ca006c360 0x7f4ca019c620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:20:51.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:51.035+0000 7f4ca586d700 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f4ca010ff40 0x7f4ca01a0960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:20:51.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:51.035+0000 7f4ca586d700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f4ca0115bb0 con 0x7f4ca006ba40 2024-05-03T21:20:51.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:51.035+0000 7f4ca586d700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f4ca0115a30 con 0x7f4ca006c360 2024-05-03T21:20:51.041 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:51.035+0000 7f4ca586d700 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f4ca0115d30 con 0x7f4ca010ff40 2024-05-03T21:20:51.041 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:51.035+0000 7f4c9effd700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4ca006ba40 0x7f4ca019c0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:20:51.041 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:51.035+0000 7f4c9effd700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4ca006ba40 0x7f4ca019c0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:59708/0 (socket says 172.21.15.67:59708) 2024-05-03T21:20:51.041 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:51.035+0000 7f4c9effd700 1 -- 172.21.15.67:0/347574890 learned_addr learned my addr 172.21.15.67:0/347574890 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-03T21:20:51.041 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:51.035+0000 7f4c9e7fc700 1 --2- 172.21.15.67:0/347574890 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f4ca006c360 0x7f4ca019c620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:20:51.041 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:51.035+0000 7f4c9f7fe700 1 --2- 172.21.15.67:0/347574890 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f4ca010ff40 0x7f4ca01a0960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:20:51.042 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:51.035+0000 7f4c9effd700 1 -- 172.21.15.67:0/347574890 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f4ca010ff40 msgr2=0x7f4ca01a0960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:20:51.042 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:51.035+0000 7f4c9effd700 1 --2- 172.21.15.67:0/347574890 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f4ca010ff40 0x7f4ca01a0960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:20:51.042 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:51.035+0000 7f4c9effd700 1 -- 172.21.15.67:0/347574890 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f4ca006c360 msgr2=0x7f4ca019c620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:20:51.042 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:51.035+0000 7f4c9effd700 1 --2- 172.21.15.67:0/347574890 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f4ca006c360 0x7f4ca019c620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:20:51.042 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:51.035+0000 7f4c9effd700 1 -- 172.21.15.67:0/347574890 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4ca01a10a0 con 0x7f4ca006ba40 2024-05-03T21:20:51.042 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:51.035+0000 7f4c9f7fe700 1 --2- 172.21.15.67:0/347574890 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f4ca010ff40 0x7f4ca01a0960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-03T21:20:51.042 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:51.035+0000 7f4c9effd700 1 --2- 172.21.15.67:0/347574890 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4ca006ba40 0x7f4ca019c0e0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f4c9000b530 tx=0x7f4c9000b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T21:20:51.042 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:51.035+0000 7f4ca486b700 1 -- 172.21.15.67:0/347574890 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c90015020 con 0x7f4ca006ba40 2024-05-03T21:20:51.043 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:51.036+0000 7f4ca486b700 1 -- 172.21.15.67:0/347574890 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(28 keys) ==== 1170+0+0 (secure 0 0 0) 0x7f4c90011d30 con 0x7f4ca006ba40 2024-05-03T21:20:51.043 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:51.036+0000 7f4ca586d700 1 -- 172.21.15.67:0/347574890 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4ca01a1390 con 0x7f4ca006ba40 2024-05-03T21:20:51.043 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:51.036+0000 7f4ca486b700 1 -- 172.21.15.67:0/347574890 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c9000f2e0 con 0x7f4ca006ba40 2024-05-03T21:20:51.043 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:51.036+0000 7f4ca586d700 1 -- 172.21.15.67:0/347574890 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4ca01a8c10 con 0x7f4ca006ba40 2024-05-03T21:20:51.043 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:51.037+0000 7f4ca586d700 1 -- 172.21.15.67:0/347574890 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4ca006a190 con 0x7f4ca006ba40 2024-05-03T21:20:51.044 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:51.039+0000 7f4ca486b700 1 -- 172.21.15.67:0/347574890 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 17) ==== 149703+0+0 (secure 0 0 0) 0x7f4c90018020 con 0x7f4ca006ba40 2024-05-03T21:20:51.044 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:51.040+0000 7f4ca486b700 1 --2- 172.21.15.67:0/347574890 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f4c8c0b2910 0x7f4c8c0b4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:20:51.044 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:51.040+0000 7f4c9e7fc700 1 --2- 172.21.15.67:0/347574890 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f4c8c0b2910 0x7f4c8c0b4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:20:51.044 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:51.040+0000 7f4ca486b700 1 -- 172.21.15.67:0/347574890 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 8211+0+0 (secure 0 0 0) 0x7f4c900e2090 con 0x7f4ca006ba40 2024-05-03T21:20:51.044 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:51.040+0000 7f4c9e7fc700 1 --2- 172.21.15.67:0/347574890 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f4c8c0b2910 0x7f4c8c0b4dd0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f4c88006fd0 tx=0x7f4c88008040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T21:20:51.044 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:51.043+0000 7f4ca486b700 1 -- 172.21.15.67:0/347574890 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191839 (secure 0 0 0) 0x7f4c9001f020 con 0x7f4ca006ba40 2024-05-03T21:20:51.257 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:51.256+0000 7f4ca586d700 1 -- 172.21.15.67:0/347574890 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f4ca019d3f0 con 0x7f4ca006ba40 2024-05-03T21:20:51.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:51.259+0000 7f4ca486b700 1 -- 172.21.15.67:0/347574890 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v84) ==== 74+0+22637 (secure 0 0 0) 0x7f4c900a9d70 con 0x7f4ca006ba40 2024-05-03T21:20:51.260 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-05-03T21:20:51.260 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":84,"fsid":"713bc57a-0991-11ef-bc97-c7b262605968","created":"2024-05-03T21:10:34.070178+0000","modified":"2024-05-03T21:19:34.967839+0000","last_up_change":"2024-05-03T21:17:39.993919+0000","last_in_change":"2024-05-03T21:17:19.978727+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-03T21:13:59.810446+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-05-03T21:18:37.953919+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"82","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-05-03T21:18:41.571707+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"82","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7939f31c-6fb7-42bd-af96-88a448d11bd2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":3692483807},{"type":"v1","addr":"172.21.15.67:6803","nonce":3692483807}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":3692483807},{"type":"v1","addr":"172.21.15.67:6805","nonce":3692483807}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":3692483807},{"type":"v1","addr":"172.21.15.67:6809","nonce":3692483807}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":3692483807},{"type":"v1","addr":"172.21.15.67:6807","nonce":3692483807}]},"public_addr":"172.21.15.67:6803/3692483807","cluster_addr":"172.21.15.67:6805/3692483807","heartbeat_back_addr":"172.21.15.67:6809/3692483807","heartbeat_front_addr":"172.21.15.67:6807/3692483807","state":["exists","up"]},{"osd":1,"uuid":"71858193-05a6-4c2c-9bb7-bc3d811d634e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6800","nonce":1139376397},{"type":"v1","addr":"172.21.15.82:6801","nonce":1139376397}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6802","nonce":1139376397},{"type":"v1","addr":"172.21.15.82:6803","nonce":1139376397}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6806","nonce":1139376397},{"type":"v1","addr":"172.21.15.82:6807","nonce":1139376397}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6804","nonce":1139376397},{"type":"v1","addr":"172.21.15.82:6805","nonce":1139376397}]},"public_addr":"172.21.15.82:6801/1139376397","cluster_addr":"172.21.15.82:6803/1139376397","heartbeat_back_addr":"172.21.15.82:6807/1139376397","heartbeat_front_addr":"172.21.15.82:6805/1139376397","state":["exists","up"]},{"osd":2,"uuid":"f9ece78d-6a19-4d46-9022-6ed184074745","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6800","nonce":2774330123},{"type":"v1","addr":"172.21.15.138:6801","nonce":2774330123}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6802","nonce":2774330123},{"type":"v1","addr":"172.21.15.138:6803","nonce":2774330123}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6806","nonce":2774330123},{"type":"v1","addr":"172.21.15.138:6807","nonce":2774330123}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6804","nonce":2774330123},{"type":"v1","addr":"172.21.15.138:6805","nonce":2774330123}]},"public_addr":"172.21.15.138:6801/2774330123","cluster_addr":"172.21.15.138:6803/2774330123","heartbeat_back_addr":"172.21.15.138:6807/2774330123","heartbeat_front_addr":"172.21.15.138:6805/2774330123","state":["exists","up"]},{"osd":3,"uuid":"3616df71-e7b3-4687-aaba-37b0b7766a3c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":27,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":3915934652},{"type":"v1","addr":"172.21.15.67:6811","nonce":3915934652}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":3915934652},{"type":"v1","addr":"172.21.15.67:6813","nonce":3915934652}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":3915934652},{"type":"v1","addr":"172.21.15.67:6817","nonce":3915934652}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":3915934652},{"type":"v1","addr":"172.21.15.67:6815","nonce":3915934652}]},"public_addr":"172.21.15.67:6811/3915934652","cluster_addr":"172.21.15.67:6813/3915934652","heartbeat_back_addr":"172.21.15.67:6817/3915934652","heartbeat_front_addr":"172.21.15.67:6815/3915934652","state":["exists","up"]},{"osd":4,"uuid":"c3fd0ffd-e23f-4279-b679-9d95a144cf32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6808","nonce":4031818951},{"type":"v1","addr":"172.21.15.82:6809","nonce":4031818951}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6810","nonce":4031818951},{"type":"v1","addr":"172.21.15.82:6811","nonce":4031818951}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6814","nonce":4031818951},{"type":"v1","addr":"172.21.15.82:6815","nonce":4031818951}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6812","nonce":4031818951},{"type":"v1","addr":"172.21.15.82:6813","nonce":4031818951}]},"public_addr":"172.21.15.82:6809/4031818951","cluster_addr":"172.21.15.82:6811/4031818951","heartbeat_back_addr":"172.21.15.82:6815/4031818951","heartbeat_front_addr":"172.21.15.82:6813/4031818951","state":["exists","up"]},{"osd":5,"uuid":"540ef0d9-a6dc-43c6-b09e-e87340ba24cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6808","nonce":2521889499},{"type":"v1","addr":"172.21.15.138:6809","nonce":2521889499}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6810","nonce":2521889499},{"type":"v1","addr":"172.21.15.138:6811","nonce":2521889499}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6814","nonce":2521889499},{"type":"v1","addr":"172.21.15.138:6815","nonce":2521889499}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6812","nonce":2521889499},{"type":"v1","addr":"172.21.15.138:6813","nonce":2521889499}]},"public_addr":"172.21.15.138:6809/2521889499","cluster_addr":"172.21.15.138:6811/2521889499","heartbeat_back_addr":"172.21.15.138:6815/2521889499","heartbeat_front_addr":"172.21.15.138:6813/2521889499","state":["exists","up"]},{"osd":6,"uuid":"49a89fd0-958a-4be9-a8db-6022f946153e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6818","nonce":3269682731},{"type":"v1","addr":"172.21.15.67:6819","nonce":3269682731}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":3269682731},{"type":"v1","addr":"172.21.15.67:6821","nonce":3269682731}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":3269682731},{"type":"v1","addr":"172.21.15.67:6825","nonce":3269682731}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":3269682731},{"type":"v1","addr":"172.21.15.67:6823","nonce":3269682731}]},"public_addr":"172.21.15.67:6819/3269682731","cluster_addr":"172.21.15.67:6821/3269682731","heartbeat_back_addr":"172.21.15.67:6825/3269682731","heartbeat_front_addr":"172.21.15.67:6823/3269682731","state":["exists","up"]},{"osd":7,"uuid":"8fa4cd64-ca6b-48ae-8db0-d369ce6102ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":50,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6816","nonce":3008589976},{"type":"v1","addr":"172.21.15.82:6817","nonce":3008589976}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6818","nonce":3008589976},{"type":"v1","addr":"172.21.15.82:6819","nonce":3008589976}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6822","nonce":3008589976},{"type":"v1","addr":"172.21.15.82:6823","nonce":3008589976}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6820","nonce":3008589976},{"type":"v1","addr":"172.21.15.82:6821","nonce":3008589976}]},"public_addr":"172.21.15.82:6817/3008589976","cluster_addr":"172.21.15.82:6819/3008589976","heartbeat_back_addr":"172.21.15.82:6823/3008589976","heartbeat_front_addr":"172.21.15.82:6821/3008589976","state":["exists","up"]},{"osd":8,"uuid":"fae39087-6c38-4771-a2b9-3f4fecb08edd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":57,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6816","nonce":2223920070},{"type":"v1","addr":"172.21.15.138:6817","nonce":2223920070}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6818","nonce":2223920070},{"type":"v1","addr":"172.21.15.138:6819","nonce":2223920070}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6822","nonce":2223920070},{"type":"v1","addr":"172.21.15.138:6823","nonce":2223920070}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6820","nonce":2223920070},{"type":"v1","addr":"172.21.15.138:6821","nonce":2223920070}]},"public_addr":"172.21.15.138:6817/2223920070","cluster_addr":"172.21.15.138:6819/2223920070","heartbeat_back_addr":"172.21.15.138:6823/2223920070","heartbeat_front_addr":"172.21.15.138:6821/2223920070","state":["exists","up"]},{"osd":9,"uuid":"3d387dbf-0163-41b7-98af-4fc45ed73ad4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":63,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":3939417129},{"type":"v1","addr":"172.21.15.67:6827","nonce":3939417129}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":3939417129},{"type":"v1","addr":"172.21.15.67:6829","nonce":3939417129}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":3939417129},{"type":"v1","addr":"172.21.15.67:6833","nonce":3939417129}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":3939417129},{"type":"v1","addr":"172.21.15.67:6831","nonce":3939417129}]},"public_addr":"172.21.15.67:6827/3939417129","cluster_addr":"172.21.15.67:6829/3939417129","heartbeat_back_addr":"172.21.15.67:6833/3939417129","heartbeat_front_addr":"172.21.15.67:6831/3939417129","state":["exists","up"]},{"osd":10,"uuid":"db0305a0-a83b-474b-837a-fbc239ac1738","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6824","nonce":2505507029},{"type":"v1","addr":"172.21.15.82:6825","nonce":2505507029}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6826","nonce":2505507029},{"type":"v1","addr":"172.21.15.82:6827","nonce":2505507029}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6830","nonce":2505507029},{"type":"v1","addr":"172.21.15.82:6831","nonce":2505507029}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6828","nonce":2505507029},{"type":"v1","addr":"172.21.15.82:6829","nonce":2505507029}]},"public_addr":"172.21.15.82:6825/2505507029","cluster_addr":"172.21.15.82:6827/2505507029","heartbeat_back_addr":"172.21.15.82:6831/2505507029","heartbeat_front_addr":"172.21.15.82:6829/2505507029","state":["exists","up"]},{"osd":11,"uuid":"479c2747-f28e-4704-87e4-d0c31782e2de","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":74,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6824","nonce":2528685018},{"type":"v1","addr":"172.21.15.138:6825","nonce":2528685018}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6826","nonce":2528685018},{"type":"v1","addr":"172.21.15.138:6827","nonce":2528685018}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6830","nonce":2528685018},{"type":"v1","addr":"172.21.15.138:6831","nonce":2528685018}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6828","nonce":2528685018},{"type":"v1","addr":"172.21.15.138:6829","nonce":2528685018}]},"public_addr":"172.21.15.138:6825/2528685018","cluster_addr":"172.21.15.138:6827/2528685018","heartbeat_back_addr":"172.21.15.138:6831/2528685018","heartbeat_front_addr":"172.21.15.138:6829/2528685018","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-03T21:13:07.768773+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-03T21:13:31.870232+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-03T21:13:55.394206+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-03T21:14:20.540281+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-03T21:14:44.318523+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-03T21:15:08.490829+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-03T21:15:34.144734+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-03T21:15:58.717240+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-03T21:16:22.743013+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-03T21:16:49.099588+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-03T21:17:13.321126+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-03T21:17:38.001978+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.0","mappings":[{"from":8,"to":4}]},{"pgid":"2.a","mappings":[{"from":2,"to":11}]},{"pgid":"2.b","mappings":[{"from":2,"to":4}]},{"pgid":"2.10","mappings":[{"from":8,"to":11}]},{"pgid":"3.1","mappings":[{"from":4,"to":9}]},{"pgid":"3.1b","mappings":[{"from":5,"to":9}]},{"pgid":"3.21","mappings":[{"from":4,"to":8}]},{"pgid":"3.3c","mappings":[{"from":4,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.67:0/1239643865":"2024-05-04T21:11:33.721416+0000","172.21.15.67:6801/2576113038":"2024-05-04T21:11:33.721416+0000","172.21.15.67:6800/2576113038":"2024-05-04T21:11:33.721416+0000","172.21.15.67:6800/1698171509":"2024-05-04T21:11:07.333651+0000","172.21.15.67:0/2217893923":"2024-05-04T21:11:33.721416+0000","172.21.15.67:0/20864811":"2024-05-04T21:11:07.333651+0000","172.21.15.67:6801/1698171509":"2024-05-04T21:11:07.333651+0000","172.21.15.67:0/773376553":"2024-05-04T21:11:33.721416+0000","172.21.15.67:0/847549114":"2024-05-04T21:11:07.333651+0000","172.21.15.67:0/1291778035":"2024-05-04T21:11:07.333651+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-03T21:20:51.262 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:51.261+0000 7f4ca586d700 1 -- 172.21.15.67:0/347574890 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f4c8c0b2910 msgr2=0x7f4c8c0b4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:20:51.262 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:51.261+0000 7f4ca586d700 1 --2- 172.21.15.67:0/347574890 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f4c8c0b2910 0x7f4c8c0b4dd0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f4c88006fd0 tx=0x7f4c88008040 comp rx=0 tx=0).stop 2024-05-03T21:20:51.262 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:51.261+0000 7f4ca586d700 1 -- 172.21.15.67:0/347574890 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4ca006ba40 msgr2=0x7f4ca019c0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:20:51.262 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:51.261+0000 7f4ca586d700 1 --2- 172.21.15.67:0/347574890 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4ca006ba40 0x7f4ca019c0e0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f4c9000b530 tx=0x7f4c9000b8f0 comp rx=0 tx=0).stop 2024-05-03T21:20:51.262 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:51.261+0000 7f4ca586d700 1 -- 172.21.15.67:0/347574890 shutdown_connections 2024-05-03T21:20:51.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:51.261+0000 7f4ca586d700 1 --2- 172.21.15.67:0/347574890 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f4c8c0b2910 0x7f4c8c0b4dd0 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:20:51.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:51.261+0000 7f4ca586d700 1 --2- 172.21.15.67:0/347574890 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4ca006ba40 0x7f4ca019c0e0 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:20:51.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:51.261+0000 7f4ca586d700 1 --2- 172.21.15.67:0/347574890 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f4ca010ff40 0x7f4ca01a0960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:20:51.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:51.261+0000 7f4ca586d700 1 --2- 172.21.15.67:0/347574890 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f4ca006c360 0x7f4ca019c620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:20:51.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:51.261+0000 7f4ca586d700 1 -- 172.21.15.67:0/347574890 >> 172.21.15.67:0/347574890 conn(0x7f4ca0079310 msgr2=0x7f4ca0111390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T21:20:51.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:51.261+0000 7f4ca586d700 1 -- 172.21.15.67:0/347574890 shutdown_connections 2024-05-03T21:20:51.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:51.261+0000 7f4ca586d700 1 -- 172.21.15.67:0/347574890 wait complete. 2024-05-03T21:20:51.297 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:20:51 smithi067 ceph-mon[29730]: pgmap v352: 97 pgs: 97 active+clean; 632 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T21:20:51.298 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:20:51 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:20:51.298 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:20:51 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:20:51.495 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:20:51 smithi082 ceph-mon[32933]: pgmap v352: 97 pgs: 97 active+clean; 632 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T21:20:51.495 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:20:51 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:20:51.496 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:20:51 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:20:51.574 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:20:51 smithi138 ceph-mon[33110]: pgmap v352: 97 pgs: 97 active+clean; 632 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T21:20:51.579 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:20:51 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:20:51.580 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:20:51 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:20:51.879 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9a047fe7599a549042d06a3fb5b6813a203eabea shell --fsid 713bc57a-0991-11ef-bc97-c7b262605968 -- ceph fs dump --format=json 2024-05-03T21:20:52.148 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/713bc57a-0991-11ef-bc97-c7b262605968/mon.a/config 2024-05-03T21:20:52.471 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:20:52 smithi067 ceph-mon[29730]: from='client.? 172.21.15.67:0/347574890' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-03T21:20:52.472 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:20:52 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T21:20:52.472 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:20:52 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T21:20:52.472 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:20:52 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:20:52.495 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:20:52 smithi082 ceph-mon[32933]: from='client.? 172.21.15.67:0/347574890' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-03T21:20:52.496 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:20:52 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T21:20:52.496 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:20:52 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T21:20:52.496 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:20:52 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:20:52.574 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:20:52 smithi138 ceph-mon[33110]: from='client.? 172.21.15.67:0/347574890' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-03T21:20:52.574 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:20:52 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T21:20:52.575 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:20:52 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T21:20:52.575 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:20:52 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:20:53.221 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:20:53 smithi067 ceph-mon[29730]: pgmap v353: 97 pgs: 97 active+clean; 632 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:20:53.496 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:20:53 smithi082 ceph-mon[32933]: pgmap v353: 97 pgs: 97 active+clean; 632 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:20:53.574 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:20:53 smithi138 ceph-mon[33110]: pgmap v353: 97 pgs: 97 active+clean; 632 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:20:53.792 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:53.786+0000 7f00df3d2700 1 -- 172.21.15.67:0/3248545417 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f00d8108850 msgr2=0x7f00d81103d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:20:53.792 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:53.786+0000 7f00df3d2700 1 --2- 172.21.15.67:0/3248545417 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f00d8108850 0x7f00d81103d0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f00d000b0d0 tx=0x7f00d000b3e0 comp rx=0 tx=0).stop 2024-05-03T21:20:53.792 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:53.786+0000 7f00df3d2700 1 -- 172.21.15.67:0/3248545417 shutdown_connections 2024-05-03T21:20:53.792 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:53.786+0000 7f00df3d2700 1 --2- 172.21.15.67:0/3248545417 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f00d8108850 0x7f00d81103d0 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:20:53.792 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:53.786+0000 7f00df3d2700 1 --2- 172.21.15.67:0/3248545417 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f00d8103750 0x7f00d8103b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:20:53.792 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:53.786+0000 7f00df3d2700 1 --2- 172.21.15.67:0/3248545417 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f00d8104100 0x7f00d8108120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:20:53.792 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:53.786+0000 7f00df3d2700 1 -- 172.21.15.67:0/3248545417 >> 172.21.15.67:0/3248545417 conn(0x7f00d807a600 msgr2=0x7f00d807aa10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T21:20:53.792 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:53.787+0000 7f00df3d2700 1 -- 172.21.15.67:0/3248545417 shutdown_connections 2024-05-03T21:20:53.792 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:53.787+0000 7f00df3d2700 1 -- 172.21.15.67:0/3248545417 wait complete. 2024-05-03T21:20:53.792 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:53.787+0000 7f00df3d2700 1 Processor -- start 2024-05-03T21:20:53.792 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:53.787+0000 7f00df3d2700 1 -- start start 2024-05-03T21:20:53.793 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:53.788+0000 7f00df3d2700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f00d8103750 0x7f00d819f7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:20:53.793 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:53.788+0000 7f00df3d2700 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f00d8104100 0x7f00d819fcf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:20:53.793 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:53.788+0000 7f00df3d2700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f00d8108850 0x7f00d8199970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:20:53.793 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:53.788+0000 7f00df3d2700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f00d81138b0 con 0x7f00d8103750 2024-05-03T21:20:53.793 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:53.788+0000 7f00df3d2700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f00d8113730 con 0x7f00d8108850 2024-05-03T21:20:53.793 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:53.788+0000 7f00df3d2700 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f00d8113a30 con 0x7f00d8104100 2024-05-03T21:20:53.793 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:53.788+0000 7f00dd16e700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f00d8103750 0x7f00d819f7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:20:53.793 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:53.788+0000 7f00dd16e700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f00d8103750 0x7f00d819f7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:59724/0 (socket says 172.21.15.67:59724) 2024-05-03T21:20:53.793 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:53.788+0000 7f00dd16e700 1 -- 172.21.15.67:0/404255889 learned_addr learned my addr 172.21.15.67:0/404255889 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-03T21:20:53.793 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:53.788+0000 7f00dd96f700 1 --2- 172.21.15.67:0/404255889 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f00d8108850 0x7f00d8199970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:20:53.793 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:53.788+0000 7f00dc96d700 1 --2- 172.21.15.67:0/404255889 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f00d8104100 0x7f00d819fcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:20:53.793 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:53.788+0000 7f00dd16e700 1 -- 172.21.15.67:0/404255889 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f00d8104100 msgr2=0x7f00d819fcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:20:53.793 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:53.788+0000 7f00dd16e700 1 --2- 172.21.15.67:0/404255889 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f00d8104100 0x7f00d819fcf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:20:53.794 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:53.788+0000 7f00dd16e700 1 -- 172.21.15.67:0/404255889 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f00d8108850 msgr2=0x7f00d8199970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:20:53.794 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:53.788+0000 7f00dd16e700 1 --2- 172.21.15.67:0/404255889 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f00d8108850 0x7f00d8199970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:20:53.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:53.788+0000 7f00dd16e700 1 -- 172.21.15.67:0/404255889 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f00d819a0d0 con 0x7f00d8103750 2024-05-03T21:20:53.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:53.788+0000 7f00dc96d700 1 --2- 172.21.15.67:0/404255889 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f00d8104100 0x7f00d819fcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-03T21:20:53.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:53.788+0000 7f00dd16e700 1 --2- 172.21.15.67:0/404255889 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f00d8103750 0x7f00d819f7b0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f00c800c9b0 tx=0x7f00c800ccc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T21:20:53.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:53.788+0000 7f00ce7fc700 1 -- 172.21.15.67:0/404255889 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00c8007b00 con 0x7f00d8103750 2024-05-03T21:20:53.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:53.788+0000 7f00df3d2700 1 -- 172.21.15.67:0/404255889 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f00d819a3c0 con 0x7f00d8103750 2024-05-03T21:20:53.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:53.788+0000 7f00ce7fc700 1 -- 172.21.15.67:0/404255889 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(28 keys) ==== 1170+0+0 (secure 0 0 0) 0x7f00c8007ca0 con 0x7f00d8103750 2024-05-03T21:20:53.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:53.789+0000 7f00ce7fc700 1 -- 172.21.15.67:0/404255889 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00c8018780 con 0x7f00d8103750 2024-05-03T21:20:53.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:53.789+0000 7f00df3d2700 1 -- 172.21.15.67:0/404255889 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f00d81a6680 con 0x7f00d8103750 2024-05-03T21:20:53.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:53.789+0000 7f00df3d2700 1 -- 172.21.15.67:0/404255889 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f00d806a190 con 0x7f00d8103750 2024-05-03T21:20:53.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:53.791+0000 7f00ce7fc700 1 -- 172.21.15.67:0/404255889 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 17) ==== 149703+0+0 (secure 0 0 0) 0x7f00c800f450 con 0x7f00d8103750 2024-05-03T21:20:53.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:53.792+0000 7f00ce7fc700 1 --2- 172.21.15.67:0/404255889 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f00c40b2910 0x7f00c40b4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:20:53.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:53.792+0000 7f00dc96d700 1 --2- 172.21.15.67:0/404255889 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f00c40b2910 0x7f00c40b4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:20:53.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:53.792+0000 7f00ce7fc700 1 -- 172.21.15.67:0/404255889 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 8211+0+0 (secure 0 0 0) 0x7f00c80e2600 con 0x7f00d8103750 2024-05-03T21:20:53.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:53.793+0000 7f00dc96d700 1 --2- 172.21.15.67:0/404255889 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f00c40b2910 0x7f00c40b4dd0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f00d4005fd0 tx=0x7f00d4005ee0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T21:20:53.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:53.795+0000 7f00ce7fc700 1 -- 172.21.15.67:0/404255889 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191839 (secure 0 0 0) 0x7f00c8014030 con 0x7f00d8103750 2024-05-03T21:20:54.058 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:54.055+0000 7f00df3d2700 1 -- 172.21.15.67:0/404255889 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f00d819aea0 con 0x7f00d8103750 2024-05-03T21:20:54.058 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:54.056+0000 7f00ce7fc700 1 -- 172.21.15.67:0/404255889 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+11868 (secure 0 0 0) 0x7f00c80aa260 con 0x7f00d8103750 2024-05-03T21:20:54.059 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-05-03T21:20:54.059 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14541,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6835/4224826546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":4224826546},{"type":"v1","addr":"172.21.15.67:6835","nonce":4224826546}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14553,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6837/2485231621","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6836","nonce":2485231621},{"type":"v1","addr":"172.21.15.67:6837","nonce":2485231621}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14598,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6839/2106078262","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6838","nonce":2106078262},{"type":"v1","addr":"172.21.15.67:6839","nonce":2106078262}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14610,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6841/1559196215","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6840","nonce":1559196215},{"type":"v1","addr":"172.21.15.67:6841","nonce":1559196215}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24376,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.138:6833/3028022602","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6832","nonce":3028022602},{"type":"v1","addr":"172.21.15.138:6833","nonce":3028022602}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24400,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.138:6835/3342864697","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6834","nonce":3342864697},{"type":"v1","addr":"172.21.15.138:6835","nonce":3342864697}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7}],"filesystems":[{"mdsmap":{"epoch":25,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-05-03T21:18:44.665825+0000","modified":"2024-05-03T21:19:03.775546+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24473,"mds_1":24449,"mds_2":24425},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24406":{"gid":24406,"name":"c","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.138:6837/2203273249","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6836","nonce":2203273249},{"type":"v1","addr":"172.21.15.138:6837","nonce":2203273249}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24407":{"gid":24407,"name":"k","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.82:6833/1127287628","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6832","nonce":1127287628},{"type":"v1","addr":"172.21.15.82:6833","nonce":1127287628}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24418":{"gid":24418,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.138:6839/334382443","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6838","nonce":334382443},{"type":"v1","addr":"172.21.15.138:6839","nonce":334382443}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24425":{"gid":24425,"name":"e","rank":2,"incarnation":21,"state":"up:active","state_seq":14,"addr":"172.21.15.82:6835/280518761","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6834","nonce":280518761},{"type":"v1","addr":"172.21.15.82:6835","nonce":280518761}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24449":{"gid":24449,"name":"h","rank":1,"incarnation":19,"state":"up:active","state_seq":11,"addr":"172.21.15.82:6837/1275987852","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6836","nonce":1275987852},{"type":"v1","addr":"172.21.15.82:6837","nonce":1275987852}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24473":{"gid":24473,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":5,"addr":"172.21.15.82:6839/3644499578","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6838","nonce":3644499578},{"type":"v1","addr":"172.21.15.82:6839","nonce":3644499578}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24473,"qdb_cluster":[24449,24473,24425]},"id":1}]} 2024-05-03T21:20:54.061 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:54.058+0000 7f00df3d2700 1 -- 172.21.15.67:0/404255889 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f00c40b2910 msgr2=0x7f00c40b4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:20:54.061 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:54.058+0000 7f00df3d2700 1 --2- 172.21.15.67:0/404255889 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f00c40b2910 0x7f00c40b4dd0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f00d4005fd0 tx=0x7f00d4005ee0 comp rx=0 tx=0).stop 2024-05-03T21:20:54.061 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:54.058+0000 7f00df3d2700 1 -- 172.21.15.67:0/404255889 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f00d8103750 msgr2=0x7f00d819f7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:20:54.061 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:54.058+0000 7f00df3d2700 1 --2- 172.21.15.67:0/404255889 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f00d8103750 0x7f00d819f7b0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f00c800c9b0 tx=0x7f00c800ccc0 comp rx=0 tx=0).stop 2024-05-03T21:20:54.061 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:54.058+0000 7f00df3d2700 1 -- 172.21.15.67:0/404255889 shutdown_connections 2024-05-03T21:20:54.061 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:54.058+0000 7f00df3d2700 1 --2- 172.21.15.67:0/404255889 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f00c40b2910 0x7f00c40b4dd0 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:20:54.061 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:54.058+0000 7f00df3d2700 1 --2- 172.21.15.67:0/404255889 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f00d8103750 0x7f00d819f7b0 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:20:54.061 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:54.058+0000 7f00df3d2700 1 --2- 172.21.15.67:0/404255889 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f00d8108850 0x7f00d8199970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:20:54.061 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:54.058+0000 7f00df3d2700 1 --2- 172.21.15.67:0/404255889 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f00d8104100 0x7f00d819fcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:20:54.061 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:54.058+0000 7f00df3d2700 1 -- 172.21.15.67:0/404255889 >> 172.21.15.67:0/404255889 conn(0x7f00d807a600 msgr2=0x7f00d8101360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T21:20:54.061 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:54.058+0000 7f00df3d2700 1 -- 172.21.15.67:0/404255889 shutdown_connections 2024-05-03T21:20:54.061 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:54.058+0000 7f00df3d2700 1 -- 172.21.15.67:0/404255889 wait complete. 2024-05-03T21:20:54.062 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 25 2024-05-03T21:20:54.354 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:20:54 smithi067 ceph-mon[29730]: from='client.? 172.21.15.67:0/404255889' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-03T21:20:54.495 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:20:54 smithi082 ceph-mon[32933]: from='client.? 172.21.15.67:0/404255889' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-03T21:20:54.574 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:20:54 smithi138 ceph-mon[33110]: from='client.? 172.21.15.67:0/404255889' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-03T21:20:54.670 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 25, 'flags': 50, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-05-03T21:18:44.665825+0000', 'modified': '2024-05-03T21:19:03.775546+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 3, 'in': [0, 1, 2], 'up': {'mds_0': 24473, 'mds_1': 24449, 'mds_2': 24425}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24406': {'gid': 24406, 'name': 'c', 'rank': 2, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.138:6837/2203273249', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.138:6836', 'nonce': 2203273249}, {'type': 'v1', 'addr': '172.21.15.138:6837', 'nonce': 2203273249}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24407': {'gid': 24407, 'name': 'k', 'rank': 1, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.82:6833/1127287628', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.82:6832', 'nonce': 1127287628}, {'type': 'v1', 'addr': '172.21.15.82:6833', 'nonce': 1127287628}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24418': {'gid': 24418, 'name': 'i', 'rank': 0, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.138:6839/334382443', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.138:6838', 'nonce': 334382443}, {'type': 'v1', 'addr': '172.21.15.138:6839', 'nonce': 334382443}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24425': {'gid': 24425, 'name': 'e', 'rank': 2, 'incarnation': 21, 'state': 'up:active', 'state_seq': 14, 'addr': '172.21.15.82:6835/280518761', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.82:6834', 'nonce': 280518761}, {'type': 'v1', 'addr': '172.21.15.82:6835', 'nonce': 280518761}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24449': {'gid': 24449, 'name': 'h', 'rank': 1, 'incarnation': 19, 'state': 'up:active', 'state_seq': 11, 'addr': '172.21.15.82:6837/1275987852', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.82:6836', 'nonce': 1275987852}, {'type': 'v1', 'addr': '172.21.15.82:6837', 'nonce': 1275987852}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24473': {'gid': 24473, 'name': 'b', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 5, 'addr': '172.21.15.82:6839/3644499578', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.82:6838', 'nonce': 3644499578}, {'type': 'v1', 'addr': '172.21.15.82:6839', 'nonce': 3644499578}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 24473, 'qdb_cluster': [24449, 24473, 24425]} 2024-05-03T21:20:54.671 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 3/3 2024-05-03T21:20:54.671 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9a047fe7599a549042d06a3fb5b6813a203eabea shell --fsid 713bc57a-0991-11ef-bc97-c7b262605968 -- ceph tell mds.e status 2024-05-03T21:20:54.924 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/713bc57a-0991-11ef-bc97-c7b262605968/mon.a/config 2024-05-03T21:20:55.376 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:20:55 smithi067 ceph-mon[29730]: pgmap v354: 97 pgs: 97 active+clean; 632 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:20:55.495 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:20:55 smithi082 ceph-mon[32933]: pgmap v354: 97 pgs: 97 active+clean; 632 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:20:55.574 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:20:55 smithi138 ceph-mon[33110]: pgmap v354: 97 pgs: 97 active+clean; 632 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:20:56.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:56.494+0000 7efc943b6700 1 -- 172.21.15.67:0/152701485 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efc8c104130 msgr2=0x7efc8c108140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:20:56.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:56.494+0000 7efc943b6700 1 --2- 172.21.15.67:0/152701485 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efc8c104130 0x7efc8c108140 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7efc7c0099c0 tx=0x7efc7c009cd0 comp rx=0 tx=0).stop 2024-05-03T21:20:56.497 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:56.494+0000 7efc943b6700 1 -- 172.21.15.67:0/152701485 shutdown_connections 2024-05-03T21:20:56.497 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:56.494+0000 7efc943b6700 1 --2- 172.21.15.67:0/152701485 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7efc8c108870 0x7efc8c1103f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:20:56.497 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:56.494+0000 7efc943b6700 1 --2- 172.21.15.67:0/152701485 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7efc8c103780 0x7efc8c103b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:20:56.497 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:56.494+0000 7efc943b6700 1 --2- 172.21.15.67:0/152701485 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efc8c104130 0x7efc8c108140 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:20:56.497 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:56.494+0000 7efc943b6700 1 -- 172.21.15.67:0/152701485 >> 172.21.15.67:0/152701485 conn(0x7efc8c07a610 msgr2=0x7efc8c07aa20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T21:20:56.497 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:56.494+0000 7efc943b6700 1 -- 172.21.15.67:0/152701485 shutdown_connections 2024-05-03T21:20:56.497 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:56.494+0000 7efc943b6700 1 -- 172.21.15.67:0/152701485 wait complete. 2024-05-03T21:20:56.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:56.495+0000 7efc943b6700 1 Processor -- start 2024-05-03T21:20:56.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:56.495+0000 7efc943b6700 1 -- start start 2024-05-03T21:20:56.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:56.495+0000 7efc943b6700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7efc8c103780 0x7efc8c19f8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:20:56.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:56.495+0000 7efc943b6700 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7efc8c104130 0x7efc8c19fe20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:20:56.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:56.495+0000 7efc943b6700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efc8c108870 0x7efc8c199aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:20:56.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:56.495+0000 7efc943b6700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7efc8c1138e0 con 0x7efc8c108870 2024-05-03T21:20:56.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:56.495+0000 7efc943b6700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7efc8c113760 con 0x7efc8c103780 2024-05-03T21:20:56.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:56.495+0000 7efc943b6700 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7efc8c113a60 con 0x7efc8c104130 2024-05-03T21:20:56.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:56.495+0000 7efc92953700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efc8c108870 0x7efc8c199aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:20:56.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:56.495+0000 7efc92953700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efc8c108870 0x7efc8c199aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:34868/0 (socket says 172.21.15.67:34868) 2024-05-03T21:20:56.501 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:56.495+0000 7efc92953700 1 -- 172.21.15.67:0/3234688192 learned_addr learned my addr 172.21.15.67:0/3234688192 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-03T21:20:56.501 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:56.495+0000 7efc92152700 1 --2- 172.21.15.67:0/3234688192 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7efc8c103780 0x7efc8c19f8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:20:56.501 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:56.495+0000 7efc91951700 1 --2- 172.21.15.67:0/3234688192 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7efc8c104130 0x7efc8c19fe20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:20:56.501 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:56.495+0000 7efc92953700 1 -- 172.21.15.67:0/3234688192 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7efc8c104130 msgr2=0x7efc8c19fe20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:20:56.501 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:56.495+0000 7efc92953700 1 --2- 172.21.15.67:0/3234688192 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7efc8c104130 0x7efc8c19fe20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:20:56.501 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:56.495+0000 7efc92953700 1 -- 172.21.15.67:0/3234688192 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7efc8c103780 msgr2=0x7efc8c19f8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:20:56.501 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:56.495+0000 7efc92953700 1 --2- 172.21.15.67:0/3234688192 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7efc8c103780 0x7efc8c19f8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:20:56.501 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:56.495+0000 7efc92953700 1 -- 172.21.15.67:0/3234688192 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efc8c19a200 con 0x7efc8c108870 2024-05-03T21:20:56.501 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:56.496+0000 7efc92953700 1 --2- 172.21.15.67:0/3234688192 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efc8c108870 0x7efc8c199aa0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7efc84007ee0 tx=0x7efc8400d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T21:20:56.501 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:56.496+0000 7efc837fe700 1 -- 172.21.15.67:0/3234688192 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc8400dbd0 con 0x7efc8c108870 2024-05-03T21:20:56.501 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:56.496+0000 7efc837fe700 1 -- 172.21.15.67:0/3234688192 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(28 keys) ==== 1170+0+0 (secure 0 0 0) 0x7efc8400dd70 con 0x7efc8c108870 2024-05-03T21:20:56.501 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:56.496+0000 7efc943b6700 1 -- 172.21.15.67:0/3234688192 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efc8c19a4f0 con 0x7efc8c108870 2024-05-03T21:20:56.502 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:56.496+0000 7efc837fe700 1 -- 172.21.15.67:0/3234688192 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc840133f0 con 0x7efc8c108870 2024-05-03T21:20:56.502 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:56.496+0000 7efc943b6700 1 -- 172.21.15.67:0/3234688192 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efc8c1a67b0 con 0x7efc8c108870 2024-05-03T21:20:56.510 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:56.498+0000 7efc837fe700 1 -- 172.21.15.67:0/3234688192 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 17) ==== 149703+0+0 (secure 0 0 0) 0x7efc8401a020 con 0x7efc8c108870 2024-05-03T21:20:56.510 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:56.499+0000 7efc837fe700 1 --2- 172.21.15.67:0/3234688192 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7efc780b2990 0x7efc780b4e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:20:56.510 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:56.499+0000 7efc92152700 1 --2- 172.21.15.67:0/3234688192 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7efc780b2990 0x7efc780b4e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:20:56.510 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:56.499+0000 7efc837fe700 1 -- 172.21.15.67:0/3234688192 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 8211+0+0 (secure 0 0 0) 0x7efc840e19b0 con 0x7efc8c108870 2024-05-03T21:20:56.510 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:56.499+0000 7efc9ec49b80 1 Processor -- start 2024-05-03T21:20:56.510 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:56.499+0000 7efc92152700 1 --2- 172.21.15.67:0/3234688192 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7efc780b2990 0x7efc780b4e50 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7efc8c19abe0 tx=0x7efc880073d0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T21:20:56.510 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:56.500+0000 7efc9ec49b80 1 -- start start 2024-05-03T21:20:56.510 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:56.500+0000 7efc9ec49b80 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55c6ce352020 0x55c6ce352400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:20:56.511 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:56.500+0000 7efc9ec49b80 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55c6ce352940 0x55c6ce356100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:20:56.511 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:56.500+0000 7efc9ec49b80 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55c6ce356640 0x55c6ce356aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:20:56.511 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:56.500+0000 7efc9ec49b80 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x55c6ce2a0810 con 0x55c6ce356640 2024-05-03T21:20:56.511 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:56.500+0000 7efc9ec49b80 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x55c6ce23f770 con 0x55c6ce352940 2024-05-03T21:20:56.511 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:56.500+0000 7efc9ec49b80 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x55c6ce29c7d0 con 0x55c6ce352020 2024-05-03T21:20:56.511 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:56.500+0000 7efc92953700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55c6ce356640 0x55c6ce356aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:20:56.511 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:56.500+0000 7efc92953700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55c6ce356640 0x55c6ce356aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:34878/0 (socket says 172.21.15.67:34878) 2024-05-03T21:20:56.511 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:56.500+0000 7efc92953700 1 -- 172.21.15.67:0/3159745052 learned_addr learned my addr 172.21.15.67:0/3159745052 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-03T21:20:56.511 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:56.500+0000 7efc91951700 1 --2- 172.21.15.67:0/3159745052 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55c6ce352940 0x55c6ce356100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:20:56.511 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:56.500+0000 7efc92152700 1 --2- 172.21.15.67:0/3159745052 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55c6ce352020 0x55c6ce352400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:20:56.511 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:56.500+0000 7efc92953700 1 -- 172.21.15.67:0/3159745052 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55c6ce352020 msgr2=0x55c6ce352400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-03T21:20:56.511 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:56.500+0000 7efc92953700 1 --2- 172.21.15.67:0/3159745052 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55c6ce352020 0x55c6ce352400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:20:56.511 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:56.500+0000 7efc92953700 1 -- 172.21.15.67:0/3159745052 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55c6ce352940 msgr2=0x55c6ce356100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-03T21:20:56.511 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:56.500+0000 7efc92953700 1 --2- 172.21.15.67:0/3159745052 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55c6ce352940 0x55c6ce356100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:20:56.512 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:56.500+0000 7efc92953700 1 -- 172.21.15.67:0/3159745052 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c6ce1ed880 con 0x55c6ce356640 2024-05-03T21:20:56.512 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:56.500+0000 7efc92152700 1 --2- 172.21.15.67:0/3159745052 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55c6ce352020 0x55c6ce352400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-03T21:20:56.512 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:56.500+0000 7efc92953700 1 --2- 172.21.15.67:0/3159745052 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55c6ce356640 0x55c6ce356aa0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7efc840a4f00 tx=0x7efc840a86f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=889fbf193cf95fe9 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T21:20:56.512 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:56.501+0000 7efc80d70700 1 -- 172.21.15.67:0/3159745052 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc840a8e80 con 0x55c6ce356640 2024-05-03T21:20:56.512 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:56.501+0000 7efc80d70700 1 -- 172.21.15.67:0/3159745052 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(28 keys) ==== 1170+0+0 (secure 0 0 0) 0x7efc840ad230 con 0x55c6ce356640 2024-05-03T21:20:56.512 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:56.501+0000 7efc80d70700 1 -- 172.21.15.67:0/3159745052 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc840b1ab0 con 0x55c6ce356640 2024-05-03T21:20:56.512 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:56.501+0000 7efc9ec49b80 1 -- 172.21.15.67:0/3159745052 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55c6ce356640 msgr2=0x55c6ce356aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:20:56.512 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:56.501+0000 7efc9ec49b80 1 --2- 172.21.15.67:0/3159745052 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55c6ce356640 0x55c6ce356aa0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7efc840a4f00 tx=0x7efc840a86f0 comp rx=0 tx=0).stop 2024-05-03T21:20:56.512 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:56.501+0000 7efc9ec49b80 1 -- 172.21.15.67:0/3159745052 shutdown_connections 2024-05-03T21:20:56.512 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:56.501+0000 7efc9ec49b80 1 --2- 172.21.15.67:0/3159745052 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55c6ce356640 0x55c6ce356aa0 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:20:56.512 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:56.501+0000 7efc9ec49b80 1 --2- 172.21.15.67:0/3159745052 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55c6ce352020 0x55c6ce352400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:20:56.512 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:56.501+0000 7efc9ec49b80 1 --2- 172.21.15.67:0/3159745052 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55c6ce352940 0x55c6ce356100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:20:56.512 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:56.501+0000 7efc9ec49b80 1 -- 172.21.15.67:0/3159745052 >> 172.21.15.67:0/3159745052 conn(0x55c6ce27a0b0 msgr2=0x55c6ce348280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T21:20:56.512 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:56.502+0000 7efc9ec49b80 1 -- 172.21.15.67:0/3159745052 shutdown_connections 2024-05-03T21:20:56.512 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:56.502+0000 7efc9ec49b80 1 -- 172.21.15.67:0/3159745052 wait complete. 2024-05-03T21:20:56.513 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:56.502+0000 7efc9ec49b80 1 -- start start 2024-05-03T21:20:56.513 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:56.502+0000 7efc9ec49b80 1 Processor -- start 2024-05-03T21:20:56.513 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:56.502+0000 7efc9ec49b80 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55c6ce352020 0x55c6ce35db90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:20:56.513 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:56.502+0000 7efc9ec49b80 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55c6ce352940 0x55c6ce35af30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:20:56.513 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:56.502+0000 7efc9ec49b80 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55c6ce356640 0x55c6ce35b470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:20:56.513 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:56.502+0000 7efc9ec49b80 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x55c6ce35e8a0 con 0x55c6ce356640 2024-05-03T21:20:56.513 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:56.502+0000 7efc9ec49b80 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x55c6ce206be0 con 0x55c6ce352940 2024-05-03T21:20:56.513 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:56.502+0000 7efc9ec49b80 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x55c6ce35ea20 con 0x55c6ce352020 2024-05-03T21:20:56.513 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:56.502+0000 7efc9ec49b80 10 client.0 _ll_register_callbacks cb 0x55c6ce2388d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-03T21:20:56.513 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:56.502+0000 7efc92953700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55c6ce356640 0x55c6ce35b470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:20:56.513 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:56.502+0000 7efc92953700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55c6ce356640 0x55c6ce35b470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:34886/0 (socket says 172.21.15.67:34886) 2024-05-03T21:20:56.513 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:56.502+0000 7efc92953700 1 -- 172.21.15.67:0/1184209067 learned_addr learned my addr 172.21.15.67:0/1184209067 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-03T21:20:56.513 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:56.502+0000 7efc91951700 1 --2- 172.21.15.67:0/1184209067 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55c6ce352940 0x55c6ce35af30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:20:56.513 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:56.502+0000 7efc92152700 1 --2- 172.21.15.67:0/1184209067 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55c6ce352020 0x55c6ce35db90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:20:56.514 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:56.503+0000 7efc92953700 1 -- 172.21.15.67:0/1184209067 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55c6ce352020 msgr2=0x55c6ce35db90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-03T21:20:56.514 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:56.503+0000 7efc92953700 1 --2- 172.21.15.67:0/1184209067 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55c6ce352020 0x55c6ce35db90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:20:56.514 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:56.503+0000 7efc92953700 1 -- 172.21.15.67:0/1184209067 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55c6ce352940 msgr2=0x55c6ce35af30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-03T21:20:56.514 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:56.503+0000 7efc92953700 1 --2- 172.21.15.67:0/1184209067 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55c6ce352940 0x55c6ce35af30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:20:56.514 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:56.503+0000 7efc92953700 1 -- 172.21.15.67:0/1184209067 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c6ce23c640 con 0x55c6ce356640 2024-05-03T21:20:56.514 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:56.503+0000 7efc91951700 1 --2- 172.21.15.67:0/1184209067 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55c6ce352940 0x55c6ce35af30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-03T21:20:56.514 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:56.503+0000 7efc92953700 1 --2- 172.21.15.67:0/1184209067 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55c6ce356640 0x55c6ce35b470 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7efc84000f80 tx=0x7efc84004f00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3c20d6132f0a9d1d server_cookie=0 in_seq=0 out_seq=0 2024-05-03T21:20:56.514 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:56.503+0000 7efc75ffb700 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-05-03T21:20:56.514 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:56.503+0000 7efc75ffb700 1 -- 172.21.15.67:0/1184209067 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc840c7020 con 0x55c6ce356640 2024-05-03T21:20:56.514 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:56.503+0000 7efc75ffb700 1 -- 172.21.15.67:0/1184209067 <== mon.0 v2:172.21.15.67:3300/0 2 ==== osd_map(84..84 src has 1..84) ==== 8211+0+0 (secure 0 0 0) 0x7efc840b1a60 con 0x55c6ce356640 2024-05-03T21:20:56.514 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:56.503+0000 7efc9ec49b80 1 -- 172.21.15.67:0/1184209067 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7efc840ad230 con 0x55c6ce356640 2024-05-03T21:20:56.514 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:56.503+0000 7efc75ffb700 1 -- 172.21.15.67:0/1184209067 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7efc840a80e0 con 0x55c6ce356640 2024-05-03T21:20:56.514 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:56.503+0000 7efc9ec49b80 10 client.14964 fetch_fsmap learned FSMap version 25 2024-05-03T21:20:56.515 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:56.503+0000 7efc9ec49b80 1 -- 172.21.15.67:0/1184209067 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55c6ce369120 con 0x55c6ce356640 2024-05-03T21:20:56.515 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:56.503+0000 7efc75ffb700 1 -- 172.21.15.67:0/1184209067 <== mon.0 v2:172.21.15.67:3300/0 4 ==== config(28 keys) ==== 1170+0+0 (secure 0 0 0) 0x7efc840ad230 con 0x55c6ce356640 2024-05-03T21:20:56.515 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:56.503+0000 7efc75ffb700 1 -- 172.21.15.67:0/1184209067 <== mon.0 v2:172.21.15.67:3300/0 5 ==== fsmap(e 25) ==== 8338+0+0 (secure 0 0 0) 0x7efc840ae3c0 con 0x55c6ce356640 2024-05-03T21:20:56.515 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:56.504+0000 7efc9ec49b80 10 client.14964 fetch_fsmap finished waiting for FSMap version 25 2024-05-03T21:20:56.515 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:56.504+0000 7efc9ec49b80 10 client.14964 resolve_mds: resolved name 'e' to daemon mds.e 2024-05-03T21:20:56.515 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:56.504+0000 7efc9ec49b80 20 client.14964 populate_metadata read hostname 'smithi067' 2024-05-03T21:20:56.515 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:56.504+0000 7efc9ec49b80 1 --2- 172.21.15.67:0/1184209067 >> [v2:172.21.15.82:6834/280518761,v1:172.21.15.82:6835/280518761] conn(0x55c6ce36a470 0x55c6ce36c560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:20:56.515 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:56.504+0000 7efc9ec49b80 4 client.14964 mds_command: new command op to 24425 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-03T21:20:56.515 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:56.504+0000 7efc9ec49b80 1 -- 172.21.15.67:0/1184209067 --> [v2:172.21.15.82:6834/280518761,v1:172.21.15.82:6835/280518761] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c6ce2ae1f0 con 0x55c6ce36a470 2024-05-03T21:20:56.515 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:56.504+0000 7efc92152700 1 --2- 172.21.15.67:0/1184209067 >> [v2:172.21.15.82:6834/280518761,v1:172.21.15.82:6835/280518761] conn(0x55c6ce36a470 0x55c6ce36c560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:20:56.515 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:56.504+0000 7efc92152700 1 --2- 172.21.15.67:0/1184209067 >> [v2:172.21.15.82:6834/280518761,v1:172.21.15.82:6835/280518761] conn(0x55c6ce36a470 0x55c6ce36c560 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=272467dcf1acc855 server_cookie=de5ad44a40e6f19b in_seq=0 out_seq=0 2024-05-03T21:20:56.515 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:56.504+0000 7efc75ffb700 10 client.14964 ms_handle_connect on v2:172.21.15.82:6834/280518761 2024-05-03T21:20:56.515 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:56.505+0000 7efc75ffb700 1 -- 172.21.15.67:0/1184209067 <== mds.2 v2:172.21.15.82:6834/280518761 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55c6ce2ae1f0 con 0x55c6ce36a470 2024-05-03T21:20:56.515 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:56.505+0000 7efc75ffb700 10 client.14964 handle_command_reply: tid=0 2024-05-03T21:20:56.515 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:56.505+0000 7efc9ec49b80 1 client.14964 shutdown 2024-05-03T21:20:56.516 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:56.505+0000 7efc9ec49b80 1 -- 172.21.15.67:0/1184209067 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55c6ce356640 msgr2=0x55c6ce35b470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:20:56.516 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:56.505+0000 7efc9ec49b80 1 --2- 172.21.15.67:0/1184209067 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55c6ce356640 0x55c6ce35b470 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7efc84000f80 tx=0x7efc84004f00 comp rx=0 tx=0).stop 2024-05-03T21:20:56.516 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:56.505+0000 7efc9ec49b80 1 -- 172.21.15.67:0/1184209067 shutdown_connections 2024-05-03T21:20:56.516 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:56.505+0000 7efc9ec49b80 1 --2- 172.21.15.67:0/1184209067 >> [v2:172.21.15.82:6834/280518761,v1:172.21.15.82:6835/280518761] conn(0x55c6ce36a470 0x55c6ce36c560 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:20:56.516 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:56.506+0000 7efc9ec49b80 1 --2- 172.21.15.67:0/1184209067 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55c6ce356640 0x55c6ce35b470 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:20:56.516 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:56.506+0000 7efc9ec49b80 1 --2- 172.21.15.67:0/1184209067 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55c6ce352020 0x55c6ce35db90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:20:56.516 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:56.506+0000 7efc9ec49b80 1 --2- 172.21.15.67:0/1184209067 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55c6ce352940 0x55c6ce35af30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:20:56.516 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:56.506+0000 7efc9ec49b80 1 -- 172.21.15.67:0/1184209067 >> 172.21.15.67:0/1184209067 conn(0x55c6ce27a0b0 msgr2=0x55c6ce357af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T21:20:56.517 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:56.506+0000 7efc75ffb700 0 client.14964 ms_handle_reset on v2:172.21.15.82:6834/280518761 2024-05-03T21:20:56.517 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:56.506+0000 7efc9ec49b80 1 -- 172.21.15.67:0/1184209067 shutdown_connections 2024-05-03T21:20:56.517 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:56.506+0000 7efc9ec49b80 1 -- 172.21.15.67:0/1184209067 wait complete. 2024-05-03T21:20:56.517 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:56.506+0000 7efc9ec49b80 20 client.14964 trim_cache size 0 max 16384 2024-05-03T21:20:56.530 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:56.526+0000 7efc9ec49b80 1 Processor -- start 2024-05-03T21:20:56.530 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:56.527+0000 7efc9ec49b80 1 -- start start 2024-05-03T21:20:56.530 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:56.527+0000 7efc9ec49b80 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55c6ce352020 0x55c6ce35b4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:20:56.530 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:56.527+0000 7efc9ec49b80 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55c6ce352940 0x55c6ce35d510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:20:56.530 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:56.527+0000 7efc9ec49b80 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55c6ce356640 0x55c6ce35da50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:20:56.530 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:56.527+0000 7efc9ec49b80 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x55c6ce36a8a0 con 0x55c6ce352020 2024-05-03T21:20:56.530 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:56.527+0000 7efc9ec49b80 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x55c6ce36af70 con 0x55c6ce352940 2024-05-03T21:20:56.530 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:56.527+0000 7efc9ec49b80 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x55c6ce36b640 con 0x55c6ce356640 2024-05-03T21:20:56.531 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:56.527+0000 7efc92152700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55c6ce352020 0x55c6ce35b4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:20:56.531 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:56.527+0000 7efc92152700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55c6ce352020 0x55c6ce35b4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:34896/0 (socket says 172.21.15.67:34896) 2024-05-03T21:20:56.531 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:56.527+0000 7efc92152700 1 -- 172.21.15.67:0/751616536 learned_addr learned my addr 172.21.15.67:0/751616536 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-03T21:20:56.531 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:56.527+0000 7efc92152700 1 -- 172.21.15.67:0/751616536 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55c6ce356640 msgr2=0x55c6ce35da50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-03T21:20:56.531 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:56.527+0000 7efc92152700 1 --2- 172.21.15.67:0/751616536 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55c6ce356640 0x55c6ce35da50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:20:56.531 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:56.527+0000 7efc92152700 1 -- 172.21.15.67:0/751616536 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55c6ce352940 msgr2=0x55c6ce35d510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-03T21:20:56.531 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:56.527+0000 7efc92152700 1 --2- 172.21.15.67:0/751616536 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55c6ce352940 0x55c6ce35d510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:20:56.531 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:56.527+0000 7efc92152700 1 -- 172.21.15.67:0/751616536 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c6ce35df90 con 0x55c6ce352020 2024-05-03T21:20:56.531 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:56.527+0000 7efc92152700 1 --2- 172.21.15.67:0/751616536 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55c6ce352020 0x55c6ce35b4a0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7efc88010e70 tx=0x7efc88013000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=54aa40c2185b5b85 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T21:20:56.531 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:56.527+0000 7efc77fff700 1 -- 172.21.15.67:0/751616536 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc880044c0 con 0x55c6ce352020 2024-05-03T21:20:56.531 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:56.527+0000 7efc77fff700 1 -- 172.21.15.67:0/751616536 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(28 keys) ==== 1170+0+0 (secure 0 0 0) 0x7efc88015070 con 0x55c6ce352020 2024-05-03T21:20:56.531 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:56.527+0000 7efc77fff700 1 -- 172.21.15.67:0/751616536 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc88011640 con 0x55c6ce352020 2024-05-03T21:20:56.535 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:56.528+0000 7efc9ec49b80 1 -- 172.21.15.67:0/751616536 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55c6ce352020 msgr2=0x55c6ce35b4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:20:56.535 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:56.528+0000 7efc9ec49b80 1 --2- 172.21.15.67:0/751616536 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55c6ce352020 0x55c6ce35b4a0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7efc88010e70 tx=0x7efc88013000 comp rx=0 tx=0).stop 2024-05-03T21:20:56.535 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:56.528+0000 7efc9ec49b80 1 -- 172.21.15.67:0/751616536 shutdown_connections 2024-05-03T21:20:56.535 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:56.528+0000 7efc9ec49b80 1 --2- 172.21.15.67:0/751616536 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55c6ce352020 0x55c6ce35b4a0 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:20:56.535 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:56.528+0000 7efc9ec49b80 1 --2- 172.21.15.67:0/751616536 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55c6ce356640 0x55c6ce35da50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:20:56.535 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:56.528+0000 7efc9ec49b80 1 --2- 172.21.15.67:0/751616536 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55c6ce352940 0x55c6ce35d510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:20:56.535 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:56.528+0000 7efc9ec49b80 1 -- 172.21.15.67:0/751616536 >> 172.21.15.67:0/751616536 conn(0x55c6ce27a0b0 msgr2=0x55c6ce359820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T21:20:56.535 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:56.528+0000 7efc9ec49b80 1 -- 172.21.15.67:0/751616536 shutdown_connections 2024-05-03T21:20:56.535 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:56.528+0000 7efc9ec49b80 1 -- 172.21.15.67:0/751616536 wait complete. 2024-05-03T21:20:56.535 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:56.528+0000 7efc9ec49b80 1 -- start start 2024-05-03T21:20:56.535 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:56.529+0000 7efc9ec49b80 1 Processor -- start 2024-05-03T21:20:56.535 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:56.529+0000 7efc9ec49b80 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55c6ce352020 0x55c6ce375f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:20:56.535 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:56.529+0000 7efc9ec49b80 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55c6ce352940 0x55c6ce372c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:20:56.536 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:56.529+0000 7efc9ec49b80 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55c6ce356640 0x55c6ce373150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:20:56.536 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:56.529+0000 7efc9ec49b80 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x55c6ce36c3e0 con 0x55c6ce352940 2024-05-03T21:20:56.536 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:56.529+0000 7efc9ec49b80 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x55c6ce36bd10 con 0x55c6ce356640 2024-05-03T21:20:56.536 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:56.529+0000 7efc9ec49b80 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x55c6ce36cab0 con 0x55c6ce352020 2024-05-03T21:20:56.536 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:56.529+0000 7efc9ec49b80 10 client.0 _ll_register_callbacks cb 0x55c6ce2388d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-03T21:20:56.536 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:56.529+0000 7efc91951700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55c6ce352940 0x55c6ce372c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:20:56.536 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:56.529+0000 7efc91951700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55c6ce352940 0x55c6ce372c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:34908/0 (socket says 172.21.15.67:34908) 2024-05-03T21:20:56.536 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:56.529+0000 7efc91951700 1 -- 172.21.15.67:0/2353189970 learned_addr learned my addr 172.21.15.67:0/2353189970 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-03T21:20:56.536 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:56.529+0000 7efc92152700 1 --2- 172.21.15.67:0/2353189970 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55c6ce352020 0x55c6ce375f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:20:56.536 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:56.529+0000 7efc92953700 1 --2- 172.21.15.67:0/2353189970 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55c6ce356640 0x55c6ce373150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:20:56.536 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:56.529+0000 7efc91951700 1 -- 172.21.15.67:0/2353189970 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55c6ce352020 msgr2=0x55c6ce375f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-03T21:20:56.536 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:56.529+0000 7efc91951700 1 --2- 172.21.15.67:0/2353189970 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55c6ce352020 0x55c6ce375f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:20:56.536 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:56.529+0000 7efc91951700 1 -- 172.21.15.67:0/2353189970 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55c6ce356640 msgr2=0x55c6ce373150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-03T21:20:56.536 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:56.529+0000 7efc91951700 1 --2- 172.21.15.67:0/2353189970 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55c6ce356640 0x55c6ce373150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:20:56.537 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:56.529+0000 7efc91951700 1 -- 172.21.15.67:0/2353189970 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c6ce3739a0 con 0x55c6ce352940 2024-05-03T21:20:56.537 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:56.529+0000 7efc91951700 1 --2- 172.21.15.67:0/2353189970 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55c6ce352940 0x55c6ce372c10 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7efc7c0098e0 tx=0x7efc7c003d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ca8d86fbf8ff591a server_cookie=0 in_seq=0 out_seq=0 2024-05-03T21:20:56.537 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:56.529+0000 7efc75ffb700 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-05-03T21:20:56.537 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:56.530+0000 7efc75ffb700 1 -- 172.21.15.67:0/2353189970 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc7c0052c0 con 0x55c6ce352940 2024-05-03T21:20:56.537 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:56.530+0000 7efc75ffb700 1 -- 172.21.15.67:0/2353189970 <== mon.0 v2:172.21.15.67:3300/0 2 ==== osd_map(84..84 src has 1..84) ==== 8211+0+0 (secure 0 0 0) 0x7efc7c005460 con 0x55c6ce352940 2024-05-03T21:20:56.537 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:56.530+0000 7efc9ec49b80 1 -- 172.21.15.67:0/2353189970 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7efc88015070 con 0x55c6ce352940 2024-05-03T21:20:56.537 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:56.530+0000 7efc75ffb700 1 -- 172.21.15.67:0/2353189970 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7efc7c005660 con 0x55c6ce352940 2024-05-03T21:20:56.537 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:56.530+0000 7efc9ec49b80 10 client.14970 fetch_fsmap learned FSMap version 25 2024-05-03T21:20:56.537 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:56.530+0000 7efc9ec49b80 1 -- 172.21.15.67:0/2353189970 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55c6ce3d65c0 con 0x55c6ce352940 2024-05-03T21:20:56.537 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:56.530+0000 7efc75ffb700 1 -- 172.21.15.67:0/2353189970 <== mon.0 v2:172.21.15.67:3300/0 4 ==== config(28 keys) ==== 1170+0+0 (secure 0 0 0) 0x7efc88015070 con 0x55c6ce352940 2024-05-03T21:20:56.537 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:56.530+0000 7efc75ffb700 1 -- 172.21.15.67:0/2353189970 <== mon.0 v2:172.21.15.67:3300/0 5 ==== fsmap(e 25) ==== 8338+0+0 (secure 0 0 0) 0x7efc7c023990 con 0x55c6ce352940 2024-05-03T21:20:56.538 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:56.530+0000 7efc9ec49b80 10 client.14970 fetch_fsmap finished waiting for FSMap version 25 2024-05-03T21:20:56.538 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:56.530+0000 7efc9ec49b80 10 client.14970 resolve_mds: resolved name 'e' to daemon mds.e 2024-05-03T21:20:56.538 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:56.530+0000 7efc9ec49b80 20 client.14970 populate_metadata read hostname 'smithi067' 2024-05-03T21:20:56.538 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:56.530+0000 7efc9ec49b80 1 --2- 172.21.15.67:0/2353189970 >> [v2:172.21.15.82:6834/280518761,v1:172.21.15.82:6835/280518761] conn(0x55c6ce3db120 0x55c6ce3d9e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:20:56.538 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:56.530+0000 7efc9ec49b80 4 client.14970 mds_command: new command op to 24425 tid=0 multi_id=0 [{"prefix": "status"}] 2024-05-03T21:20:56.538 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:56.530+0000 7efc9ec49b80 1 -- 172.21.15.67:0/2353189970 --> [v2:172.21.15.82:6834/280518761,v1:172.21.15.82:6835/280518761] -- command(tid 0: {"prefix": "status"}) -- 0x55c6ce2a8290 con 0x55c6ce3db120 2024-05-03T21:20:56.538 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:56.530+0000 7efc92152700 1 --2- 172.21.15.67:0/2353189970 >> [v2:172.21.15.82:6834/280518761,v1:172.21.15.82:6835/280518761] conn(0x55c6ce3db120 0x55c6ce3d9e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:20:56.538 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:56.531+0000 7efc92152700 1 --2- 172.21.15.67:0/2353189970 >> [v2:172.21.15.82:6834/280518761,v1:172.21.15.82:6835/280518761] conn(0x55c6ce3db120 0x55c6ce3d9e30 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=ff630583fe75fc41 server_cookie=f92c3369a51a2e1 in_seq=0 out_seq=0 2024-05-03T21:20:56.538 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:56.531+0000 7efc75ffb700 10 client.14970 ms_handle_connect on v2:172.21.15.82:6834/280518761 2024-05-03T21:20:56.538 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:56.531+0000 7efc75ffb700 1 -- 172.21.15.67:0/2353189970 <== mds.2 v2:172.21.15.82:6834/280518761 1 ==== command_reply(tid 0: 0 ) ==== 8+0+327 (crc 0 0 0) 0x55c6ce2a8290 con 0x55c6ce3db120 2024-05-03T21:20:56.539 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:56.531+0000 7efc75ffb700 10 client.14970 handle_command_reply: tid=0 2024-05-03T21:20:56.539 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:56.531+0000 7efc9ec49b80 1 client.14970 shutdown 2024-05-03T21:20:56.539 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:56.531+0000 7efc9ec49b80 1 -- 172.21.15.67:0/2353189970 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55c6ce352940 msgr2=0x55c6ce372c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:20:56.539 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:56.531+0000 7efc9ec49b80 1 --2- 172.21.15.67:0/2353189970 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55c6ce352940 0x55c6ce372c10 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7efc7c0098e0 tx=0x7efc7c003d50 comp rx=0 tx=0).stop 2024-05-03T21:20:56.539 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:56.531+0000 7efc9ec49b80 1 -- 172.21.15.67:0/2353189970 shutdown_connections 2024-05-03T21:20:56.539 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:56.531+0000 7efc9ec49b80 1 --2- 172.21.15.67:0/2353189970 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55c6ce356640 0x55c6ce373150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:20:56.539 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:56.531+0000 7efc9ec49b80 1 --2- 172.21.15.67:0/2353189970 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55c6ce352020 0x55c6ce375f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:20:56.539 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:56.531+0000 7efc9ec49b80 1 --2- 172.21.15.67:0/2353189970 >> [v2:172.21.15.82:6834/280518761,v1:172.21.15.82:6835/280518761] conn(0x55c6ce3db120 0x55c6ce3d9e30 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:20:56.539 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:56.531+0000 7efc9ec49b80 1 --2- 172.21.15.67:0/2353189970 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55c6ce352940 0x55c6ce372c10 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:20:56.539 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:56.531+0000 7efc75ffb700 0 client.14970 ms_handle_reset on v2:172.21.15.82:6834/280518761 2024-05-03T21:20:56.539 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:56.531+0000 7efc9ec49b80 1 -- 172.21.15.67:0/2353189970 >> 172.21.15.67:0/2353189970 conn(0x55c6ce27a0b0 msgr2=0x55c6ce34ca20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T21:20:56.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:56.531+0000 7efc9ec49b80 1 -- 172.21.15.67:0/2353189970 shutdown_connections 2024-05-03T21:20:56.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:56.531+0000 7efc9ec49b80 1 -- 172.21.15.67:0/2353189970 wait complete. 2024-05-03T21:20:56.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:56.532+0000 7efc9ec49b80 20 client.14970 trim_cache size 0 max 16384 2024-05-03T21:20:56.540 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-05-03T21:20:56.540 INFO:teuthology.orchestra.run.smithi067.stdout: "cluster_fsid": "713bc57a-0991-11ef-bc97-c7b262605968", 2024-05-03T21:20:56.540 INFO:teuthology.orchestra.run.smithi067.stdout: "whoami": 2, 2024-05-03T21:20:56.540 INFO:teuthology.orchestra.run.smithi067.stdout: "id": 24425, 2024-05-03T21:20:56.540 INFO:teuthology.orchestra.run.smithi067.stdout: "want_state": "up:active", 2024-05-03T21:20:56.541 INFO:teuthology.orchestra.run.smithi067.stdout: "state": "up:active", 2024-05-03T21:20:56.541 INFO:teuthology.orchestra.run.smithi067.stdout: "fs_name": "cephfs", 2024-05-03T21:20:56.541 INFO:teuthology.orchestra.run.smithi067.stdout: "rank_uptime": 118.667590632, 2024-05-03T21:20:56.541 INFO:teuthology.orchestra.run.smithi067.stdout: "mdsmap_epoch": 25, 2024-05-03T21:20:56.541 INFO:teuthology.orchestra.run.smithi067.stdout: "osdmap_epoch": 84, 2024-05-03T21:20:56.541 INFO:teuthology.orchestra.run.smithi067.stdout: "osdmap_epoch_barrier": 82, 2024-05-03T21:20:56.541 INFO:teuthology.orchestra.run.smithi067.stdout: "uptime": 167.16412156000001 2024-05-03T21:20:56.541 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-05-03T21:20:56.542 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:56.532+0000 7efc943b6700 1 -- 172.21.15.67:0/3234688192 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7efc780b2990 msgr2=0x7efc780b4e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:20:56.543 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:56.532+0000 7efc943b6700 1 --2- 172.21.15.67:0/3234688192 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7efc780b2990 0x7efc780b4e50 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7efc8c19abe0 tx=0x7efc880073d0 comp rx=0 tx=0).stop 2024-05-03T21:20:56.543 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:56.532+0000 7efc943b6700 1 -- 172.21.15.67:0/3234688192 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efc8c108870 msgr2=0x7efc8c199aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:20:56.543 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:56.532+0000 7efc943b6700 1 --2- 172.21.15.67:0/3234688192 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efc8c108870 0x7efc8c199aa0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7efc84007ee0 tx=0x7efc8400d3b0 comp rx=0 tx=0).stop 2024-05-03T21:20:56.543 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:56.532+0000 7efc943b6700 1 -- 172.21.15.67:0/3234688192 shutdown_connections 2024-05-03T21:20:56.543 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:56.532+0000 7efc943b6700 1 --2- 172.21.15.67:0/3234688192 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7efc780b2990 0x7efc780b4e50 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:20:56.543 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:56.532+0000 7efc943b6700 1 --2- 172.21.15.67:0/3234688192 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efc8c108870 0x7efc8c199aa0 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:20:56.544 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:56.532+0000 7efc943b6700 1 --2- 172.21.15.67:0/3234688192 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7efc8c103780 0x7efc8c19f8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:20:56.544 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:56.532+0000 7efc943b6700 1 --2- 172.21.15.67:0/3234688192 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7efc8c104130 0x7efc8c19fe20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:20:56.544 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:56.532+0000 7efc943b6700 1 -- 172.21.15.67:0/3234688192 >> 172.21.15.67:0/3234688192 conn(0x7efc8c07a610 msgr2=0x7efc8c102740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T21:20:56.544 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:56.532+0000 7efc943b6700 1 -- 172.21.15.67:0/3234688192 shutdown_connections 2024-05-03T21:20:56.544 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:56.532+0000 7efc943b6700 1 -- 172.21.15.67:0/3234688192 wait complete. 2024-05-03T21:20:57.198 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9a047fe7599a549042d06a3fb5b6813a203eabea shell --fsid 713bc57a-0991-11ef-bc97-c7b262605968 -- ceph tell mds.h status 2024-05-03T21:20:57.447 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/713bc57a-0991-11ef-bc97-c7b262605968/mon.a/config 2024-05-03T21:20:57.471 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:20:57 smithi067 ceph-mon[29730]: pgmap v355: 97 pgs: 97 active+clean; 632 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:20:57.495 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:20:57 smithi082 ceph-mon[32933]: pgmap v355: 97 pgs: 97 active+clean; 632 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:20:57.574 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:20:57 smithi138 ceph-mon[33110]: pgmap v355: 97 pgs: 97 active+clean; 632 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:20:59.120 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:59.117+0000 7fa1f8aa8700 1 -- 172.21.15.67:0/1490056453 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa1f406bb40 msgr2=0x7fa1f4110330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:20:59.120 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:59.117+0000 7fa1f8aa8700 1 --2- 172.21.15.67:0/1490056453 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa1f406bb40 0x7fa1f4110330 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7fa1ec00b0a0 tx=0x7fa1ec00b3b0 comp rx=0 tx=0).stop 2024-05-03T21:20:59.120 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:59.117+0000 7fa1f8aa8700 1 -- 172.21.15.67:0/1490056453 shutdown_connections 2024-05-03T21:20:59.120 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:59.117+0000 7fa1f8aa8700 1 --2- 172.21.15.67:0/1490056453 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa1f406bb40 0x7fa1f4110330 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:20:59.120 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:59.117+0000 7fa1f8aa8700 1 --2- 172.21.15.67:0/1490056453 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fa1f4109510 0x7fa1f41098f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:20:59.120 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:59.117+0000 7fa1f8aa8700 1 --2- 172.21.15.67:0/1490056453 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fa1f406b180 0x7fa1f406b600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:20:59.120 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:59.117+0000 7fa1f8aa8700 1 -- 172.21.15.67:0/1490056453 >> 172.21.15.67:0/1490056453 conn(0x7fa1f4079350 msgr2=0x7fa1f4079760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T21:20:59.121 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:59.118+0000 7fa1f8aa8700 1 -- 172.21.15.67:0/1490056453 shutdown_connections 2024-05-03T21:20:59.121 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:59.118+0000 7fa1f8aa8700 1 -- 172.21.15.67:0/1490056453 wait complete. 2024-05-03T21:20:59.122 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:59.118+0000 7fa1f8aa8700 1 Processor -- start 2024-05-03T21:20:59.122 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:59.118+0000 7fa1f8aa8700 1 -- start start 2024-05-03T21:20:59.122 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:59.119+0000 7fa1f8aa8700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fa1f406b180 0x7fa1f419c100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:20:59.122 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:59.119+0000 7fa1f8aa8700 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fa1f406bb40 0x7fa1f419c640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:20:59.122 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:59.119+0000 7fa1f8aa8700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa1f4109510 0x7fa1f41a0980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:20:59.123 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:59.119+0000 7fa1f8aa8700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fa1f41139b0 con 0x7fa1f4109510 2024-05-03T21:20:59.123 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:59.119+0000 7fa1f8aa8700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7fa1f4113830 con 0x7fa1f406b180 2024-05-03T21:20:59.123 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:59.119+0000 7fa1f8aa8700 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7fa1f4113b30 con 0x7fa1f406bb40 2024-05-03T21:20:59.123 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:59.119+0000 7fa1f2d9d700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa1f4109510 0x7fa1f41a0980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:20:59.123 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:59.119+0000 7fa1f2d9d700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa1f4109510 0x7fa1f41a0980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:34930/0 (socket says 172.21.15.67:34930) 2024-05-03T21:20:59.123 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:59.119+0000 7fa1f2d9d700 1 -- 172.21.15.67:0/1983788611 learned_addr learned my addr 172.21.15.67:0/1983788611 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-03T21:20:59.123 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:59.119+0000 7fa1f259c700 1 --2- 172.21.15.67:0/1983788611 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fa1f406b180 0x7fa1f419c100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:20:59.123 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:59.119+0000 7fa1ebfff700 1 --2- 172.21.15.67:0/1983788611 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fa1f406bb40 0x7fa1f419c640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:20:59.123 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:59.119+0000 7fa1f2d9d700 1 -- 172.21.15.67:0/1983788611 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fa1f406bb40 msgr2=0x7fa1f419c640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:20:59.123 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:59.119+0000 7fa1f2d9d700 1 --2- 172.21.15.67:0/1983788611 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fa1f406bb40 0x7fa1f419c640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:20:59.123 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:59.119+0000 7fa1f2d9d700 1 -- 172.21.15.67:0/1983788611 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fa1f406b180 msgr2=0x7fa1f419c100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:20:59.123 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:59.119+0000 7fa1f2d9d700 1 --2- 172.21.15.67:0/1983788611 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fa1f406b180 0x7fa1f419c100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:20:59.123 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:59.119+0000 7fa1f2d9d700 1 -- 172.21.15.67:0/1983788611 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa1f41a10c0 con 0x7fa1f4109510 2024-05-03T21:20:59.123 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:59.119+0000 7fa1f2d9d700 1 --2- 172.21.15.67:0/1983788611 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa1f4109510 0x7fa1f41a0980 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7fa1ec006070 tx=0x7fa1ec004e20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T21:20:59.123 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:59.120+0000 7fa1eb7fe700 1 -- 172.21.15.67:0/1983788611 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1ec021070 con 0x7fa1f4109510 2024-05-03T21:20:59.124 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:59.120+0000 7fa1f8aa8700 1 -- 172.21.15.67:0/1983788611 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa1f41a1350 con 0x7fa1f4109510 2024-05-03T21:20:59.124 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:59.120+0000 7fa1f8aa8700 1 -- 172.21.15.67:0/1983788611 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa1f41a8c30 con 0x7fa1f4109510 2024-05-03T21:20:59.124 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:59.120+0000 7fa1eb7fe700 1 -- 172.21.15.67:0/1983788611 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(28 keys) ==== 1170+0+0 (secure 0 0 0) 0x7fa1ec003c00 con 0x7fa1f4109510 2024-05-03T21:20:59.124 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:59.120+0000 7fa1eb7fe700 1 -- 172.21.15.67:0/1983788611 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1ec01d450 con 0x7fa1f4109510 2024-05-03T21:20:59.124 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:59.122+0000 7fa1eb7fe700 1 -- 172.21.15.67:0/1983788611 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 17) ==== 149703+0+0 (secure 0 0 0) 0x7fa1ec007a00 con 0x7fa1f4109510 2024-05-03T21:20:59.125 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:59.123+0000 7fa1eb7fe700 1 --2- 172.21.15.67:0/1983788611 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7fa1e00b2990 0x7fa1e00b4e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:20:59.125 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:59.123+0000 7fa1f259c700 1 --2- 172.21.15.67:0/1983788611 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7fa1e00b2990 0x7fa1e00b4e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:20:59.125 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:59.123+0000 7fa1eb7fe700 1 -- 172.21.15.67:0/1983788611 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 8211+0+0 (secure 0 0 0) 0x7fa1ec00f020 con 0x7fa1f4109510 2024-05-03T21:20:59.125 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:59.123+0000 7fa20333bb80 1 Processor -- start 2024-05-03T21:20:59.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:59.124+0000 7fa20333bb80 1 -- start start 2024-05-03T21:20:59.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:59.124+0000 7fa1f259c700 1 --2- 172.21.15.67:0/1983788611 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7fa1e00b2990 0x7fa1e00b4e50 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fa1e4009a10 tx=0x7fa1e4009190 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T21:20:59.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:59.124+0000 7fa20333bb80 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x555945de8740 0x555945de8b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:20:59.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:59.124+0000 7fa20333bb80 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x555945de9060 0x555945de3f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:20:59.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:59.124+0000 7fa20333bb80 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x555945de4480 0x555945de48c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:20:59.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:59.124+0000 7fa20333bb80 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x555945d368f0 con 0x555945de4480 2024-05-03T21:20:59.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:59.124+0000 7fa20333bb80 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x555945c9bd50 con 0x555945de9060 2024-05-03T21:20:59.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:59.124+0000 7fa20333bb80 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x555945d328b0 con 0x555945de8740 2024-05-03T21:20:59.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:59.124+0000 7fa1f2d9d700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x555945de4480 0x555945de48c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:20:59.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:59.124+0000 7fa1f2d9d700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x555945de4480 0x555945de48c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:34944/0 (socket says 172.21.15.67:34944) 2024-05-03T21:20:59.128 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:59.124+0000 7fa1f2d9d700 1 -- 172.21.15.67:0/765171528 learned_addr learned my addr 172.21.15.67:0/765171528 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-03T21:20:59.128 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:59.124+0000 7fa1ebfff700 1 --2- 172.21.15.67:0/765171528 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x555945de9060 0x555945de3f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:20:59.128 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:59.124+0000 7fa1f2d9d700 1 -- 172.21.15.67:0/765171528 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x555945de8740 msgr2=0x555945de8b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-03T21:20:59.128 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:59.124+0000 7fa1f2d9d700 1 --2- 172.21.15.67:0/765171528 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x555945de8740 0x555945de8b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:20:59.128 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:59.124+0000 7fa1f2d9d700 1 -- 172.21.15.67:0/765171528 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x555945de9060 msgr2=0x555945de3f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-03T21:20:59.128 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:59.124+0000 7fa1f2d9d700 1 --2- 172.21.15.67:0/765171528 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x555945de9060 0x555945de3f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:20:59.128 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:59.124+0000 7fa1f2d9d700 1 -- 172.21.15.67:0/765171528 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555945c83880 con 0x555945de4480 2024-05-03T21:20:59.128 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:59.124+0000 7fa1f2d9d700 1 --2- 172.21.15.67:0/765171528 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x555945de4480 0x555945de48c0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7fa1ec0aaa70 tx=0x7fa1ec007f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f574a2a6a2f6058a server_cookie=0 in_seq=0 out_seq=0 2024-05-03T21:20:59.128 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:59.125+0000 7fa1e8ff9700 1 -- 172.21.15.67:0/765171528 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1ec0af120 con 0x555945de4480 2024-05-03T21:20:59.128 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:59.125+0000 7fa1e8ff9700 1 -- 172.21.15.67:0/765171528 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(28 keys) ==== 1170+0+0 (secure 0 0 0) 0x7fa1ec0b3b40 con 0x555945de4480 2024-05-03T21:20:59.128 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:59.125+0000 7fa1e8ff9700 1 -- 172.21.15.67:0/765171528 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1ec0aafd0 con 0x555945de4480 2024-05-03T21:20:59.131 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:59.125+0000 7fa20333bb80 1 -- 172.21.15.67:0/765171528 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x555945de4480 msgr2=0x555945de48c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:20:59.131 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:59.125+0000 7fa20333bb80 1 --2- 172.21.15.67:0/765171528 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x555945de4480 0x555945de48c0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7fa1ec0aaa70 tx=0x7fa1ec007f90 comp rx=0 tx=0).stop 2024-05-03T21:20:59.131 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:59.125+0000 7fa20333bb80 1 -- 172.21.15.67:0/765171528 shutdown_connections 2024-05-03T21:20:59.131 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:59.125+0000 7fa20333bb80 1 --2- 172.21.15.67:0/765171528 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x555945de4480 0x555945de48c0 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:20:59.131 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:59.125+0000 7fa20333bb80 1 --2- 172.21.15.67:0/765171528 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x555945de8740 0x555945de8b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:20:59.131 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:59.125+0000 7fa20333bb80 1 --2- 172.21.15.67:0/765171528 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x555945de9060 0x555945de3f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:20:59.131 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:59.125+0000 7fa20333bb80 1 -- 172.21.15.67:0/765171528 >> 172.21.15.67:0/765171528 conn(0x555945d10b20 msgr2=0x555945dde9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T21:20:59.131 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:59.125+0000 7fa20333bb80 1 -- 172.21.15.67:0/765171528 shutdown_connections 2024-05-03T21:20:59.131 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:59.126+0000 7fa20333bb80 1 -- 172.21.15.67:0/765171528 wait complete. 2024-05-03T21:20:59.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:59.126+0000 7fa20333bb80 1 -- start start 2024-05-03T21:20:59.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:59.126+0000 7fa20333bb80 1 Processor -- start 2024-05-03T21:20:59.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:59.126+0000 7fa20333bb80 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x555945de4480 0x555945defdf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:20:59.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:59.126+0000 7fa20333bb80 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x555945de8740 0x555945ded200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:20:59.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:59.126+0000 7fa20333bb80 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x555945de9060 0x555945ded740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:20:59.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:59.126+0000 7fa20333bb80 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x555945df0960 con 0x555945de9060 2024-05-03T21:20:59.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:59.126+0000 7fa20333bb80 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x555945cefe80 con 0x555945de8740 2024-05-03T21:20:59.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:59.126+0000 7fa20333bb80 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x555945df0ae0 con 0x555945de4480 2024-05-03T21:20:59.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:59.126+0000 7fa20333bb80 10 client.0 _ll_register_callbacks cb 0x555945cca560 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-03T21:20:59.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:59.126+0000 7fa1f2d9d700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x555945de9060 0x555945ded740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:20:59.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:59.126+0000 7fa1f2d9d700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x555945de9060 0x555945ded740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:34956/0 (socket says 172.21.15.67:34956) 2024-05-03T21:20:59.133 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:59.126+0000 7fa1f2d9d700 1 -- 172.21.15.67:0/2514444112 learned_addr learned my addr 172.21.15.67:0/2514444112 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-03T21:20:59.133 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:59.126+0000 7fa1ebfff700 1 --2- 172.21.15.67:0/2514444112 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x555945de8740 0x555945ded200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:20:59.133 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:59.127+0000 7fa1f259c700 1 --2- 172.21.15.67:0/2514444112 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x555945de4480 0x555945defdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:20:59.133 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:59.127+0000 7fa1f2d9d700 1 -- 172.21.15.67:0/2514444112 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x555945de4480 msgr2=0x555945defdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-03T21:20:59.133 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:59.127+0000 7fa1f2d9d700 1 --2- 172.21.15.67:0/2514444112 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x555945de4480 0x555945defdf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:20:59.133 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:59.127+0000 7fa1f2d9d700 1 -- 172.21.15.67:0/2514444112 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x555945de8740 msgr2=0x555945ded200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-03T21:20:59.133 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:59.127+0000 7fa1f2d9d700 1 --2- 172.21.15.67:0/2514444112 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x555945de8740 0x555945ded200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:20:59.133 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:59.127+0000 7fa1f2d9d700 1 -- 172.21.15.67:0/2514444112 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555945ced710 con 0x555945de9060 2024-05-03T21:20:59.134 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:59.127+0000 7fa1f259c700 1 --2- 172.21.15.67:0/2514444112 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x555945de4480 0x555945defdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-03T21:20:59.134 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:59.127+0000 7fa1f2d9d700 1 --2- 172.21.15.67:0/2514444112 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x555945de9060 0x555945ded740 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7fa1ec004430 tx=0x7fa1ec0af1a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ab6c36f26d68b6a0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T21:20:59.134 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:59.127+0000 7fa1d9ffb700 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-05-03T21:20:59.134 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:59.127+0000 7fa1d9ffb700 1 -- 172.21.15.67:0/2514444112 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1ec0c53d0 con 0x555945de9060 2024-05-03T21:20:59.134 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:59.127+0000 7fa1d9ffb700 1 -- 172.21.15.67:0/2514444112 <== mon.0 v2:172.21.15.67:3300/0 2 ==== osd_map(84..84 src has 1..84) ==== 8211+0+0 (secure 0 0 0) 0x7fa1ec003fc0 con 0x555945de9060 2024-05-03T21:20:59.134 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:59.127+0000 7fa20333bb80 1 -- 172.21.15.67:0/2514444112 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa1ec0b3b40 con 0x555945de9060 2024-05-03T21:20:59.134 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:59.127+0000 7fa1d9ffb700 1 -- 172.21.15.67:0/2514444112 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7fa1ec007db0 con 0x555945de9060 2024-05-03T21:20:59.134 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:59.127+0000 7fa20333bb80 10 client.14982 fetch_fsmap learned FSMap version 25 2024-05-03T21:20:59.134 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:59.127+0000 7fa20333bb80 1 -- 172.21.15.67:0/2514444112 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x555945df0960 con 0x555945de9060 2024-05-03T21:20:59.134 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:59.127+0000 7fa1d9ffb700 1 -- 172.21.15.67:0/2514444112 <== mon.0 v2:172.21.15.67:3300/0 4 ==== config(28 keys) ==== 1170+0+0 (secure 0 0 0) 0x7fa1ec0b3b40 con 0x555945de9060 2024-05-03T21:20:59.134 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:59.128+0000 7fa1d9ffb700 1 -- 172.21.15.67:0/2514444112 <== mon.0 v2:172.21.15.67:3300/0 5 ==== fsmap(e 25) ==== 8338+0+0 (secure 0 0 0) 0x7fa1ec011070 con 0x555945de9060 2024-05-03T21:20:59.134 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:59.128+0000 7fa20333bb80 10 client.14982 fetch_fsmap finished waiting for FSMap version 25 2024-05-03T21:20:59.134 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:59.128+0000 7fa20333bb80 10 client.14982 resolve_mds: resolved name 'h' to daemon mds.h 2024-05-03T21:20:59.135 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:59.128+0000 7fa20333bb80 20 client.14982 populate_metadata read hostname 'smithi067' 2024-05-03T21:20:59.135 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:59.128+0000 7fa20333bb80 1 --2- 172.21.15.67:0/2514444112 >> [v2:172.21.15.82:6836/1275987852,v1:172.21.15.82:6837/1275987852] conn(0x555945dfd2b0 0x555945dfe720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:20:59.135 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:59.128+0000 7fa20333bb80 4 client.14982 mds_command: new command op to 24449 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-03T21:20:59.135 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:59.128+0000 7fa20333bb80 1 -- 172.21.15.67:0/2514444112 --> [v2:172.21.15.82:6836/1275987852,v1:172.21.15.82:6837/1275987852] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x555945d442d0 con 0x555945dfd2b0 2024-05-03T21:20:59.135 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:59.128+0000 7fa1f259c700 1 --2- 172.21.15.67:0/2514444112 >> [v2:172.21.15.82:6836/1275987852,v1:172.21.15.82:6837/1275987852] conn(0x555945dfd2b0 0x555945dfe720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:20:59.135 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:59.129+0000 7fa1d9ffb700 10 client.14982 ms_handle_connect on v2:172.21.15.82:6836/1275987852 2024-05-03T21:20:59.135 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:59.129+0000 7fa1f259c700 1 --2- 172.21.15.67:0/2514444112 >> [v2:172.21.15.82:6836/1275987852,v1:172.21.15.82:6837/1275987852] conn(0x555945dfd2b0 0x555945dfe720 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=24a46b513a72214c server_cookie=f0e24d816f17f88 in_seq=0 out_seq=0 2024-05-03T21:20:59.136 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:59.129+0000 7fa1d9ffb700 1 -- 172.21.15.67:0/2514444112 <== mds.1 v2:172.21.15.82:6836/1275987852 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x555945d442d0 con 0x555945dfd2b0 2024-05-03T21:20:59.136 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:59.129+0000 7fa1d9ffb700 10 client.14982 handle_command_reply: tid=0 2024-05-03T21:20:59.136 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:59.129+0000 7fa20333bb80 1 client.14982 shutdown 2024-05-03T21:20:59.136 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:59.129+0000 7fa20333bb80 1 -- 172.21.15.67:0/2514444112 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x555945de9060 msgr2=0x555945ded740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:20:59.137 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:59.129+0000 7fa20333bb80 1 --2- 172.21.15.67:0/2514444112 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x555945de9060 0x555945ded740 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7fa1ec004430 tx=0x7fa1ec0af1a0 comp rx=0 tx=0).stop 2024-05-03T21:20:59.137 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:59.130+0000 7fa20333bb80 1 -- 172.21.15.67:0/2514444112 shutdown_connections 2024-05-03T21:20:59.137 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:59.130+0000 7fa20333bb80 1 --2- 172.21.15.67:0/2514444112 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x555945de9060 0x555945ded740 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:20:59.137 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:59.130+0000 7fa20333bb80 1 --2- 172.21.15.67:0/2514444112 >> [v2:172.21.15.82:6836/1275987852,v1:172.21.15.82:6837/1275987852] conn(0x555945dfd2b0 0x555945dfe720 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:20:59.137 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:59.130+0000 7fa20333bb80 1 --2- 172.21.15.67:0/2514444112 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x555945de4480 0x555945defdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:20:59.137 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:59.130+0000 7fa20333bb80 1 --2- 172.21.15.67:0/2514444112 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x555945de8740 0x555945ded200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:20:59.137 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:59.130+0000 7fa20333bb80 1 -- 172.21.15.67:0/2514444112 >> 172.21.15.67:0/2514444112 conn(0x555945d10b20 msgr2=0x555945de9d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T21:20:59.137 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:59.130+0000 7fa1d9ffb700 0 client.14982 ms_handle_reset on v2:172.21.15.82:6836/1275987852 2024-05-03T21:20:59.137 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:59.130+0000 7fa20333bb80 1 -- 172.21.15.67:0/2514444112 shutdown_connections 2024-05-03T21:20:59.137 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:59.130+0000 7fa20333bb80 1 -- 172.21.15.67:0/2514444112 wait complete. 2024-05-03T21:20:59.137 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:59.130+0000 7fa20333bb80 20 client.14982 trim_cache size 0 max 16384 2024-05-03T21:20:59.161 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:59.151+0000 7fa20333bb80 1 Processor -- start 2024-05-03T21:20:59.162 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:59.151+0000 7fa20333bb80 1 -- start start 2024-05-03T21:20:59.162 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:59.151+0000 7fa20333bb80 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x555945de4480 0x555945de7ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:20:59.162 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:59.151+0000 7fa20333bb80 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x555945de8740 0x555945dde810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:20:59.162 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:59.151+0000 7fa20333bb80 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x555945de9060 0x555945de7380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:20:59.162 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:59.151+0000 7fa20333bb80 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x555945dfca60 con 0x555945de9060 2024-05-03T21:20:59.162 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:59.151+0000 7fa20333bb80 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x555945dfd130 con 0x555945de4480 2024-05-03T21:20:59.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:59.151+0000 7fa20333bb80 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x555945dfd800 con 0x555945de8740 2024-05-03T21:20:59.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:59.151+0000 7fa1f2d9d700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x555945de9060 0x555945de7380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:20:59.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:59.151+0000 7fa1f2d9d700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x555945de9060 0x555945de7380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:34964/0 (socket says 172.21.15.67:34964) 2024-05-03T21:20:59.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:59.151+0000 7fa1f2d9d700 1 -- 172.21.15.67:0/166442160 learned_addr learned my addr 172.21.15.67:0/166442160 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-03T21:20:59.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:59.151+0000 7fa1f259c700 1 --2- 172.21.15.67:0/166442160 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x555945de4480 0x555945de7ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:20:59.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:59.151+0000 7fa1f2d9d700 1 -- 172.21.15.67:0/166442160 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x555945de8740 msgr2=0x555945dde810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-03T21:20:59.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:59.151+0000 7fa1f2d9d700 1 --2- 172.21.15.67:0/166442160 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x555945de8740 0x555945dde810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:20:59.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:59.151+0000 7fa1f2d9d700 1 -- 172.21.15.67:0/166442160 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x555945de4480 msgr2=0x555945de7ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-03T21:20:59.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:59.151+0000 7fa1f2d9d700 1 --2- 172.21.15.67:0/166442160 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x555945de4480 0x555945de7ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:20:59.164 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:59.151+0000 7fa1f2d9d700 1 -- 172.21.15.67:0/166442160 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555945df0960 con 0x555945de9060 2024-05-03T21:20:59.164 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:59.151+0000 7fa1f2d9d700 1 --2- 172.21.15.67:0/166442160 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x555945de9060 0x555945de7380 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7fa1ec009e60 tx=0x7fa1ec0e1630 comp rx=0 tx=0).ready entity=mon.0 client_cookie=38a8b3ab34b56f7e server_cookie=0 in_seq=0 out_seq=0 2024-05-03T21:20:59.164 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:59.152+0000 7fa1dbfff700 1 -- 172.21.15.67:0/166442160 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1ec0ab240 con 0x555945de9060 2024-05-03T21:20:59.164 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:59.152+0000 7fa1dbfff700 1 -- 172.21.15.67:0/166442160 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(28 keys) ==== 1170+0+0 (secure 0 0 0) 0x7fa1ec0b3b40 con 0x555945de9060 2024-05-03T21:20:59.164 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:59.152+0000 7fa1dbfff700 1 -- 172.21.15.67:0/166442160 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1ec0a7000 con 0x555945de9060 2024-05-03T21:20:59.164 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:59.152+0000 7fa20333bb80 1 -- 172.21.15.67:0/166442160 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x555945de9060 msgr2=0x555945de7380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:20:59.164 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:59.152+0000 7fa20333bb80 1 --2- 172.21.15.67:0/166442160 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x555945de9060 0x555945de7380 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7fa1ec009e60 tx=0x7fa1ec0e1630 comp rx=0 tx=0).stop 2024-05-03T21:20:59.164 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:59.152+0000 7fa20333bb80 1 -- 172.21.15.67:0/166442160 shutdown_connections 2024-05-03T21:20:59.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:59.152+0000 7fa20333bb80 1 --2- 172.21.15.67:0/166442160 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x555945de9060 0x555945de7380 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:20:59.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:59.152+0000 7fa20333bb80 1 --2- 172.21.15.67:0/166442160 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x555945de4480 0x555945de7ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:20:59.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:59.152+0000 7fa20333bb80 1 --2- 172.21.15.67:0/166442160 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x555945de8740 0x555945dde810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:20:59.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:59.152+0000 7fa20333bb80 1 -- 172.21.15.67:0/166442160 >> 172.21.15.67:0/166442160 conn(0x555945d10b20 msgr2=0x555945dfc2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T21:20:59.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:59.152+0000 7fa20333bb80 1 -- 172.21.15.67:0/166442160 shutdown_connections 2024-05-03T21:20:59.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:59.153+0000 7fa20333bb80 1 -- 172.21.15.67:0/166442160 wait complete. 2024-05-03T21:20:59.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:59.153+0000 7fa20333bb80 1 -- start start 2024-05-03T21:20:59.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:59.153+0000 7fa20333bb80 1 Processor -- start 2024-05-03T21:20:59.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:59.153+0000 7fa20333bb80 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x555945de4480 0x555945e07f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:20:59.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:59.153+0000 7fa20333bb80 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x555945de8740 0x555945e01370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:20:59.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:59.153+0000 7fa20333bb80 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x555945de9060 0x555945e018b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:20:59.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:59.153+0000 7fa20333bb80 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x555945dfe5a0 con 0x555945de8740 2024-05-03T21:20:59.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:59.153+0000 7fa20333bb80 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x555945dfded0 con 0x555945de4480 2024-05-03T21:20:59.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:59.153+0000 7fa20333bb80 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x555945dfec70 con 0x555945de9060 2024-05-03T21:20:59.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:59.153+0000 7fa20333bb80 10 client.0 _ll_register_callbacks cb 0x555945cca560 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-03T21:20:59.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:59.153+0000 7fa1ebfff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x555945de8740 0x555945e01370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:20:59.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:59.153+0000 7fa1f259c700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x555945de4480 0x555945e07f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:20:59.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:59.153+0000 7fa1f2d9d700 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x555945de9060 0x555945e018b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:20:59.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:59.153+0000 7fa1f259c700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x555945de4480 0x555945e07f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.67:36028/0 (socket says 172.21.15.67:36028) 2024-05-03T21:20:59.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:59.153+0000 7fa1ebfff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x555945de8740 0x555945e01370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:34980/0 (socket says 172.21.15.67:34980) 2024-05-03T21:20:59.167 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:59.153+0000 7fa1f259c700 1 -- 172.21.15.67:0/3664852183 learned_addr learned my addr 172.21.15.67:0/3664852183 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-03T21:20:59.167 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:59.154+0000 7fa1f259c700 1 -- 172.21.15.67:0/3664852183 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x555945de9060 msgr2=0x555945e018b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-03T21:20:59.167 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:59.154+0000 7fa1f259c700 1 --2- 172.21.15.67:0/3664852183 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x555945de9060 0x555945e018b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:20:59.167 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:59.154+0000 7fa1f259c700 1 -- 172.21.15.67:0/3664852183 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x555945de8740 msgr2=0x555945e01370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-03T21:20:59.167 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:59.154+0000 7fa1f259c700 1 --2- 172.21.15.67:0/3664852183 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x555945de8740 0x555945e01370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:20:59.167 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:59.154+0000 7fa1f259c700 1 -- 172.21.15.67:0/3664852183 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555945e01f80 con 0x555945de4480 2024-05-03T21:20:59.167 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:59.154+0000 7fa1ebfff700 1 --2- 172.21.15.67:0/3664852183 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x555945de8740 0x555945e01370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-03T21:20:59.167 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:59.154+0000 7fa1f2d9d700 1 --2- 172.21.15.67:0/3664852183 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x555945de9060 0x555945e018b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-03T21:20:59.167 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:59.154+0000 7fa1f259c700 1 --2- 172.21.15.67:0/3664852183 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x555945de4480 0x555945e07f70 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fa1e400ec30 tx=0x7fa1e4004860 comp rx=0 tx=0).ready entity=mon.2 client_cookie=27c999df2a044392 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T21:20:59.167 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:59.154+0000 7fa1d9ffb700 10 client.0 ms_handle_connect on v2:172.21.15.82:3300/0 2024-05-03T21:20:59.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:59.154+0000 7fa1d9ffb700 1 -- 172.21.15.67:0/3664852183 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1e40157d0 con 0x555945de4480 2024-05-03T21:20:59.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:59.154+0000 7fa20333bb80 1 -- 172.21.15.67:0/3664852183 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa1ec0b3b40 con 0x555945de4480 2024-05-03T21:20:59.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:59.154+0000 7fa1d9ffb700 1 -- 172.21.15.67:0/3664852183 <== mon.2 v2:172.21.15.82:3300/0 2 ==== osd_map(84..84 src has 1..84) ==== 8211+0+0 (secure 0 0 0) 0x7fa1e4015970 con 0x555945de4480 2024-05-03T21:20:59.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:59.154+0000 7fa1d9ffb700 1 -- 172.21.15.67:0/3664852183 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7fa1e4015b70 con 0x555945de4480 2024-05-03T21:20:59.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:59.154+0000 7fa20333bb80 10 client.24749 fetch_fsmap learned FSMap version 25 2024-05-03T21:20:59.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:59.154+0000 7fa20333bb80 1 -- 172.21.15.67:0/3664852183 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x555945e68720 con 0x555945de4480 2024-05-03T21:20:59.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:59.155+0000 7fa1d9ffb700 1 -- 172.21.15.67:0/3664852183 <== mon.2 v2:172.21.15.82:3300/0 4 ==== config(28 keys) ==== 1170+0+0 (secure 0 0 0) 0x7fa1ec0b3b40 con 0x555945de4480 2024-05-03T21:20:59.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:59.155+0000 7fa1d9ffb700 1 -- 172.21.15.67:0/3664852183 <== mon.2 v2:172.21.15.82:3300/0 5 ==== fsmap(e 25) ==== 8338+0+0 (secure 0 0 0) 0x7fa1e4037020 con 0x555945de4480 2024-05-03T21:20:59.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:59.155+0000 7fa20333bb80 10 client.24749 fetch_fsmap finished waiting for FSMap version 25 2024-05-03T21:20:59.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:59.155+0000 7fa20333bb80 10 client.24749 resolve_mds: resolved name 'h' to daemon mds.h 2024-05-03T21:20:59.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:59.155+0000 7fa20333bb80 20 client.24749 populate_metadata read hostname 'smithi067' 2024-05-03T21:20:59.169 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:59.155+0000 7fa20333bb80 1 --2- 172.21.15.67:0/3664852183 >> [v2:172.21.15.82:6836/1275987852,v1:172.21.15.82:6837/1275987852] conn(0x555945e6d230 0x555945e6bf40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:20:59.169 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:59.155+0000 7fa20333bb80 4 client.24749 mds_command: new command op to 24449 tid=0 multi_id=0 [{"prefix": "status"}] 2024-05-03T21:20:59.169 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:59.155+0000 7fa20333bb80 1 -- 172.21.15.67:0/3664852183 --> [v2:172.21.15.82:6836/1275987852,v1:172.21.15.82:6837/1275987852] -- command(tid 0: {"prefix": "status"}) -- 0x555945d3e370 con 0x555945e6d230 2024-05-03T21:20:59.169 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:59.155+0000 7fa1ebfff700 1 --2- 172.21.15.67:0/3664852183 >> [v2:172.21.15.82:6836/1275987852,v1:172.21.15.82:6837/1275987852] conn(0x555945e6d230 0x555945e6bf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:20:59.169 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:59.156+0000 7fa1ebfff700 1 --2- 172.21.15.67:0/3664852183 >> [v2:172.21.15.82:6836/1275987852,v1:172.21.15.82:6837/1275987852] conn(0x555945e6d230 0x555945e6bf40 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=4a0b0a448d7357d9 server_cookie=18fb5602d5aac125 in_seq=0 out_seq=0 2024-05-03T21:20:59.169 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:59.156+0000 7fa1d9ffb700 10 client.24749 ms_handle_connect on v2:172.21.15.82:6836/1275987852 2024-05-03T21:20:59.169 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:59.156+0000 7fa1d9ffb700 1 -- 172.21.15.67:0/3664852183 <== mds.1 v2:172.21.15.82:6836/1275987852 1 ==== command_reply(tid 0: 0 ) ==== 8+0+332 (crc 0 0 0) 0x555945d3e370 con 0x555945e6d230 2024-05-03T21:20:59.169 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:59.156+0000 7fa1d9ffb700 10 client.24749 handle_command_reply: tid=0 2024-05-03T21:20:59.169 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:59.156+0000 7fa20333bb80 1 client.24749 shutdown 2024-05-03T21:20:59.169 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:59.156+0000 7fa20333bb80 1 -- 172.21.15.67:0/3664852183 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x555945de4480 msgr2=0x555945e07f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:20:59.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:59.156+0000 7fa20333bb80 1 --2- 172.21.15.67:0/3664852183 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x555945de4480 0x555945e07f70 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fa1e400ec30 tx=0x7fa1e4004860 comp rx=0 tx=0).stop 2024-05-03T21:20:59.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:59.156+0000 7fa20333bb80 1 -- 172.21.15.67:0/3664852183 shutdown_connections 2024-05-03T21:20:59.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:59.156+0000 7fa20333bb80 1 --2- 172.21.15.67:0/3664852183 >> [v2:172.21.15.82:6836/1275987852,v1:172.21.15.82:6837/1275987852] conn(0x555945e6d230 0x555945e6bf40 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:20:59.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:59.156+0000 7fa20333bb80 1 --2- 172.21.15.67:0/3664852183 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x555945de9060 0x555945e018b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:20:59.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:59.156+0000 7fa20333bb80 1 --2- 172.21.15.67:0/3664852183 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x555945de4480 0x555945e07f70 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:20:59.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:59.156+0000 7fa1d9ffb700 0 client.24749 ms_handle_reset on v2:172.21.15.82:6836/1275987852 2024-05-03T21:20:59.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:59.156+0000 7fa20333bb80 1 --2- 172.21.15.67:0/3664852183 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x555945de8740 0x555945e01370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:20:59.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:59.156+0000 7fa20333bb80 1 -- 172.21.15.67:0/3664852183 >> 172.21.15.67:0/3664852183 conn(0x555945d10b20 msgr2=0x555945de7380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T21:20:59.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:59.156+0000 7fa20333bb80 1 -- 172.21.15.67:0/3664852183 shutdown_connections 2024-05-03T21:20:59.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:59.156+0000 7fa20333bb80 1 -- 172.21.15.67:0/3664852183 wait complete. 2024-05-03T21:20:59.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:59.156+0000 7fa20333bb80 20 client.24749 trim_cache size 0 max 16384 2024-05-03T21:20:59.171 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-05-03T21:20:59.171 INFO:teuthology.orchestra.run.smithi067.stdout: "cluster_fsid": "713bc57a-0991-11ef-bc97-c7b262605968", 2024-05-03T21:20:59.171 INFO:teuthology.orchestra.run.smithi067.stdout: "whoami": 1, 2024-05-03T21:20:59.171 INFO:teuthology.orchestra.run.smithi067.stdout: "id": 24449, 2024-05-03T21:20:59.171 INFO:teuthology.orchestra.run.smithi067.stdout: "want_state": "up:active", 2024-05-03T21:20:59.171 INFO:teuthology.orchestra.run.smithi067.stdout: "state": "up:active", 2024-05-03T21:20:59.171 INFO:teuthology.orchestra.run.smithi067.stdout: "fs_name": "cephfs", 2024-05-03T21:20:59.172 INFO:teuthology.orchestra.run.smithi067.stdout: "rank_uptime": 122.30191708300001, 2024-05-03T21:20:59.172 INFO:teuthology.orchestra.run.smithi067.stdout: "mdsmap_epoch": 25, 2024-05-03T21:20:59.172 INFO:teuthology.orchestra.run.smithi067.stdout: "osdmap_epoch": 84, 2024-05-03T21:20:59.172 INFO:teuthology.orchestra.run.smithi067.stdout: "osdmap_epoch_barrier": 82, 2024-05-03T21:20:59.172 INFO:teuthology.orchestra.run.smithi067.stdout: "uptime": 161.71904422899999 2024-05-03T21:20:59.172 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-05-03T21:20:59.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:59.157+0000 7fa1f8aa8700 1 -- 172.21.15.67:0/1983788611 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7fa1e00b2990 msgr2=0x7fa1e00b4e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:20:59.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:59.157+0000 7fa1f8aa8700 1 --2- 172.21.15.67:0/1983788611 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7fa1e00b2990 0x7fa1e00b4e50 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fa1e4009a10 tx=0x7fa1e4009190 comp rx=0 tx=0).stop 2024-05-03T21:20:59.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:59.157+0000 7fa1f8aa8700 1 -- 172.21.15.67:0/1983788611 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa1f4109510 msgr2=0x7fa1f41a0980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:20:59.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:59.157+0000 7fa1f8aa8700 1 --2- 172.21.15.67:0/1983788611 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa1f4109510 0x7fa1f41a0980 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7fa1ec006070 tx=0x7fa1ec004e20 comp rx=0 tx=0).stop 2024-05-03T21:20:59.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:59.157+0000 7fa1f8aa8700 1 -- 172.21.15.67:0/1983788611 shutdown_connections 2024-05-03T21:20:59.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:59.157+0000 7fa1f8aa8700 1 --2- 172.21.15.67:0/1983788611 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7fa1e00b2990 0x7fa1e00b4e50 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:20:59.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:59.157+0000 7fa1f8aa8700 1 --2- 172.21.15.67:0/1983788611 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa1f4109510 0x7fa1f41a0980 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:20:59.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:59.157+0000 7fa1f8aa8700 1 --2- 172.21.15.67:0/1983788611 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fa1f406b180 0x7fa1f419c100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:20:59.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:59.157+0000 7fa1f8aa8700 1 --2- 172.21.15.67:0/1983788611 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fa1f406bb40 0x7fa1f419c640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:20:59.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:59.157+0000 7fa1f8aa8700 1 -- 172.21.15.67:0/1983788611 >> 172.21.15.67:0/1983788611 conn(0x7fa1f4079350 msgr2=0x7fa1f4100d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T21:20:59.174 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:59.157+0000 7fa1f8aa8700 1 -- 172.21.15.67:0/1983788611 shutdown_connections 2024-05-03T21:20:59.174 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:20:59.157+0000 7fa1f8aa8700 1 -- 172.21.15.67:0/1983788611 wait complete. 2024-05-03T21:20:59.471 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:20:59 smithi067 ceph-mon[29730]: pgmap v356: 97 pgs: 97 active+clean; 632 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:20:59.495 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:20:59 smithi082 ceph-mon[32933]: pgmap v356: 97 pgs: 97 active+clean; 632 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:20:59.574 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:20:59 smithi138 ceph-mon[33110]: pgmap v356: 97 pgs: 97 active+clean; 632 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:20:59.890 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9a047fe7599a549042d06a3fb5b6813a203eabea shell --fsid 713bc57a-0991-11ef-bc97-c7b262605968 -- ceph tell mds.b status 2024-05-03T21:21:00.140 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/713bc57a-0991-11ef-bc97-c7b262605968/mon.a/config 2024-05-03T21:21:01.221 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:21:01 smithi067 ceph-mon[29730]: pgmap v357: 97 pgs: 97 active+clean; 632 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T21:21:01.496 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:21:01 smithi082 ceph-mon[32933]: pgmap v357: 97 pgs: 97 active+clean; 632 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T21:21:01.574 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:21:01 smithi138 ceph-mon[33110]: pgmap v357: 97 pgs: 97 active+clean; 632 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T21:21:01.776 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:01.773+0000 7f7dc8fed700 1 -- 172.21.15.67:0/3152610085 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7dc4102fe0 msgr2=0x7f7dc4106fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:21:01.776 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:01.773+0000 7f7dc8fed700 1 --2- 172.21.15.67:0/3152610085 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7dc4102fe0 0x7f7dc4106fe0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f7db80099c0 tx=0x7f7db8009cd0 comp rx=0 tx=0).stop 2024-05-03T21:21:01.776 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:01.773+0000 7f7dc8fed700 1 -- 172.21.15.67:0/3152610085 shutdown_connections 2024-05-03T21:21:01.776 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:01.773+0000 7f7dc8fed700 1 --2- 172.21.15.67:0/3152610085 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f7dc41076c0 0x7f7dc410f290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:21:01.776 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:01.773+0000 7f7dc8fed700 1 --2- 172.21.15.67:0/3152610085 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f7dc4102630 0x7f7dc4102a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:21:01.776 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:01.773+0000 7f7dc8fed700 1 --2- 172.21.15.67:0/3152610085 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7dc4102fe0 0x7f7dc4106fe0 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:21:01.776 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:01.773+0000 7f7dc8fed700 1 -- 172.21.15.67:0/3152610085 >> 172.21.15.67:0/3152610085 conn(0x7f7dc4079350 msgr2=0x7f7dc4079760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T21:21:01.776 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:01.773+0000 7f7dc8fed700 1 -- 172.21.15.67:0/3152610085 shutdown_connections 2024-05-03T21:21:01.777 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:01.773+0000 7f7dc8fed700 1 -- 172.21.15.67:0/3152610085 wait complete. 2024-05-03T21:21:01.777 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:01.774+0000 7f7dc8fed700 1 Processor -- start 2024-05-03T21:21:01.777 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:01.774+0000 7f7dc8fed700 1 -- start start 2024-05-03T21:21:01.779 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:01.774+0000 7f7dc8fed700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f7dc4102630 0x7f7dc4199d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:21:01.779 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:01.774+0000 7f7dc8fed700 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f7dc4102fe0 0x7f7dc419a2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:21:01.779 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:01.774+0000 7f7dc8fed700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7dc41076c0 0x7f7dc419e5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:21:01.779 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:01.774+0000 7f7dc8fed700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f7dc41138e0 con 0x7f7dc41076c0 2024-05-03T21:21:01.779 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:01.774+0000 7f7dc8fed700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f7dc4113760 con 0x7f7dc4102630 2024-05-03T21:21:01.779 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:01.774+0000 7f7dc8fed700 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f7dc4113a60 con 0x7f7dc4102fe0 2024-05-03T21:21:01.779 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:01.774+0000 7f7dc2d9d700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7dc41076c0 0x7f7dc419e5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:21:01.779 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:01.774+0000 7f7dc2d9d700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7dc41076c0 0x7f7dc419e5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:34996/0 (socket says 172.21.15.67:34996) 2024-05-03T21:21:01.780 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:01.774+0000 7f7dc2d9d700 1 -- 172.21.15.67:0/3645689064 learned_addr learned my addr 172.21.15.67:0/3645689064 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-03T21:21:01.780 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:01.774+0000 7f7dc1d9b700 1 --2- 172.21.15.67:0/3645689064 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f7dc4102fe0 0x7f7dc419a2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:21:01.780 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:01.774+0000 7f7dc259c700 1 --2- 172.21.15.67:0/3645689064 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f7dc4102630 0x7f7dc4199d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:21:01.780 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:01.775+0000 7f7dc2d9d700 1 -- 172.21.15.67:0/3645689064 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f7dc4102fe0 msgr2=0x7f7dc419a2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:21:01.780 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:01.775+0000 7f7dc2d9d700 1 --2- 172.21.15.67:0/3645689064 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f7dc4102fe0 0x7f7dc419a2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:21:01.780 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:01.775+0000 7f7dc2d9d700 1 -- 172.21.15.67:0/3645689064 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f7dc4102630 msgr2=0x7f7dc4199d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:21:01.780 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:01.775+0000 7f7dc2d9d700 1 --2- 172.21.15.67:0/3645689064 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f7dc4102630 0x7f7dc4199d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:21:01.780 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:01.775+0000 7f7dc2d9d700 1 -- 172.21.15.67:0/3645689064 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7dc419ed30 con 0x7f7dc41076c0 2024-05-03T21:21:01.780 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:01.775+0000 7f7dc2d9d700 1 --2- 172.21.15.67:0/3645689064 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7dc41076c0 0x7f7dc419e5f0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f7dbc007ee0 tx=0x7f7dbc00d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T21:21:01.780 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:01.775+0000 7f7db37fe700 1 -- 172.21.15.67:0/3645689064 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7dbc004bb0 con 0x7f7dc41076c0 2024-05-03T21:21:01.780 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:01.775+0000 7f7dc8fed700 1 -- 172.21.15.67:0/3645689064 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7dc419f020 con 0x7f7dc41076c0 2024-05-03T21:21:01.780 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:01.775+0000 7f7db37fe700 1 -- 172.21.15.67:0/3645689064 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(28 keys) ==== 1170+0+0 (secure 0 0 0) 0x7f7dbc004d50 con 0x7f7dc41076c0 2024-05-03T21:21:01.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:01.775+0000 7f7db37fe700 1 -- 172.21.15.67:0/3645689064 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7dbc0059b0 con 0x7f7dc41076c0 2024-05-03T21:21:01.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:01.775+0000 7f7dc8fed700 1 -- 172.21.15.67:0/3645689064 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7dc410f9a0 con 0x7f7dc41076c0 2024-05-03T21:21:01.783 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:01.778+0000 7f7db37fe700 1 -- 172.21.15.67:0/3645689064 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 17) ==== 149703+0+0 (secure 0 0 0) 0x7f7dbc01a020 con 0x7f7dc41076c0 2024-05-03T21:21:01.784 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:01.778+0000 7f7db37fe700 1 --2- 172.21.15.67:0/3645689064 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f7dac0b2910 0x7f7dac0b4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:21:01.784 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:01.778+0000 7f7dc259c700 1 --2- 172.21.15.67:0/3645689064 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f7dac0b2910 0x7f7dac0b4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:21:01.784 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:01.778+0000 7f7db37fe700 1 -- 172.21.15.67:0/3645689064 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 8211+0+0 (secure 0 0 0) 0x7f7dbc0e1a00 con 0x7f7dc41076c0 2024-05-03T21:21:01.784 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:01.779+0000 7f7dd3880b80 1 Processor -- start 2024-05-03T21:21:01.784 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:01.779+0000 7f7dc259c700 1 --2- 172.21.15.67:0/3645689064 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f7dac0b2910 0x7f7dac0b4dd0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f7db4004630 tx=0x7f7db400b3c0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T21:21:01.784 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:01.779+0000 7f7dd3880b80 1 -- start start 2024-05-03T21:21:01.784 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:01.779+0000 7f7dd3880b80 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b4c3d28740 0x55b4c3d28b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:21:01.784 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:01.779+0000 7f7dd3880b80 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55b4c3d29060 0x55b4c3d23f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:21:01.784 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:01.779+0000 7f7dd3880b80 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55b4c3d24480 0x55b4c3d248c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:21:01.785 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:01.779+0000 7f7dd3880b80 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x55b4c3c768f0 con 0x55b4c3d28740 2024-05-03T21:21:01.785 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:01.779+0000 7f7dd3880b80 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x55b4c3bdbd50 con 0x55b4c3d29060 2024-05-03T21:21:01.785 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:01.779+0000 7f7dd3880b80 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x55b4c3c728b0 con 0x55b4c3d24480 2024-05-03T21:21:01.785 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:01.779+0000 7f7dc259c700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b4c3d28740 0x55b4c3d28b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:21:01.785 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:01.779+0000 7f7dc259c700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b4c3d28740 0x55b4c3d28b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:35000/0 (socket says 172.21.15.67:35000) 2024-05-03T21:21:01.785 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:01.779+0000 7f7dc259c700 1 -- 172.21.15.67:0/3139838303 learned_addr learned my addr 172.21.15.67:0/3139838303 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-03T21:21:01.785 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:01.780+0000 7f7dc2d9d700 1 --2- 172.21.15.67:0/3139838303 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55b4c3d24480 0x55b4c3d248c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:21:01.785 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:01.780+0000 7f7dc1d9b700 1 --2- 172.21.15.67:0/3139838303 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55b4c3d29060 0x55b4c3d23f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:21:01.785 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:01.780+0000 7f7dc259c700 1 -- 172.21.15.67:0/3139838303 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55b4c3d24480 msgr2=0x55b4c3d248c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-03T21:21:01.785 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:01.780+0000 7f7dc259c700 1 --2- 172.21.15.67:0/3139838303 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55b4c3d24480 0x55b4c3d248c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:21:01.785 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:01.780+0000 7f7dc259c700 1 -- 172.21.15.67:0/3139838303 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55b4c3d29060 msgr2=0x55b4c3d23f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-03T21:21:01.785 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:01.780+0000 7f7dc259c700 1 --2- 172.21.15.67:0/3139838303 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55b4c3d29060 0x55b4c3d23f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:21:01.785 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:01.780+0000 7f7dc259c700 1 -- 172.21.15.67:0/3139838303 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b4c3bc3880 con 0x55b4c3d28740 2024-05-03T21:21:01.785 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:01.780+0000 7f7dc2d9d700 1 --2- 172.21.15.67:0/3139838303 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55b4c3d24480 0x55b4c3d248c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-03T21:21:01.786 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:01.780+0000 7f7dc259c700 1 --2- 172.21.15.67:0/3139838303 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b4c3d28740 0x55b4c3d28b20 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f7db4003b10 tx=0x7f7db4013000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e12d61164fd00b78 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T21:21:01.786 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:01.780+0000 7f7db0ff9700 1 -- 172.21.15.67:0/3139838303 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7db4011ad0 con 0x55b4c3d28740 2024-05-03T21:21:01.786 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:01.780+0000 7f7db0ff9700 1 -- 172.21.15.67:0/3139838303 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(28 keys) ==== 1170+0+0 (secure 0 0 0) 0x7f7db4011c70 con 0x55b4c3d28740 2024-05-03T21:21:01.786 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:01.780+0000 7f7db0ff9700 1 -- 172.21.15.67:0/3139838303 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7db401c610 con 0x55b4c3d28740 2024-05-03T21:21:01.786 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:01.781+0000 7f7dd3880b80 1 -- 172.21.15.67:0/3139838303 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b4c3d28740 msgr2=0x55b4c3d28b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:21:01.786 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:01.781+0000 7f7dd3880b80 1 --2- 172.21.15.67:0/3139838303 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b4c3d28740 0x55b4c3d28b20 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f7db4003b10 tx=0x7f7db4013000 comp rx=0 tx=0).stop 2024-05-03T21:21:01.786 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:01.781+0000 7f7dd3880b80 1 -- 172.21.15.67:0/3139838303 shutdown_connections 2024-05-03T21:21:01.786 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:01.781+0000 7f7dd3880b80 1 --2- 172.21.15.67:0/3139838303 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b4c3d28740 0x55b4c3d28b20 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:21:01.786 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:01.781+0000 7f7dd3880b80 1 --2- 172.21.15.67:0/3139838303 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55b4c3d24480 0x55b4c3d248c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:21:01.786 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:01.781+0000 7f7dd3880b80 1 --2- 172.21.15.67:0/3139838303 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55b4c3d29060 0x55b4c3d23f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:21:01.786 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:01.781+0000 7f7dd3880b80 1 -- 172.21.15.67:0/3139838303 >> 172.21.15.67:0/3139838303 conn(0x55b4c3c50b20 msgr2=0x55b4c3d1e9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T21:21:01.786 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:01.781+0000 7f7dd3880b80 1 -- 172.21.15.67:0/3139838303 shutdown_connections 2024-05-03T21:21:01.786 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:01.781+0000 7f7dd3880b80 1 -- 172.21.15.67:0/3139838303 wait complete. 2024-05-03T21:21:01.786 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:01.781+0000 7f7dd3880b80 1 -- start start 2024-05-03T21:21:01.787 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:01.782+0000 7f7dd3880b80 1 Processor -- start 2024-05-03T21:21:01.790 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:01.782+0000 7f7dd3880b80 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55b4c3d24480 0x55b4c3d2fd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:21:01.790 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:01.782+0000 7f7dd3880b80 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b4c3d28740 0x55b4c3d2d1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:21:01.790 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:01.782+0000 7f7dd3880b80 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55b4c3d29060 0x55b4c3d2d730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:21:01.790 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:01.782+0000 7f7dd3880b80 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x55b4c3d30ba0 con 0x55b4c3d28740 2024-05-03T21:21:01.790 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:01.782+0000 7f7dd3880b80 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x55b4c3c2fe80 con 0x55b4c3d29060 2024-05-03T21:21:01.790 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:01.782+0000 7f7dd3880b80 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x55b4c3d30d20 con 0x55b4c3d24480 2024-05-03T21:21:01.790 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:01.782+0000 7f7dc1d9b700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b4c3d28740 0x55b4c3d2d1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:21:01.791 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:01.782+0000 7f7dc1d9b700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b4c3d28740 0x55b4c3d2d1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:35016/0 (socket says 172.21.15.67:35016) 2024-05-03T21:21:01.791 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:01.782+0000 7f7dc1d9b700 1 -- 172.21.15.67:0/2672632587 learned_addr learned my addr 172.21.15.67:0/2672632587 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-03T21:21:01.791 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:01.782+0000 7f7dd3880b80 10 client.0 _ll_register_callbacks cb 0x55b4c3c0a560 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-03T21:21:01.791 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:01.782+0000 7f7dc2d9d700 1 --2- 172.21.15.67:0/2672632587 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55b4c3d29060 0x55b4c3d2d730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:21:01.791 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:01.782+0000 7f7dc259c700 1 --2- 172.21.15.67:0/2672632587 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55b4c3d24480 0x55b4c3d2fd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:21:01.791 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:01.782+0000 7f7dc1d9b700 1 -- 172.21.15.67:0/2672632587 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55b4c3d24480 msgr2=0x55b4c3d2fd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-03T21:21:01.791 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:01.782+0000 7f7dc1d9b700 1 --2- 172.21.15.67:0/2672632587 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55b4c3d24480 0x55b4c3d2fd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:21:01.791 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:01.782+0000 7f7dc1d9b700 1 -- 172.21.15.67:0/2672632587 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55b4c3d29060 msgr2=0x55b4c3d2d730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-03T21:21:01.791 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:01.782+0000 7f7dc1d9b700 1 --2- 172.21.15.67:0/2672632587 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55b4c3d29060 0x55b4c3d2d730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:21:01.791 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:01.782+0000 7f7dc1d9b700 1 -- 172.21.15.67:0/2672632587 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b4c3c2d710 con 0x55b4c3d28740 2024-05-03T21:21:01.791 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:01.782+0000 7f7dc259c700 1 --2- 172.21.15.67:0/2672632587 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55b4c3d24480 0x55b4c3d2fd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-03T21:21:01.791 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:01.782+0000 7f7dc2d9d700 1 --2- 172.21.15.67:0/2672632587 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55b4c3d29060 0x55b4c3d2d730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-03T21:21:01.791 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:01.783+0000 7f7dc1d9b700 1 --2- 172.21.15.67:0/2672632587 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b4c3d28740 0x55b4c3d2d1f0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f7db80099c0 tx=0x7f7db80038b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5cf24fa35ccd15b3 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T21:21:01.792 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:01.783+0000 7f7da9ffb700 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-05-03T21:21:01.792 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:01.783+0000 7f7da9ffb700 1 -- 172.21.15.67:0/2672632587 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7db8028060 con 0x55b4c3d28740 2024-05-03T21:21:01.792 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:01.783+0000 7f7da9ffb700 1 -- 172.21.15.67:0/2672632587 <== mon.0 v2:172.21.15.67:3300/0 2 ==== osd_map(84..84 src has 1..84) ==== 8211+0+0 (secure 0 0 0) 0x7f7db80054d0 con 0x55b4c3d28740 2024-05-03T21:21:01.792 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:01.783+0000 7f7dd3880b80 1 -- 172.21.15.67:0/2672632587 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7db4011c70 con 0x55b4c3d28740 2024-05-03T21:21:01.792 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:01.783+0000 7f7da9ffb700 1 -- 172.21.15.67:0/2672632587 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f7db80058e0 con 0x55b4c3d28740 2024-05-03T21:21:01.792 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:01.783+0000 7f7dd3880b80 10 client.15000 fetch_fsmap learned FSMap version 25 2024-05-03T21:21:01.792 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:01.783+0000 7f7dd3880b80 1 -- 172.21.15.67:0/2672632587 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55b4c3d3b3c0 con 0x55b4c3d28740 2024-05-03T21:21:01.792 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:01.783+0000 7f7da9ffb700 1 -- 172.21.15.67:0/2672632587 <== mon.0 v2:172.21.15.67:3300/0 4 ==== config(28 keys) ==== 1170+0+0 (secure 0 0 0) 0x7f7db4011c70 con 0x55b4c3d28740 2024-05-03T21:21:01.792 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:01.783+0000 7f7da9ffb700 1 -- 172.21.15.67:0/2672632587 <== mon.0 v2:172.21.15.67:3300/0 5 ==== fsmap(e 25) ==== 8338+0+0 (secure 0 0 0) 0x7f7db800f8a0 con 0x55b4c3d28740 2024-05-03T21:21:01.792 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:01.783+0000 7f7dd3880b80 10 client.15000 fetch_fsmap finished waiting for FSMap version 25 2024-05-03T21:21:01.792 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:01.783+0000 7f7dd3880b80 10 client.15000 resolve_mds: resolved name 'b' to daemon mds.b 2024-05-03T21:21:01.792 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:01.783+0000 7f7dd3880b80 20 client.15000 populate_metadata read hostname 'smithi067' 2024-05-03T21:21:01.792 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:01.783+0000 7f7dd3880b80 1 --2- 172.21.15.67:0/2672632587 >> [v2:172.21.15.82:6838/3644499578,v1:172.21.15.82:6839/3644499578] conn(0x55b4c3d3da50 0x55b4c3d3eec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:21:01.793 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:01.783+0000 7f7dd3880b80 4 client.15000 mds_command: new command op to 24473 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-03T21:21:01.793 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:01.783+0000 7f7dd3880b80 1 -- 172.21.15.67:0/2672632587 --> [v2:172.21.15.82:6838/3644499578,v1:172.21.15.82:6839/3644499578] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b4c3c842d0 con 0x55b4c3d3da50 2024-05-03T21:21:01.793 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:01.784+0000 7f7dc259c700 1 --2- 172.21.15.67:0/2672632587 >> [v2:172.21.15.82:6838/3644499578,v1:172.21.15.82:6839/3644499578] conn(0x55b4c3d3da50 0x55b4c3d3eec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:21:01.793 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:01.784+0000 7f7dc259c700 1 --2- 172.21.15.67:0/2672632587 >> [v2:172.21.15.82:6838/3644499578,v1:172.21.15.82:6839/3644499578] conn(0x55b4c3d3da50 0x55b4c3d3eec0 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1837218b361c7c9 server_cookie=5017b6a4495251ce in_seq=0 out_seq=0 2024-05-03T21:21:01.793 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:01.784+0000 7f7da9ffb700 10 client.15000 ms_handle_connect on v2:172.21.15.82:6838/3644499578 2024-05-03T21:21:01.793 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:01.785+0000 7f7da9ffb700 1 -- 172.21.15.67:0/2672632587 <== mds.0 v2:172.21.15.82:6838/3644499578 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55b4c3c842d0 con 0x55b4c3d3da50 2024-05-03T21:21:01.793 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:01.785+0000 7f7da9ffb700 10 client.15000 handle_command_reply: tid=0 2024-05-03T21:21:01.793 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:01.785+0000 7f7dd3880b80 1 client.15000 shutdown 2024-05-03T21:21:01.793 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:01.785+0000 7f7dd3880b80 1 -- 172.21.15.67:0/2672632587 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b4c3d28740 msgr2=0x55b4c3d2d1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:21:01.793 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:01.785+0000 7f7dd3880b80 1 --2- 172.21.15.67:0/2672632587 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b4c3d28740 0x55b4c3d2d1f0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f7db80099c0 tx=0x7f7db80038b0 comp rx=0 tx=0).stop 2024-05-03T21:21:01.793 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:01.785+0000 7f7dd3880b80 1 -- 172.21.15.67:0/2672632587 shutdown_connections 2024-05-03T21:21:01.793 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:01.785+0000 7f7dd3880b80 1 --2- 172.21.15.67:0/2672632587 >> [v2:172.21.15.82:6838/3644499578,v1:172.21.15.82:6839/3644499578] conn(0x55b4c3d3da50 0x55b4c3d3eec0 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:21:01.794 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:01.785+0000 7f7dd3880b80 1 --2- 172.21.15.67:0/2672632587 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55b4c3d29060 0x55b4c3d2d730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:21:01.794 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:01.785+0000 7f7dd3880b80 1 --2- 172.21.15.67:0/2672632587 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55b4c3d24480 0x55b4c3d2fd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:21:01.794 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:01.785+0000 7f7dd3880b80 1 --2- 172.21.15.67:0/2672632587 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b4c3d28740 0x55b4c3d2d1f0 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:21:01.794 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:01.785+0000 7f7dd3880b80 1 -- 172.21.15.67:0/2672632587 >> 172.21.15.67:0/2672632587 conn(0x55b4c3c50b20 msgr2=0x55b4c3d1d800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T21:21:01.794 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:01.785+0000 7f7da9ffb700 0 client.15000 ms_handle_reset on v2:172.21.15.82:6838/3644499578 2024-05-03T21:21:01.794 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:01.785+0000 7f7dd3880b80 1 -- 172.21.15.67:0/2672632587 shutdown_connections 2024-05-03T21:21:01.794 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:01.785+0000 7f7dd3880b80 1 -- 172.21.15.67:0/2672632587 wait complete. 2024-05-03T21:21:01.794 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:01.785+0000 7f7dd3880b80 20 client.15000 trim_cache size 0 max 16384 2024-05-03T21:21:01.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:01.806+0000 7f7dd3880b80 1 Processor -- start 2024-05-03T21:21:01.810 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:01.806+0000 7f7dd3880b80 1 -- start start 2024-05-03T21:21:01.810 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:01.806+0000 7f7dd3880b80 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b4c3d24480 0x55b4c3d2f640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:21:01.810 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:01.806+0000 7f7dd3880b80 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55b4c3d28740 0x55b4c3d2d760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:21:01.810 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:01.806+0000 7f7dd3880b80 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55b4c3d29060 0x55b4c3d1e940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:21:01.810 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:01.806+0000 7f7dd3880b80 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x55b4c3d3d200 con 0x55b4c3d24480 2024-05-03T21:21:01.810 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:01.806+0000 7f7dd3880b80 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x55b4c3d3d8d0 con 0x55b4c3d29060 2024-05-03T21:21:01.810 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:01.806+0000 7f7dd3880b80 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x55b4c3d3dfa0 con 0x55b4c3d28740 2024-05-03T21:21:01.810 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:01.807+0000 7f7dc259c700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b4c3d24480 0x55b4c3d2f640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:21:01.810 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:01.807+0000 7f7dc259c700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b4c3d24480 0x55b4c3d2f640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:35030/0 (socket says 172.21.15.67:35030) 2024-05-03T21:21:01.810 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:01.807+0000 7f7dc259c700 1 -- 172.21.15.67:0/3741483895 learned_addr learned my addr 172.21.15.67:0/3741483895 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-03T21:21:01.810 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:01.807+0000 7f7dc2d9d700 1 --2- 172.21.15.67:0/3741483895 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55b4c3d29060 0x55b4c3d1e940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:21:01.811 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:01.807+0000 7f7dc1d9b700 1 --2- 172.21.15.67:0/3741483895 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55b4c3d28740 0x55b4c3d2d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:21:01.811 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:01.807+0000 7f7dc259c700 1 -- 172.21.15.67:0/3741483895 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55b4c3d28740 msgr2=0x55b4c3d2d760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-03T21:21:01.811 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:01.807+0000 7f7dc259c700 1 --2- 172.21.15.67:0/3741483895 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55b4c3d28740 0x55b4c3d2d760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:21:01.811 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:01.807+0000 7f7dc259c700 1 -- 172.21.15.67:0/3741483895 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55b4c3d29060 msgr2=0x55b4c3d1e940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-03T21:21:01.811 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:01.807+0000 7f7dc259c700 1 --2- 172.21.15.67:0/3741483895 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55b4c3d29060 0x55b4c3d1e940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:21:01.811 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:01.807+0000 7f7dc259c700 1 -- 172.21.15.67:0/3741483895 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b4c3d24bd0 con 0x55b4c3d24480 2024-05-03T21:21:01.811 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:01.807+0000 7f7dc1d9b700 1 --2- 172.21.15.67:0/3741483895 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55b4c3d28740 0x55b4c3d2d760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-03T21:21:01.811 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:01.807+0000 7f7dc259c700 1 --2- 172.21.15.67:0/3741483895 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b4c3d24480 0x55b4c3d2f640 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f7db4003b10 tx=0x7f7db400b450 comp rx=0 tx=0).ready entity=mon.0 client_cookie=16d21bbe9ceed9a0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T21:21:01.811 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:01.807+0000 7f7dabfff700 1 -- 172.21.15.67:0/3741483895 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7db4005050 con 0x55b4c3d24480 2024-05-03T21:21:01.812 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:01.807+0000 7f7dabfff700 1 -- 172.21.15.67:0/3741483895 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(28 keys) ==== 1170+0+0 (secure 0 0 0) 0x7f7db4004710 con 0x55b4c3d24480 2024-05-03T21:21:01.812 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:01.807+0000 7f7dabfff700 1 -- 172.21.15.67:0/3741483895 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7db401e680 con 0x55b4c3d24480 2024-05-03T21:21:01.815 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:01.808+0000 7f7dd3880b80 1 -- 172.21.15.67:0/3741483895 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b4c3d24480 msgr2=0x55b4c3d2f640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:21:01.815 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:01.808+0000 7f7dd3880b80 1 --2- 172.21.15.67:0/3741483895 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b4c3d24480 0x55b4c3d2f640 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f7db4003b10 tx=0x7f7db400b450 comp rx=0 tx=0).stop 2024-05-03T21:21:01.815 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:01.808+0000 7f7dd3880b80 1 -- 172.21.15.67:0/3741483895 shutdown_connections 2024-05-03T21:21:01.816 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:01.808+0000 7f7dd3880b80 1 --2- 172.21.15.67:0/3741483895 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b4c3d24480 0x55b4c3d2f640 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:21:01.816 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:01.808+0000 7f7dd3880b80 1 --2- 172.21.15.67:0/3741483895 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55b4c3d29060 0x55b4c3d1e940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:21:01.816 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:01.808+0000 7f7dd3880b80 1 --2- 172.21.15.67:0/3741483895 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55b4c3d28740 0x55b4c3d2d760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:21:01.816 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:01.808+0000 7f7dd3880b80 1 -- 172.21.15.67:0/3741483895 >> 172.21.15.67:0/3741483895 conn(0x55b4c3c50b20 msgr2=0x55b4c3d1b050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T21:21:01.816 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:01.808+0000 7f7dd3880b80 1 -- 172.21.15.67:0/3741483895 shutdown_connections 2024-05-03T21:21:01.816 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:01.808+0000 7f7dd3880b80 1 -- 172.21.15.67:0/3741483895 wait complete. 2024-05-03T21:21:01.816 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:01.808+0000 7f7dd3880b80 1 -- start start 2024-05-03T21:21:01.816 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:01.808+0000 7f7dd3880b80 1 Processor -- start 2024-05-03T21:21:01.816 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:01.809+0000 7f7dd3880b80 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55b4c3d24480 0x55b4c3d489e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:21:01.816 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:01.809+0000 7f7dd3880b80 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55b4c3d28740 0x55b4c3d41d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:21:01.816 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:01.809+0000 7f7dd3880b80 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b4c3d29060 0x55b4c3d42430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:21:01.816 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:01.809+0000 7f7dd3880b80 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x55b4c3d3ed40 con 0x55b4c3d29060 2024-05-03T21:21:01.816 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:01.809+0000 7f7dd3880b80 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x55b4c3d3e670 con 0x55b4c3d24480 2024-05-03T21:21:01.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:01.809+0000 7f7dd3880b80 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x55b4c3d3f410 con 0x55b4c3d28740 2024-05-03T21:21:01.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:01.809+0000 7f7dd3880b80 10 client.0 _ll_register_callbacks cb 0x55b4c3c0a560 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-03T21:21:01.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:01.809+0000 7f7dc2d9d700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b4c3d29060 0x55b4c3d42430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:21:01.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:01.809+0000 7f7dc2d9d700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b4c3d29060 0x55b4c3d42430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:35040/0 (socket says 172.21.15.67:35040) 2024-05-03T21:21:01.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:01.809+0000 7f7dc2d9d700 1 -- 172.21.15.67:0/2032768478 learned_addr learned my addr 172.21.15.67:0/2032768478 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-03T21:21:01.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:01.809+0000 7f7dc259c700 1 --2- 172.21.15.67:0/2032768478 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55b4c3d24480 0x55b4c3d489e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:21:01.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:01.809+0000 7f7dc1d9b700 1 --2- 172.21.15.67:0/2032768478 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55b4c3d28740 0x55b4c3d41d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:21:01.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:01.809+0000 7f7dc2d9d700 1 -- 172.21.15.67:0/2032768478 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55b4c3d28740 msgr2=0x55b4c3d41d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-03T21:21:01.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:01.809+0000 7f7dc2d9d700 1 --2- 172.21.15.67:0/2032768478 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55b4c3d28740 0x55b4c3d41d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:21:01.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:01.809+0000 7f7dc2d9d700 1 -- 172.21.15.67:0/2032768478 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55b4c3d24480 msgr2=0x55b4c3d489e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-03T21:21:01.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:01.809+0000 7f7dc2d9d700 1 --2- 172.21.15.67:0/2032768478 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55b4c3d24480 0x55b4c3d489e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:21:01.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:01.809+0000 7f7dc2d9d700 1 -- 172.21.15.67:0/2032768478 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b4c3d46500 con 0x55b4c3d29060 2024-05-03T21:21:01.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:01.809+0000 7f7dc2d9d700 1 --2- 172.21.15.67:0/2032768478 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b4c3d29060 0x55b4c3d42430 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f7dbc0df3f0 tx=0x7f7dbc0a8db0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1a57de0914052646 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T21:21:01.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:01.809+0000 7f7da9ffb700 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-05-03T21:21:01.818 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:01.809+0000 7f7da9ffb700 1 -- 172.21.15.67:0/2032768478 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7dbc0a8ed0 con 0x55b4c3d29060 2024-05-03T21:21:01.818 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:01.809+0000 7f7da9ffb700 1 -- 172.21.15.67:0/2032768478 <== mon.0 v2:172.21.15.67:3300/0 2 ==== osd_map(84..84 src has 1..84) ==== 8211+0+0 (secure 0 0 0) 0x7f7dbc0b1ab0 con 0x55b4c3d29060 2024-05-03T21:21:01.818 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:01.809+0000 7f7dd3880b80 1 -- 172.21.15.67:0/2032768478 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7db4004710 con 0x55b4c3d29060 2024-05-03T21:21:01.818 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:01.810+0000 7f7da9ffb700 1 -- 172.21.15.67:0/2032768478 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f7dbc0ad280 con 0x55b4c3d29060 2024-05-03T21:21:01.818 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:01.810+0000 7f7dd3880b80 10 client.15006 fetch_fsmap learned FSMap version 25 2024-05-03T21:21:01.818 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:01.810+0000 7f7dd3880b80 1 -- 172.21.15.67:0/2032768478 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55b4c3da9010 con 0x55b4c3d29060 2024-05-03T21:21:01.818 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:01.810+0000 7f7da9ffb700 1 -- 172.21.15.67:0/2032768478 <== mon.0 v2:172.21.15.67:3300/0 4 ==== config(28 keys) ==== 1170+0+0 (secure 0 0 0) 0x7f7db4004710 con 0x55b4c3d29060 2024-05-03T21:21:01.818 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:01.810+0000 7f7da9ffb700 1 -- 172.21.15.67:0/2032768478 <== mon.0 v2:172.21.15.67:3300/0 5 ==== fsmap(e 25) ==== 8338+0+0 (secure 0 0 0) 0x7f7dbc0a7c30 con 0x55b4c3d29060 2024-05-03T21:21:01.818 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:01.810+0000 7f7dd3880b80 10 client.15006 fetch_fsmap finished waiting for FSMap version 25 2024-05-03T21:21:01.818 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:01.810+0000 7f7dd3880b80 10 client.15006 resolve_mds: resolved name 'b' to daemon mds.b 2024-05-03T21:21:01.818 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:01.810+0000 7f7dd3880b80 20 client.15006 populate_metadata read hostname 'smithi067' 2024-05-03T21:21:01.818 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:01.810+0000 7f7dd3880b80 1 --2- 172.21.15.67:0/2032768478 >> [v2:172.21.15.82:6838/3644499578,v1:172.21.15.82:6839/3644499578] conn(0x55b4c3dadb50 0x55b4c3dac860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:21:01.818 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:01.810+0000 7f7dd3880b80 4 client.15006 mds_command: new command op to 24473 tid=0 multi_id=0 [{"prefix": "status"}] 2024-05-03T21:21:01.818 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:01.810+0000 7f7dd3880b80 1 -- 172.21.15.67:0/2032768478 --> [v2:172.21.15.82:6838/3644499578,v1:172.21.15.82:6839/3644499578] -- command(tid 0: {"prefix": "status"}) -- 0x55b4c3c7e370 con 0x55b4c3dadb50 2024-05-03T21:21:01.819 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:01.810+0000 7f7dc259c700 1 --2- 172.21.15.67:0/2032768478 >> [v2:172.21.15.82:6838/3644499578,v1:172.21.15.82:6839/3644499578] conn(0x55b4c3dadb50 0x55b4c3dac860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:21:01.819 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:01.811+0000 7f7dc259c700 1 --2- 172.21.15.67:0/2032768478 >> [v2:172.21.15.82:6838/3644499578,v1:172.21.15.82:6839/3644499578] conn(0x55b4c3dadb50 0x55b4c3dac860 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=52ca3ed16c9cb894 server_cookie=d05be827c260f9c7 in_seq=0 out_seq=0 2024-05-03T21:21:01.819 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:01.811+0000 7f7da9ffb700 10 client.15006 ms_handle_connect on v2:172.21.15.82:6838/3644499578 2024-05-03T21:21:01.819 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:01.811+0000 7f7da9ffb700 1 -- 172.21.15.67:0/2032768478 <== mds.0 v2:172.21.15.82:6838/3644499578 1 ==== command_reply(tid 0: 0 ) ==== 8+0+322 (crc 0 0 0) 0x55b4c3c7e370 con 0x55b4c3dadb50 2024-05-03T21:21:01.819 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:01.811+0000 7f7da9ffb700 10 client.15006 handle_command_reply: tid=0 2024-05-03T21:21:01.819 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:01.811+0000 7f7dd3880b80 1 client.15006 shutdown 2024-05-03T21:21:01.819 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:01.811+0000 7f7dd3880b80 1 -- 172.21.15.67:0/2032768478 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b4c3d29060 msgr2=0x55b4c3d42430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:21:01.819 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:01.811+0000 7f7dd3880b80 1 --2- 172.21.15.67:0/2032768478 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b4c3d29060 0x55b4c3d42430 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f7dbc0df3f0 tx=0x7f7dbc0a8db0 comp rx=0 tx=0).stop 2024-05-03T21:21:01.819 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:01.811+0000 7f7dd3880b80 1 -- 172.21.15.67:0/2032768478 shutdown_connections 2024-05-03T21:21:01.819 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:01.811+0000 7f7dd3880b80 1 --2- 172.21.15.67:0/2032768478 >> [v2:172.21.15.82:6838/3644499578,v1:172.21.15.82:6839/3644499578] conn(0x55b4c3dadb50 0x55b4c3dac860 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:21:01.819 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:01.811+0000 7f7dd3880b80 1 --2- 172.21.15.67:0/2032768478 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b4c3d29060 0x55b4c3d42430 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:21:01.819 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:01.811+0000 7f7dd3880b80 1 --2- 172.21.15.67:0/2032768478 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55b4c3d24480 0x55b4c3d489e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:21:01.819 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:01.811+0000 7f7da9ffb700 0 client.15006 ms_handle_reset on v2:172.21.15.82:6838/3644499578 2024-05-03T21:21:01.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:01.811+0000 7f7dd3880b80 1 --2- 172.21.15.67:0/2032768478 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55b4c3d28740 0x55b4c3d41d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:21:01.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:01.811+0000 7f7dd3880b80 1 -- 172.21.15.67:0/2032768478 >> 172.21.15.67:0/2032768478 conn(0x55b4c3c50b20 msgr2=0x55b4c3d22f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T21:21:01.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:01.811+0000 7f7dd3880b80 1 -- 172.21.15.67:0/2032768478 shutdown_connections 2024-05-03T21:21:01.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:01.812+0000 7f7dd3880b80 1 -- 172.21.15.67:0/2032768478 wait complete. 2024-05-03T21:21:01.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:01.812+0000 7f7dd3880b80 20 client.15006 trim_cache size 0 max 16384 2024-05-03T21:21:01.820 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-05-03T21:21:01.820 INFO:teuthology.orchestra.run.smithi067.stdout: "cluster_fsid": "713bc57a-0991-11ef-bc97-c7b262605968", 2024-05-03T21:21:01.820 INFO:teuthology.orchestra.run.smithi067.stdout: "whoami": 0, 2024-05-03T21:21:01.820 INFO:teuthology.orchestra.run.smithi067.stdout: "id": 24473, 2024-05-03T21:21:01.820 INFO:teuthology.orchestra.run.smithi067.stdout: "want_state": "up:active", 2024-05-03T21:21:01.821 INFO:teuthology.orchestra.run.smithi067.stdout: "state": "up:active", 2024-05-03T21:21:01.821 INFO:teuthology.orchestra.run.smithi067.stdout: "fs_name": "cephfs", 2024-05-03T21:21:01.821 INFO:teuthology.orchestra.run.smithi067.stdout: "rank_uptime": 136.996477516, 2024-05-03T21:21:01.821 INFO:teuthology.orchestra.run.smithi067.stdout: "mdsmap_epoch": 25, 2024-05-03T21:21:01.821 INFO:teuthology.orchestra.run.smithi067.stdout: "osdmap_epoch": 84, 2024-05-03T21:21:01.821 INFO:teuthology.orchestra.run.smithi067.stdout: "osdmap_epoch_barrier": 82, 2024-05-03T21:21:01.821 INFO:teuthology.orchestra.run.smithi067.stdout: "uptime": 151.196974074 2024-05-03T21:21:01.821 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-05-03T21:21:01.821 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:01.812+0000 7f7dc8fed700 1 -- 172.21.15.67:0/3645689064 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f7dac0b2910 msgr2=0x7f7dac0b4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:21:01.822 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:01.812+0000 7f7dc8fed700 1 --2- 172.21.15.67:0/3645689064 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f7dac0b2910 0x7f7dac0b4dd0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f7db4004630 tx=0x7f7db400b3c0 comp rx=0 tx=0).stop 2024-05-03T21:21:01.822 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:01.812+0000 7f7dc8fed700 1 -- 172.21.15.67:0/3645689064 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7dc41076c0 msgr2=0x7f7dc419e5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:21:01.822 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:01.812+0000 7f7dc8fed700 1 --2- 172.21.15.67:0/3645689064 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7dc41076c0 0x7f7dc419e5f0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f7dbc007ee0 tx=0x7f7dbc00d3b0 comp rx=0 tx=0).stop 2024-05-03T21:21:01.822 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:01.812+0000 7f7dc8fed700 1 -- 172.21.15.67:0/3645689064 shutdown_connections 2024-05-03T21:21:01.822 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:01.812+0000 7f7dc8fed700 1 --2- 172.21.15.67:0/3645689064 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f7dac0b2910 0x7f7dac0b4dd0 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:21:01.822 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:01.812+0000 7f7dc8fed700 1 --2- 172.21.15.67:0/3645689064 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7dc41076c0 0x7f7dc419e5f0 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:21:01.822 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:01.812+0000 7f7dc8fed700 1 --2- 172.21.15.67:0/3645689064 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f7dc4102630 0x7f7dc4199d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:21:01.822 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:01.812+0000 7f7dc8fed700 1 --2- 172.21.15.67:0/3645689064 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f7dc4102fe0 0x7f7dc419a2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:21:01.822 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:01.812+0000 7f7dc8fed700 1 -- 172.21.15.67:0/3645689064 >> 172.21.15.67:0/3645689064 conn(0x7f7dc4079350 msgr2=0x7f7dc4100420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T21:21:01.822 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:01.812+0000 7f7dc8fed700 1 -- 172.21.15.67:0/3645689064 shutdown_connections 2024-05-03T21:21:01.822 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:01.812+0000 7f7dc8fed700 1 -- 172.21.15.67:0/3645689064 wait complete. 2024-05-03T21:21:02.471 INFO:tasks.cephadm:Setup complete, yielding 2024-05-03T21:21:02.472 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-05-03T21:21:02.482 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi067.front.sepia.ceph.com 2024-05-03T21:21:02.482 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9a047fe7599a549042d06a3fb5b6813a203eabea shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 713bc57a-0991-11ef-bc97-c7b262605968 -- bash -c 'ceph orch status' 2024-05-03T21:21:02.733 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/713bc57a-0991-11ef-bc97-c7b262605968/mon.a/config 2024-05-03T21:21:03.471 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:21:03 smithi067 ceph-mon[29730]: pgmap v358: 97 pgs: 97 active+clean; 632 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:21:03.495 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:21:03 smithi082 ceph-mon[32933]: pgmap v358: 97 pgs: 97 active+clean; 632 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:21:03.574 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:21:03 smithi138 ceph-mon[33110]: pgmap v358: 97 pgs: 97 active+clean; 632 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:21:04.307 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:04.304+0000 7f83865fb700 1 -- 172.21.15.67:0/3243050912 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8380104700 msgr2=0x7f8380104b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:21:04.308 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:04.304+0000 7f83865fb700 1 --2- 172.21.15.67:0/3243050912 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8380104700 0x7f8380104b60 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f83740099c0 tx=0x7f8374009cd0 comp rx=0 tx=0).stop 2024-05-03T21:21:04.308 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:04.305+0000 7f83865fb700 1 -- 172.21.15.67:0/3243050912 shutdown_connections 2024-05-03T21:21:04.308 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:04.305+0000 7f83865fb700 1 --2- 172.21.15.67:0/3243050912 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f83801050a0 0x7f838010b950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:21:04.308 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:04.305+0000 7f83865fb700 1 --2- 172.21.15.67:0/3243050912 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f8380103500 0x7f8380103920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:21:04.308 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:04.305+0000 7f83865fb700 1 --2- 172.21.15.67:0/3243050912 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8380104700 0x7f8380104b60 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:21:04.308 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:04.305+0000 7f83865fb700 1 -- 172.21.15.67:0/3243050912 >> 172.21.15.67:0/3243050912 conn(0x7f83800feaa0 msgr2=0x7f8380100ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T21:21:04.308 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:04.305+0000 7f83865fb700 1 -- 172.21.15.67:0/3243050912 shutdown_connections 2024-05-03T21:21:04.308 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:04.305+0000 7f83865fb700 1 -- 172.21.15.67:0/3243050912 wait complete. 2024-05-03T21:21:04.308 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:04.305+0000 7f83865fb700 1 Processor -- start 2024-05-03T21:21:04.308 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:04.305+0000 7f83865fb700 1 -- start start 2024-05-03T21:21:04.308 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:04.306+0000 7f83865fb700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8380103500 0x7f8380199820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:21:04.308 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:04.306+0000 7f83865fb700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f8380104700 0x7f8380199d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:21:04.310 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:04.306+0000 7f83865fb700 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f83801050a0 0x7f83801a0de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:21:04.310 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:04.306+0000 7f83865fb700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f838010f210 con 0x7f8380103500 2024-05-03T21:21:04.310 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:04.306+0000 7f83865fb700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f838010f090 con 0x7f8380104700 2024-05-03T21:21:04.311 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:04.306+0000 7f83865fb700 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f838010f390 con 0x7f83801050a0 2024-05-03T21:21:04.311 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:04.306+0000 7f837ffff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8380103500 0x7f8380199820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:21:04.311 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:04.306+0000 7f837ffff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8380103500 0x7f8380199820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:35052/0 (socket says 172.21.15.67:35052) 2024-05-03T21:21:04.311 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:04.306+0000 7f837ffff700 1 -- 172.21.15.67:0/1981217993 learned_addr learned my addr 172.21.15.67:0/1981217993 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-03T21:21:04.311 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:04.306+0000 7f837f7fe700 1 --2- 172.21.15.67:0/1981217993 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f8380104700 0x7f8380199d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:21:04.311 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:04.306+0000 7f8384b98700 1 --2- 172.21.15.67:0/1981217993 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f83801050a0 0x7f83801a0de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:21:04.311 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:04.306+0000 7f837ffff700 1 -- 172.21.15.67:0/1981217993 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f83801050a0 msgr2=0x7f83801a0de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:21:04.311 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:04.306+0000 7f837ffff700 1 --2- 172.21.15.67:0/1981217993 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f83801050a0 0x7f83801a0de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:21:04.311 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:04.306+0000 7f837ffff700 1 -- 172.21.15.67:0/1981217993 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f8380104700 msgr2=0x7f8380199d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:21:04.311 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:04.306+0000 7f837ffff700 1 --2- 172.21.15.67:0/1981217993 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f8380104700 0x7f8380199d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:21:04.311 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:04.306+0000 7f837ffff700 1 -- 172.21.15.67:0/1981217993 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f83800756d0 con 0x7f8380103500 2024-05-03T21:21:04.311 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:04.306+0000 7f837ffff700 1 --2- 172.21.15.67:0/1981217993 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8380103500 0x7f8380199820 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f837000d9b0 tx=0x7f837000dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T21:21:04.311 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:04.306+0000 7f837d7fa700 1 -- 172.21.15.67:0/1981217993 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8370017030 con 0x7f8380103500 2024-05-03T21:21:04.311 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:04.307+0000 7f83865fb700 1 -- 172.21.15.67:0/1981217993 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f83800759c0 con 0x7f8380103500 2024-05-03T21:21:04.311 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:04.307+0000 7f837d7fa700 1 -- 172.21.15.67:0/1981217993 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(28 keys) ==== 1170+0+0 (secure 0 0 0) 0x7f8370004520 con 0x7f8380103500 2024-05-03T21:21:04.312 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:04.307+0000 7f837d7fa700 1 -- 172.21.15.67:0/1981217993 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8370003b10 con 0x7f8380103500 2024-05-03T21:21:04.312 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:04.307+0000 7f83865fb700 1 -- 172.21.15.67:0/1981217993 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8380075f50 con 0x7f8380103500 2024-05-03T21:21:04.312 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:04.307+0000 7f83865fb700 1 -- 172.21.15.67:0/1981217993 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f838006a190 con 0x7f8380103500 2024-05-03T21:21:04.312 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:04.309+0000 7f837d7fa700 1 -- 172.21.15.67:0/1981217993 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 17) ==== 149703+0+0 (secure 0 0 0) 0x7f837001a070 con 0x7f8380103500 2024-05-03T21:21:04.313 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:04.310+0000 7f837d7fa700 1 --2- 172.21.15.67:0/1981217993 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f83680b2910 0x7f83680b4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:21:04.313 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:04.310+0000 7f837f7fe700 1 --2- 172.21.15.67:0/1981217993 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f83680b2910 0x7f83680b4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:21:04.313 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:04.310+0000 7f837d7fa700 1 -- 172.21.15.67:0/1981217993 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 8211+0+0 (secure 0 0 0) 0x7f83700e20e0 con 0x7f8380103500 2024-05-03T21:21:04.313 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:04.310+0000 7f837f7fe700 1 --2- 172.21.15.67:0/1981217993 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f83680b2910 0x7f83680b4dd0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f8374000c00 tx=0x7f8374005fb0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T21:21:04.315 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:04.313+0000 7f837d7fa700 1 -- 172.21.15.67:0/1981217993 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191839 (secure 0 0 0) 0x7f83700a9cb0 con 0x7f8380103500 2024-05-03T21:21:04.549 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:04.547+0000 7f83865fb700 1 -- 172.21.15.67:0/1981217993 --> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] -- mgr_command(tid 0: {"prefix": "orch status", "target": ["mon-mgr", ""]}) -- 0x7f838010a4f0 con 0x7f83680b2910 2024-05-03T21:21:04.550 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:04.548+0000 7f837d7fa700 1 -- 172.21.15.67:0/1981217993 <== mgr.14150 v2:172.21.15.67:6800/604451592 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+43 (secure 0 0 0) 0x7f838010a4f0 con 0x7f83680b2910 2024-05-03T21:21:04.550 INFO:teuthology.orchestra.run.smithi067.stdout:Backend: cephadm 2024-05-03T21:21:04.550 INFO:teuthology.orchestra.run.smithi067.stdout:Available: Yes 2024-05-03T21:21:04.550 INFO:teuthology.orchestra.run.smithi067.stdout:Paused: No 2024-05-03T21:21:04.552 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:04.550+0000 7f83865fb700 1 -- 172.21.15.67:0/1981217993 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f83680b2910 msgr2=0x7f83680b4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:21:04.553 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:04.550+0000 7f83865fb700 1 --2- 172.21.15.67:0/1981217993 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f83680b2910 0x7f83680b4dd0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f8374000c00 tx=0x7f8374005fb0 comp rx=0 tx=0).stop 2024-05-03T21:21:04.553 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:04.550+0000 7f83865fb700 1 -- 172.21.15.67:0/1981217993 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8380103500 msgr2=0x7f8380199820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:21:04.553 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:04.550+0000 7f83865fb700 1 --2- 172.21.15.67:0/1981217993 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8380103500 0x7f8380199820 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f837000d9b0 tx=0x7f837000dcc0 comp rx=0 tx=0).stop 2024-05-03T21:21:04.553 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:04.550+0000 7f83865fb700 1 -- 172.21.15.67:0/1981217993 shutdown_connections 2024-05-03T21:21:04.553 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:04.550+0000 7f83865fb700 1 --2- 172.21.15.67:0/1981217993 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f83680b2910 0x7f83680b4dd0 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:21:04.553 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:04.550+0000 7f83865fb700 1 --2- 172.21.15.67:0/1981217993 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8380103500 0x7f8380199820 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:21:04.553 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:04.550+0000 7f83865fb700 1 --2- 172.21.15.67:0/1981217993 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f83801050a0 0x7f83801a0de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:21:04.553 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:04.550+0000 7f83865fb700 1 --2- 172.21.15.67:0/1981217993 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f8380104700 0x7f8380199d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:21:04.553 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:04.550+0000 7f83865fb700 1 -- 172.21.15.67:0/1981217993 >> 172.21.15.67:0/1981217993 conn(0x7f83800feaa0 msgr2=0x7f8380107a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T21:21:04.553 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:04.550+0000 7f83865fb700 1 -- 172.21.15.67:0/1981217993 shutdown_connections 2024-05-03T21:21:04.553 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:04.550+0000 7f83865fb700 1 -- 172.21.15.67:0/1981217993 wait complete. 2024-05-03T21:21:05.159 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9a047fe7599a549042d06a3fb5b6813a203eabea shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 713bc57a-0991-11ef-bc97-c7b262605968 -- bash -c 'ceph orch ps' 2024-05-03T21:21:05.409 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/713bc57a-0991-11ef-bc97-c7b262605968/mon.a/config 2024-05-03T21:21:05.471 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:21:05 smithi067 ceph-mon[29730]: pgmap v359: 97 pgs: 97 active+clean; 632 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:21:05.495 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:21:05 smithi082 ceph-mon[32933]: pgmap v359: 97 pgs: 97 active+clean; 632 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:21:05.574 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:21:05 smithi138 ceph-mon[33110]: pgmap v359: 97 pgs: 97 active+clean; 632 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:21:06.471 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:21:06 smithi067 ceph-mon[29730]: from='client.15012 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-03T21:21:06.495 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:21:06 smithi082 ceph-mon[32933]: from='client.15012 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-03T21:21:06.574 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:21:06 smithi138 ceph-mon[33110]: from='client.15012 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-03T21:21:07.081 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:07.052+0000 7f03c8e58700 1 -- 172.21.15.67:0/2923495623 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f03c4103520 msgr2=0x7f03c4103940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:21:07.081 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:07.052+0000 7f03c8e58700 1 --2- 172.21.15.67:0/2923495623 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f03c4103520 0x7f03c4103940 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f03b80099c0 tx=0x7f03b8009cd0 comp rx=0 tx=0).stop 2024-05-03T21:21:07.081 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:07.052+0000 7f03c8e58700 1 -- 172.21.15.67:0/2923495623 shutdown_connections 2024-05-03T21:21:07.081 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:07.052+0000 7f03c8e58700 1 --2- 172.21.15.67:0/2923495623 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f03c4103520 0x7f03c4103940 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:21:07.081 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:07.052+0000 7f03c8e58700 1 --2- 172.21.15.67:0/2923495623 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f03c41050c0 0x7f03c410b970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:21:07.081 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:07.052+0000 7f03c8e58700 1 --2- 172.21.15.67:0/2923495623 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f03c4104720 0x7f03c4104b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:21:07.081 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:07.052+0000 7f03c8e58700 1 -- 172.21.15.67:0/2923495623 >> 172.21.15.67:0/2923495623 conn(0x7f03c40feac0 msgr2=0x7f03c4100f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T21:21:07.081 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:07.052+0000 7f03c8e58700 1 -- 172.21.15.67:0/2923495623 shutdown_connections 2024-05-03T21:21:07.081 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:07.052+0000 7f03c8e58700 1 -- 172.21.15.67:0/2923495623 wait complete. 2024-05-03T21:21:07.082 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:07.052+0000 7f03c8e58700 1 Processor -- start 2024-05-03T21:21:07.082 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:07.052+0000 7f03c8e58700 1 -- start start 2024-05-03T21:21:07.082 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:07.053+0000 7f03c8e58700 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f03c4103520 0x7f03c4199800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:21:07.082 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:07.053+0000 7f03c8e58700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f03c4104720 0x7f03c4199d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:21:07.082 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:07.053+0000 7f03c8e58700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f03c41050c0 0x7f03c41a0dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:21:07.082 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:07.053+0000 7f03c8e58700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f03c410f240 con 0x7f03c41050c0 2024-05-03T21:21:07.082 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:07.053+0000 7f03c8e58700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f03c410f0c0 con 0x7f03c4104720 2024-05-03T21:21:07.082 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:07.053+0000 7f03c8e58700 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f03c410f3c0 con 0x7f03c4103520 2024-05-03T21:21:07.082 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:07.053+0000 7f03c2d9d700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f03c41050c0 0x7f03c41a0dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:21:07.082 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:07.053+0000 7f03c2d9d700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f03c41050c0 0x7f03c41a0dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:49314/0 (socket says 172.21.15.67:49314) 2024-05-03T21:21:07.082 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:07.053+0000 7f03c2d9d700 1 -- 172.21.15.67:0/3525038037 learned_addr learned my addr 172.21.15.67:0/3525038037 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-03T21:21:07.082 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:07.053+0000 7f03c1d9b700 1 --2- 172.21.15.67:0/3525038037 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f03c4104720 0x7f03c4199d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:21:07.082 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:07.053+0000 7f03c259c700 1 --2- 172.21.15.67:0/3525038037 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f03c4103520 0x7f03c4199800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:21:07.083 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:07.053+0000 7f03c2d9d700 1 -- 172.21.15.67:0/3525038037 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f03c4103520 msgr2=0x7f03c4199800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:21:07.083 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:07.053+0000 7f03c2d9d700 1 --2- 172.21.15.67:0/3525038037 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f03c4103520 0x7f03c4199800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:21:07.083 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:07.053+0000 7f03c2d9d700 1 -- 172.21.15.67:0/3525038037 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f03c4104720 msgr2=0x7f03c4199d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:21:07.083 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:07.053+0000 7f03c2d9d700 1 --2- 172.21.15.67:0/3525038037 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f03c4104720 0x7f03c4199d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:21:07.083 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:07.053+0000 7f03c2d9d700 1 -- 172.21.15.67:0/3525038037 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f03c41a1460 con 0x7f03c41050c0 2024-05-03T21:21:07.083 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:07.053+0000 7f03c259c700 1 --2- 172.21.15.67:0/3525038037 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f03c4103520 0x7f03c4199800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-03T21:21:07.083 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:07.053+0000 7f03c2d9d700 1 --2- 172.21.15.67:0/3525038037 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f03c41050c0 0x7f03c41a0dc0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f03bc007f10 tx=0x7f03bc00d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T21:21:07.083 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:07.053+0000 7f03b37fe700 1 -- 172.21.15.67:0/3525038037 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03bc004bb0 con 0x7f03c41050c0 2024-05-03T21:21:07.083 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:07.053+0000 7f03c8e58700 1 -- 172.21.15.67:0/3525038037 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f03c41a1750 con 0x7f03c41050c0 2024-05-03T21:21:07.083 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:07.054+0000 7f03b37fe700 1 -- 172.21.15.67:0/3525038037 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(28 keys) ==== 1170+0+0 (secure 0 0 0) 0x7f03bc004d50 con 0x7f03c41050c0 2024-05-03T21:21:07.083 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:07.054+0000 7f03b37fe700 1 -- 172.21.15.67:0/3525038037 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03bc0133f0 con 0x7f03c41050c0 2024-05-03T21:21:07.083 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:07.054+0000 7f03c8e58700 1 -- 172.21.15.67:0/3525038037 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f03c41a1c90 con 0x7f03c41050c0 2024-05-03T21:21:07.084 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:07.055+0000 7f03c8e58700 1 -- 172.21.15.67:0/3525038037 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f03c406a190 con 0x7f03c41050c0 2024-05-03T21:21:07.084 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:07.056+0000 7f03b37fe700 1 -- 172.21.15.67:0/3525038037 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 17) ==== 149703+0+0 (secure 0 0 0) 0x7f03bc01a020 con 0x7f03c41050c0 2024-05-03T21:21:07.084 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:07.057+0000 7f03b37fe700 1 --2- 172.21.15.67:0/3525038037 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f03ac0b2910 0x7f03ac0b4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:21:07.084 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:07.057+0000 7f03c259c700 1 --2- 172.21.15.67:0/3525038037 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f03ac0b2910 0x7f03ac0b4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:21:07.084 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:07.057+0000 7f03b37fe700 1 -- 172.21.15.67:0/3525038037 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 8211+0+0 (secure 0 0 0) 0x7f03bc0e1aa0 con 0x7f03c41050c0 2024-05-03T21:21:07.084 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:07.057+0000 7f03c259c700 1 --2- 172.21.15.67:0/3525038037 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f03ac0b2910 0x7f03ac0b4dd0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f03b80039a0 tx=0x7f03b8005fd0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T21:21:07.084 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:07.061+0000 7f03b37fe700 1 -- 172.21.15.67:0/3525038037 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191839 (secure 0 0 0) 0x7f03bc010070 con 0x7f03c41050c0 2024-05-03T21:21:07.287 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:07.284+0000 7f03c8e58700 1 -- 172.21.15.67:0/3525038037 --> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) -- 0x7f03c40638e0 con 0x7f03ac0b2910 2024-05-03T21:21:07.300 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:07.298+0000 7f03b37fe700 1 -- 172.21.15.67:0/3525038037 <== mgr.14150 v2:172.21.15.67:6800/604451592 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+4123 (secure 0 0 0) 0x7f03c40638e0 con 0x7f03ac0b2910 2024-05-03T21:21:07.301 INFO:teuthology.orchestra.run.smithi067.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-05-03T21:21:07.301 INFO:teuthology.orchestra.run.smithi067.stdout:mds.a smithi067 running (2m) 16s ago 2m 13.5M - 19.0.0-3450-g9a047fe7 ee4d84620249 5d2acf9b2666 2024-05-03T21:21:07.301 INFO:teuthology.orchestra.run.smithi067.stdout:mds.b smithi082 running (2m) 18s ago 2m 28.8M - 19.0.0-3450-g9a047fe7 ee4d84620249 3b7d6f82f0cb 2024-05-03T21:21:07.301 INFO:teuthology.orchestra.run.smithi067.stdout:mds.c smithi138 running (2m) 18s ago 2m 18.4M - 19.0.0-3450-g9a047fe7 ee4d84620249 34446d8bd98e 2024-05-03T21:21:07.301 INFO:teuthology.orchestra.run.smithi067.stdout:mds.d smithi067 running (3m) 16s ago 3m 14.5M - 19.0.0-3450-g9a047fe7 ee4d84620249 62837578d309 2024-05-03T21:21:07.301 INFO:teuthology.orchestra.run.smithi067.stdout:mds.e smithi082 running (2m) 18s ago 2m 19.2M - 19.0.0-3450-g9a047fe7 ee4d84620249 0e80ad566fa7 2024-05-03T21:21:07.301 INFO:teuthology.orchestra.run.smithi067.stdout:mds.f smithi138 running (2m) 18s ago 2m 15.9M - 19.0.0-3450-g9a047fe7 ee4d84620249 753ae948ece5 2024-05-03T21:21:07.301 INFO:teuthology.orchestra.run.smithi067.stdout:mds.g smithi067 running (2m) 16s ago 2m 13.7M - 19.0.0-3450-g9a047fe7 ee4d84620249 2b1a49582154 2024-05-03T21:21:07.301 INFO:teuthology.orchestra.run.smithi067.stdout:mds.h smithi082 running (2m) 18s ago 2m 20.0M - 19.0.0-3450-g9a047fe7 ee4d84620249 27462ab2a791 2024-05-03T21:21:07.301 INFO:teuthology.orchestra.run.smithi067.stdout:mds.i smithi138 running (2m) 18s ago 2m 20.6M - 19.0.0-3450-g9a047fe7 ee4d84620249 c4cc8b05b928 2024-05-03T21:21:07.301 INFO:teuthology.orchestra.run.smithi067.stdout:mds.j smithi067 running (3m) 16s ago 3m 14.2M - 19.0.0-3450-g9a047fe7 ee4d84620249 8315d5c4c638 2024-05-03T21:21:07.301 INFO:teuthology.orchestra.run.smithi067.stdout:mds.k smithi082 running (3m) 18s ago 3m 21.9M - 19.0.0-3450-g9a047fe7 ee4d84620249 9acc39735ce2 2024-05-03T21:21:07.302 INFO:teuthology.orchestra.run.smithi067.stdout:mds.l smithi138 running (3m) 18s ago 3m 14.6M - 19.0.0-3450-g9a047fe7 ee4d84620249 5f5854e64898 2024-05-03T21:21:07.302 INFO:teuthology.orchestra.run.smithi067.stdout:mgr.x smithi067 *:9283,8765 running (10m) 16s ago 10m 530M - 19.0.0-3450-g9a047fe7 ee4d84620249 c7ff7c041cc7 2024-05-03T21:21:07.302 INFO:teuthology.orchestra.run.smithi067.stdout:mgr.y smithi082 *:8443,8765 running (8m) 18s ago 8m 465M - 19.0.0-3450-g9a047fe7 ee4d84620249 cea5d1d3105e 2024-05-03T21:21:07.302 INFO:teuthology.orchestra.run.smithi067.stdout:mgr.z smithi138 *:8443,8765 running (8m) 18s ago 8m 467M - 19.0.0-3450-g9a047fe7 ee4d84620249 b67b795589f4 2024-05-03T21:21:07.302 INFO:teuthology.orchestra.run.smithi067.stdout:mon.a smithi067 running (10m) 16s ago 10m 51.9M 2048M 19.0.0-3450-g9a047fe7 ee4d84620249 a7aeb235f811 2024-05-03T21:21:07.302 INFO:teuthology.orchestra.run.smithi067.stdout:mon.b smithi082 running (8m) 18s ago 8m 40.9M 2048M 19.0.0-3450-g9a047fe7 ee4d84620249 072030cb27ff 2024-05-03T21:21:07.302 INFO:teuthology.orchestra.run.smithi067.stdout:mon.c smithi138 running (8m) 18s ago 8m 40.7M 2048M 19.0.0-3450-g9a047fe7 ee4d84620249 b833bba2b5cf 2024-05-03T21:21:07.302 INFO:teuthology.orchestra.run.smithi067.stdout:osd.0 smithi067 running (8m) 16s ago 8m 54.8M 4272M 19.0.0-3450-g9a047fe7 ee4d84620249 29510a6802ef 2024-05-03T21:21:07.302 INFO:teuthology.orchestra.run.smithi067.stdout:osd.1 smithi082 running (7m) 18s ago 7m 57.5M 4272M 19.0.0-3450-g9a047fe7 ee4d84620249 a35a94fe3d84 2024-05-03T21:21:07.302 INFO:teuthology.orchestra.run.smithi067.stdout:osd.2 smithi138 running (7m) 18s ago 7m 57.8M 4272M 19.0.0-3450-g9a047fe7 ee4d84620249 05849c3316c4 2024-05-03T21:21:07.302 INFO:teuthology.orchestra.run.smithi067.stdout:osd.3 smithi067 running (6m) 16s ago 6m 57.2M 4272M 19.0.0-3450-g9a047fe7 ee4d84620249 290dc8a7488b 2024-05-03T21:21:07.302 INFO:teuthology.orchestra.run.smithi067.stdout:osd.4 smithi082 running (6m) 18s ago 6m 45.2M 4272M 19.0.0-3450-g9a047fe7 ee4d84620249 28ed3b7118a0 2024-05-03T21:21:07.303 INFO:teuthology.orchestra.run.smithi067.stdout:osd.5 smithi138 running (6m) 18s ago 6m 56.5M 4272M 19.0.0-3450-g9a047fe7 ee4d84620249 a25605f9bc96 2024-05-03T21:21:07.303 INFO:teuthology.orchestra.run.smithi067.stdout:osd.6 smithi067 running (5m) 16s ago 5m 45.3M 4272M 19.0.0-3450-g9a047fe7 ee4d84620249 d96a92031749 2024-05-03T21:21:07.303 INFO:teuthology.orchestra.run.smithi067.stdout:osd.7 smithi082 running (5m) 18s ago 5m 47.0M 4272M 19.0.0-3450-g9a047fe7 ee4d84620249 9cfd2cc49f13 2024-05-03T21:21:07.303 INFO:teuthology.orchestra.run.smithi067.stdout:osd.8 smithi138 running (4m) 18s ago 4m 56.8M 4272M 19.0.0-3450-g9a047fe7 ee4d84620249 0726426ae706 2024-05-03T21:21:07.303 INFO:teuthology.orchestra.run.smithi067.stdout:osd.9 smithi067 running (4m) 16s ago 4m 53.0M 4272M 19.0.0-3450-g9a047fe7 ee4d84620249 90d64d93ab3e 2024-05-03T21:21:07.303 INFO:teuthology.orchestra.run.smithi067.stdout:osd.10 smithi082 running (3m) 18s ago 3m 47.8M 4272M 19.0.0-3450-g9a047fe7 ee4d84620249 1f2947907122 2024-05-03T21:21:07.303 INFO:teuthology.orchestra.run.smithi067.stdout:osd.11 smithi138 running (3m) 18s ago 3m 44.4M 4272M 19.0.0-3450-g9a047fe7 ee4d84620249 b6f3e5c4917b 2024-05-03T21:21:07.304 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:07.301+0000 7f03c8e58700 1 -- 172.21.15.67:0/3525038037 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f03ac0b2910 msgr2=0x7f03ac0b4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:21:07.304 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:07.301+0000 7f03c8e58700 1 --2- 172.21.15.67:0/3525038037 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f03ac0b2910 0x7f03ac0b4dd0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f03b80039a0 tx=0x7f03b8005fd0 comp rx=0 tx=0).stop 2024-05-03T21:21:07.304 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:07.301+0000 7f03c8e58700 1 -- 172.21.15.67:0/3525038037 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f03c41050c0 msgr2=0x7f03c41a0dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:21:07.304 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:07.301+0000 7f03c8e58700 1 --2- 172.21.15.67:0/3525038037 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f03c41050c0 0x7f03c41a0dc0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f03bc007f10 tx=0x7f03bc00d3b0 comp rx=0 tx=0).stop 2024-05-03T21:21:07.304 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:07.301+0000 7f03c8e58700 1 -- 172.21.15.67:0/3525038037 shutdown_connections 2024-05-03T21:21:07.304 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:07.301+0000 7f03c8e58700 1 --2- 172.21.15.67:0/3525038037 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f03ac0b2910 0x7f03ac0b4dd0 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:21:07.304 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:07.301+0000 7f03c8e58700 1 --2- 172.21.15.67:0/3525038037 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f03c41050c0 0x7f03c41a0dc0 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:21:07.304 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:07.301+0000 7f03c8e58700 1 --2- 172.21.15.67:0/3525038037 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f03c4103520 0x7f03c4199800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:21:07.305 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:07.301+0000 7f03c8e58700 1 --2- 172.21.15.67:0/3525038037 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f03c4104720 0x7f03c4199d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:21:07.305 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:07.301+0000 7f03c8e58700 1 -- 172.21.15.67:0/3525038037 >> 172.21.15.67:0/3525038037 conn(0x7f03c40feac0 msgr2=0x7f03c4107950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T21:21:07.305 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:07.301+0000 7f03c8e58700 1 -- 172.21.15.67:0/3525038037 shutdown_connections 2024-05-03T21:21:07.305 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:07.301+0000 7f03c8e58700 1 -- 172.21.15.67:0/3525038037 wait complete. 2024-05-03T21:21:07.314 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:21:07 smithi067 ceph-mon[29730]: pgmap v360: 97 pgs: 97 active+clean; 632 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:21:07.495 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:21:07 smithi082 ceph-mon[32933]: pgmap v360: 97 pgs: 97 active+clean; 632 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:21:07.574 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:21:07 smithi138 ceph-mon[33110]: pgmap v360: 97 pgs: 97 active+clean; 632 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:21:07.908 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9a047fe7599a549042d06a3fb5b6813a203eabea shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 713bc57a-0991-11ef-bc97-c7b262605968 -- bash -c 'ceph orch ls' 2024-05-03T21:21:08.156 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/713bc57a-0991-11ef-bc97-c7b262605968/mon.a/config 2024-05-03T21:21:08.471 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:21:08 smithi067 ceph-mon[29730]: from='client.15018 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-05-03T21:21:08.495 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:21:08 smithi082 ceph-mon[32933]: from='client.15018 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-05-03T21:21:08.574 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:21:08 smithi138 ceph-mon[33110]: from='client.15018 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-05-03T21:21:09.328 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:21:09 smithi067 ceph-mon[29730]: pgmap v361: 97 pgs: 97 active+clean; 632 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:21:09.495 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:21:09 smithi082 ceph-mon[32933]: pgmap v361: 97 pgs: 97 active+clean; 632 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:21:09.574 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:21:09 smithi138 ceph-mon[33110]: pgmap v361: 97 pgs: 97 active+clean; 632 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:21:09.881 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:09.879+0000 7fef60f84700 1 -- 172.21.15.67:0/301175082 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fef5c103520 msgr2=0x7fef5c103940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:21:09.881 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:09.879+0000 7fef60f84700 1 --2- 172.21.15.67:0/301175082 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fef5c103520 0x7fef5c103940 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7fef4c0099c0 tx=0x7fef4c009cd0 comp rx=0 tx=0).stop 2024-05-03T21:21:09.882 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:09.879+0000 7fef60f84700 1 -- 172.21.15.67:0/301175082 shutdown_connections 2024-05-03T21:21:09.882 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:09.879+0000 7fef60f84700 1 --2- 172.21.15.67:0/301175082 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fef5c103520 0x7fef5c103940 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:21:09.882 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:09.879+0000 7fef60f84700 1 --2- 172.21.15.67:0/301175082 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fef5c1050c0 0x7fef5c10b970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:21:09.882 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:09.879+0000 7fef60f84700 1 --2- 172.21.15.67:0/301175082 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fef5c104720 0x7fef5c104b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:21:09.882 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:09.879+0000 7fef60f84700 1 -- 172.21.15.67:0/301175082 >> 172.21.15.67:0/301175082 conn(0x7fef5c0feac0 msgr2=0x7fef5c100f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T21:21:09.882 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:09.879+0000 7fef60f84700 1 -- 172.21.15.67:0/301175082 shutdown_connections 2024-05-03T21:21:09.882 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:09.879+0000 7fef60f84700 1 -- 172.21.15.67:0/301175082 wait complete. 2024-05-03T21:21:09.884 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:09.880+0000 7fef60f84700 1 Processor -- start 2024-05-03T21:21:09.884 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:09.880+0000 7fef60f84700 1 -- start start 2024-05-03T21:21:09.884 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:09.880+0000 7fef60f84700 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fef5c103520 0x7fef5c077240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:21:09.884 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:09.880+0000 7fef60f84700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fef5c104720 0x7fef5c075890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:21:09.884 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:09.880+0000 7fef60f84700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fef5c1050c0 0x7fef5c075dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:21:09.884 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:09.880+0000 7fef60f84700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fef5c10f230 con 0x7fef5c1050c0 2024-05-03T21:21:09.884 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:09.880+0000 7fef60f84700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7fef5c10f0b0 con 0x7fef5c104720 2024-05-03T21:21:09.884 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:09.880+0000 7fef60f84700 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7fef5c10f3b0 con 0x7fef5c103520 2024-05-03T21:21:09.884 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:09.880+0000 7fef5ad9d700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fef5c1050c0 0x7fef5c075dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:21:09.884 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:09.880+0000 7fef5ad9d700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fef5c1050c0 0x7fef5c075dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:49334/0 (socket says 172.21.15.67:49334) 2024-05-03T21:21:09.884 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:09.880+0000 7fef5ad9d700 1 -- 172.21.15.67:0/2777235471 learned_addr learned my addr 172.21.15.67:0/2777235471 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-03T21:21:09.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:09.880+0000 7fef59d9b700 1 --2- 172.21.15.67:0/2777235471 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fef5c104720 0x7fef5c075890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:21:09.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:09.880+0000 7fef5a59c700 1 --2- 172.21.15.67:0/2777235471 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fef5c103520 0x7fef5c077240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:21:09.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:09.881+0000 7fef5ad9d700 1 -- 172.21.15.67:0/2777235471 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fef5c103520 msgr2=0x7fef5c077240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:21:09.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:09.881+0000 7fef5ad9d700 1 --2- 172.21.15.67:0/2777235471 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fef5c103520 0x7fef5c077240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:21:09.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:09.881+0000 7fef5ad9d700 1 -- 172.21.15.67:0/2777235471 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fef5c104720 msgr2=0x7fef5c075890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:21:09.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:09.881+0000 7fef5ad9d700 1 --2- 172.21.15.67:0/2777235471 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fef5c104720 0x7fef5c075890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:21:09.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:09.881+0000 7fef5ad9d700 1 -- 172.21.15.67:0/2777235471 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fef5c076680 con 0x7fef5c1050c0 2024-05-03T21:21:09.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:09.881+0000 7fef5ad9d700 1 --2- 172.21.15.67:0/2777235471 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fef5c1050c0 0x7fef5c075dd0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7fef5400ada0 tx=0x7fef5400c330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T21:21:09.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:09.881+0000 7fef4b7fe700 1 -- 172.21.15.67:0/2777235471 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef54017020 con 0x7fef5c1050c0 2024-05-03T21:21:09.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:09.881+0000 7fef60f84700 1 -- 172.21.15.67:0/2777235471 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fef5c1a39b0 con 0x7fef5c1050c0 2024-05-03T21:21:09.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:09.881+0000 7fef4b7fe700 1 -- 172.21.15.67:0/2777235471 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(28 keys) ==== 1170+0+0 (secure 0 0 0) 0x7fef5400cdd0 con 0x7fef5c1050c0 2024-05-03T21:21:09.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:09.881+0000 7fef4b7fe700 1 -- 172.21.15.67:0/2777235471 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef54005440 con 0x7fef5c1050c0 2024-05-03T21:21:09.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:09.881+0000 7fef60f84700 1 -- 172.21.15.67:0/2777235471 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fef5c1a3e00 con 0x7fef5c1050c0 2024-05-03T21:21:09.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:09.882+0000 7fef60f84700 1 -- 172.21.15.67:0/2777235471 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fef5c06a190 con 0x7fef5c1050c0 2024-05-03T21:21:09.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:09.884+0000 7fef4b7fe700 1 -- 172.21.15.67:0/2777235471 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 17) ==== 149703+0+0 (secure 0 0 0) 0x7fef5400c8d0 con 0x7fef5c1050c0 2024-05-03T21:21:09.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:09.884+0000 7fef4b7fe700 1 --2- 172.21.15.67:0/2777235471 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7fef440b2990 0x7fef440b4e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:21:09.887 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:09.884+0000 7fef5a59c700 1 --2- 172.21.15.67:0/2777235471 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7fef440b2990 0x7fef440b4e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:21:09.887 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:09.884+0000 7fef4b7fe700 1 -- 172.21.15.67:0/2777235471 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 8211+0+0 (secure 0 0 0) 0x7fef540e1a20 con 0x7fef5c1050c0 2024-05-03T21:21:09.887 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:09.885+0000 7fef5a59c700 1 --2- 172.21.15.67:0/2777235471 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7fef440b2990 0x7fef440b4e50 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fef4c00b580 tx=0x7fef4c005fb0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T21:21:09.891 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:09.889+0000 7fef4b7fe700 1 -- 172.21.15.67:0/2777235471 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191839 (secure 0 0 0) 0x7fef54021020 con 0x7fef5c1050c0 2024-05-03T21:21:10.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:10.114+0000 7fef60f84700 1 -- 172.21.15.67:0/2777235471 --> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] -- mgr_command(tid 0: {"prefix": "orch ls", "target": ["mon-mgr", ""]}) -- 0x7fef5c0638e0 con 0x7fef440b2990 2024-05-03T21:21:10.122 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:10.120+0000 7fef4b7fe700 1 -- 172.21.15.67:0/2777235471 <== mgr.14150 v2:172.21.15.67:6800/604451592 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+980 (secure 0 0 0) 0x7fef5c0638e0 con 0x7fef440b2990 2024-05-03T21:21:10.122 INFO:teuthology.orchestra.run.smithi067.stdout:NAME PORTS RUNNING REFRESHED AGE PLACEMENT 2024-05-03T21:21:10.122 INFO:teuthology.orchestra.run.smithi067.stdout:mds.all 12/12 21s ago 3m smithi067=a;smithi067=d;smithi067=g;smithi067=j;smithi082=b;smithi082=e;smithi082=h;smithi082=k;smithi138=c;smithi138=f;smithi138=i;smithi138=l;count:12 2024-05-03T21:21:10.122 INFO:teuthology.orchestra.run.smithi067.stdout:mgr 3/3 21s ago 8m smithi067=x;smithi082=y;smithi138=z;count:3 2024-05-03T21:21:10.122 INFO:teuthology.orchestra.run.smithi067.stdout:mon 3/3 21s ago 8m smithi067:172.21.15.67=a;smithi082:172.21.15.82=b;smithi138:172.21.15.138=c;count:3 2024-05-03T21:21:10.122 INFO:teuthology.orchestra.run.smithi067.stdout:osd 12 21s ago - 2024-05-03T21:21:10.124 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:10.122+0000 7fef60f84700 1 -- 172.21.15.67:0/2777235471 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7fef440b2990 msgr2=0x7fef440b4e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:21:10.125 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:10.122+0000 7fef60f84700 1 --2- 172.21.15.67:0/2777235471 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7fef440b2990 0x7fef440b4e50 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fef4c00b580 tx=0x7fef4c005fb0 comp rx=0 tx=0).stop 2024-05-03T21:21:10.125 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:10.122+0000 7fef60f84700 1 -- 172.21.15.67:0/2777235471 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fef5c1050c0 msgr2=0x7fef5c075dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:21:10.125 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:10.122+0000 7fef60f84700 1 --2- 172.21.15.67:0/2777235471 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fef5c1050c0 0x7fef5c075dd0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7fef5400ada0 tx=0x7fef5400c330 comp rx=0 tx=0).stop 2024-05-03T21:21:10.125 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:10.122+0000 7fef60f84700 1 -- 172.21.15.67:0/2777235471 shutdown_connections 2024-05-03T21:21:10.125 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:10.122+0000 7fef60f84700 1 --2- 172.21.15.67:0/2777235471 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7fef440b2990 0x7fef440b4e50 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:21:10.125 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:10.122+0000 7fef60f84700 1 --2- 172.21.15.67:0/2777235471 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fef5c1050c0 0x7fef5c075dd0 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:21:10.125 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:10.122+0000 7fef60f84700 1 --2- 172.21.15.67:0/2777235471 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fef5c103520 0x7fef5c077240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:21:10.125 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:10.122+0000 7fef60f84700 1 --2- 172.21.15.67:0/2777235471 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fef5c104720 0x7fef5c075890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:21:10.125 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:10.122+0000 7fef60f84700 1 -- 172.21.15.67:0/2777235471 >> 172.21.15.67:0/2777235471 conn(0x7fef5c0feac0 msgr2=0x7fef5c107a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T21:21:10.125 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:10.122+0000 7fef60f84700 1 -- 172.21.15.67:0/2777235471 shutdown_connections 2024-05-03T21:21:10.125 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:10.122+0000 7fef60f84700 1 -- 172.21.15.67:0/2777235471 wait complete. 2024-05-03T21:21:10.706 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9a047fe7599a549042d06a3fb5b6813a203eabea shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 713bc57a-0991-11ef-bc97-c7b262605968 -- bash -c 'ceph orch host ls' 2024-05-03T21:21:10.955 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/713bc57a-0991-11ef-bc97-c7b262605968/mon.a/config 2024-05-03T21:21:11.397 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:21:11 smithi067 ceph-mon[29730]: pgmap v362: 97 pgs: 97 active+clean; 632 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T21:21:11.397 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:21:11 smithi067 ceph-mon[29730]: from='client.15024 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-05-03T21:21:11.495 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:21:11 smithi082 ceph-mon[32933]: pgmap v362: 97 pgs: 97 active+clean; 632 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T21:21:11.495 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:21:11 smithi082 ceph-mon[32933]: from='client.15024 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-05-03T21:21:11.574 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:21:11 smithi138 ceph-mon[33110]: pgmap v362: 97 pgs: 97 active+clean; 632 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T21:21:11.574 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:21:11 smithi138 ceph-mon[33110]: from='client.15024 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-05-03T21:21:12.564 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:12.559+0000 7fd8869e4700 1 -- 172.21.15.67:0/672571835 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd8801050c0 msgr2=0x7fd88010b970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:21:12.564 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:12.559+0000 7fd8869e4700 1 --2- 172.21.15.67:0/672571835 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd8801050c0 0x7fd88010b970 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7fd87800b0d0 tx=0x7fd87800b3e0 comp rx=0 tx=0).stop 2024-05-03T21:21:12.564 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:12.559+0000 7fd8869e4700 1 -- 172.21.15.67:0/672571835 shutdown_connections 2024-05-03T21:21:12.564 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:12.559+0000 7fd8869e4700 1 --2- 172.21.15.67:0/672571835 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd8801050c0 0x7fd88010b970 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:21:12.564 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:12.559+0000 7fd8869e4700 1 --2- 172.21.15.67:0/672571835 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fd880103520 0x7fd880103940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:21:12.564 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:12.559+0000 7fd8869e4700 1 --2- 172.21.15.67:0/672571835 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fd880104720 0x7fd880104b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:21:12.564 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:12.559+0000 7fd8869e4700 1 -- 172.21.15.67:0/672571835 >> 172.21.15.67:0/672571835 conn(0x7fd8800feac0 msgr2=0x7fd880100f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T21:21:12.565 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:12.559+0000 7fd8869e4700 1 -- 172.21.15.67:0/672571835 shutdown_connections 2024-05-03T21:21:12.565 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:12.559+0000 7fd8869e4700 1 -- 172.21.15.67:0/672571835 wait complete. 2024-05-03T21:21:12.565 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:12.560+0000 7fd8869e4700 1 Processor -- start 2024-05-03T21:21:12.565 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:12.560+0000 7fd8869e4700 1 -- start start 2024-05-03T21:21:12.565 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:12.560+0000 7fd8869e4700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fd880103520 0x7fd880077240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:21:12.565 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:12.561+0000 7fd8869e4700 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fd880104720 0x7fd880075890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:21:12.565 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:12.561+0000 7fd8869e4700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd8801050c0 0x7fd880075dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:21:12.565 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:12.561+0000 7fd8869e4700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fd88010f280 con 0x7fd8801050c0 2024-05-03T21:21:12.565 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:12.561+0000 7fd8869e4700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7fd88010f100 con 0x7fd880103520 2024-05-03T21:21:12.565 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:12.561+0000 7fd8869e4700 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7fd88010f400 con 0x7fd880104720 2024-05-03T21:21:12.565 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:12.561+0000 7fd884f81700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd8801050c0 0x7fd880075dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:21:12.565 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:12.561+0000 7fd884f81700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd8801050c0 0x7fd880075dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:49352/0 (socket says 172.21.15.67:49352) 2024-05-03T21:21:12.565 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:12.561+0000 7fd884f81700 1 -- 172.21.15.67:0/3282270925 learned_addr learned my addr 172.21.15.67:0/3282270925 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-03T21:21:12.565 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:12.561+0000 7fd87ffff700 1 --2- 172.21.15.67:0/3282270925 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fd880103520 0x7fd880077240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:21:12.566 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:12.561+0000 7fd87f7fe700 1 --2- 172.21.15.67:0/3282270925 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fd880104720 0x7fd880075890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:21:12.566 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:12.561+0000 7fd884f81700 1 -- 172.21.15.67:0/3282270925 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fd880104720 msgr2=0x7fd880075890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:21:12.566 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:12.561+0000 7fd884f81700 1 --2- 172.21.15.67:0/3282270925 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fd880104720 0x7fd880075890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:21:12.566 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:12.561+0000 7fd884f81700 1 -- 172.21.15.67:0/3282270925 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fd880103520 msgr2=0x7fd880077240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:21:12.566 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:12.561+0000 7fd884f81700 1 --2- 172.21.15.67:0/3282270925 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fd880103520 0x7fd880077240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:21:12.566 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:12.561+0000 7fd884f81700 1 -- 172.21.15.67:0/3282270925 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd880076680 con 0x7fd8801050c0 2024-05-03T21:21:12.566 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:12.561+0000 7fd884f81700 1 --2- 172.21.15.67:0/3282270925 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd8801050c0 0x7fd880075dd0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7fd878009c60 tx=0x7fd878007dc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T21:21:12.566 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:12.561+0000 7fd87d7fa700 1 -- 172.21.15.67:0/3282270925 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd878021020 con 0x7fd8801050c0 2024-05-03T21:21:12.566 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:12.561+0000 7fd8869e4700 1 -- 172.21.15.67:0/3282270925 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd8801a3770 con 0x7fd8801050c0 2024-05-03T21:21:12.566 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:12.561+0000 7fd87d7fa700 1 -- 172.21.15.67:0/3282270925 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(28 keys) ==== 1170+0+0 (secure 0 0 0) 0x7fd87801dd80 con 0x7fd8801050c0 2024-05-03T21:21:12.566 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:12.561+0000 7fd87d7fa700 1 -- 172.21.15.67:0/3282270925 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8780123f0 con 0x7fd8801050c0 2024-05-03T21:21:12.566 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:12.561+0000 7fd8869e4700 1 -- 172.21.15.67:0/3282270925 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd8801a3bd0 con 0x7fd8801050c0 2024-05-03T21:21:12.567 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:12.563+0000 7fd8869e4700 1 -- 172.21.15.67:0/3282270925 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd88006a190 con 0x7fd8801050c0 2024-05-03T21:21:12.567 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:12.564+0000 7fd87d7fa700 1 -- 172.21.15.67:0/3282270925 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 17) ==== 149703+0+0 (secure 0 0 0) 0x7fd878024020 con 0x7fd8801050c0 2024-05-03T21:21:12.567 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:12.565+0000 7fd87d7fa700 1 --2- 172.21.15.67:0/3282270925 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7fd8680b2910 0x7fd8680b4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:21:12.567 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:12.565+0000 7fd87ffff700 1 --2- 172.21.15.67:0/3282270925 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7fd8680b2910 0x7fd8680b4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:21:12.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:12.565+0000 7fd87d7fa700 1 -- 172.21.15.67:0/3282270925 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 8211+0+0 (secure 0 0 0) 0x7fd8780e3de0 con 0x7fd8801050c0 2024-05-03T21:21:12.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:12.565+0000 7fd87ffff700 1 --2- 172.21.15.67:0/3282270925 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7fd8680b2910 0x7fd8680b4dd0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fd870005fd0 tx=0x7fd870005e50 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T21:21:12.570 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:12.568+0000 7fd87d7fa700 1 -- 172.21.15.67:0/3282270925 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191839 (secure 0 0 0) 0x7fd87802b020 con 0x7fd8801050c0 2024-05-03T21:21:12.790 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:12.788+0000 7fd8869e4700 1 -- 172.21.15.67:0/3282270925 --> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""]}) -- 0x7fd8800638e0 con 0x7fd8680b2910 2024-05-03T21:21:12.792 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:12.790+0000 7fd87d7fa700 1 -- 172.21.15.67:0/3282270925 <== mgr.14150 v2:172.21.15.67:6800/604451592 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+191 (secure 0 0 0) 0x7fd8800638e0 con 0x7fd8680b2910 2024-05-03T21:21:12.792 INFO:teuthology.orchestra.run.smithi067.stdout:HOST ADDR LABELS STATUS 2024-05-03T21:21:12.792 INFO:teuthology.orchestra.run.smithi067.stdout:smithi067 172.21.15.67 2024-05-03T21:21:12.792 INFO:teuthology.orchestra.run.smithi067.stdout:smithi082 172.21.15.82 2024-05-03T21:21:12.792 INFO:teuthology.orchestra.run.smithi067.stdout:smithi138 172.21.15.138 2024-05-03T21:21:12.792 INFO:teuthology.orchestra.run.smithi067.stdout:3 hosts in cluster 2024-05-03T21:21:12.794 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:12.792+0000 7fd8869e4700 1 -- 172.21.15.67:0/3282270925 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7fd8680b2910 msgr2=0x7fd8680b4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:21:12.794 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:12.792+0000 7fd8869e4700 1 --2- 172.21.15.67:0/3282270925 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7fd8680b2910 0x7fd8680b4dd0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fd870005fd0 tx=0x7fd870005e50 comp rx=0 tx=0).stop 2024-05-03T21:21:12.794 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:12.792+0000 7fd8869e4700 1 -- 172.21.15.67:0/3282270925 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd8801050c0 msgr2=0x7fd880075dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:21:12.794 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:12.792+0000 7fd8869e4700 1 --2- 172.21.15.67:0/3282270925 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd8801050c0 0x7fd880075dd0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7fd878009c60 tx=0x7fd878007dc0 comp rx=0 tx=0).stop 2024-05-03T21:21:12.794 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:12.792+0000 7fd8869e4700 1 -- 172.21.15.67:0/3282270925 shutdown_connections 2024-05-03T21:21:12.794 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:12.792+0000 7fd8869e4700 1 --2- 172.21.15.67:0/3282270925 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7fd8680b2910 0x7fd8680b4dd0 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:21:12.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:12.792+0000 7fd8869e4700 1 --2- 172.21.15.67:0/3282270925 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd8801050c0 0x7fd880075dd0 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:21:12.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:12.792+0000 7fd8869e4700 1 --2- 172.21.15.67:0/3282270925 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fd880103520 0x7fd880077240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:21:12.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:12.792+0000 7fd8869e4700 1 --2- 172.21.15.67:0/3282270925 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fd880104720 0x7fd880075890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:21:12.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:12.792+0000 7fd8869e4700 1 -- 172.21.15.67:0/3282270925 >> 172.21.15.67:0/3282270925 conn(0x7fd8800feac0 msgr2=0x7fd880107950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T21:21:12.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:12.792+0000 7fd8869e4700 1 -- 172.21.15.67:0/3282270925 shutdown_connections 2024-05-03T21:21:12.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:12.792+0000 7fd8869e4700 1 -- 172.21.15.67:0/3282270925 wait complete. 2024-05-03T21:21:13.425 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:21:13 smithi067 ceph-mon[29730]: pgmap v363: 97 pgs: 97 active+clean; 632 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:21:13.456 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9a047fe7599a549042d06a3fb5b6813a203eabea shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 713bc57a-0991-11ef-bc97-c7b262605968 -- bash -c 'ceph orch device ls' 2024-05-03T21:21:13.495 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:21:13 smithi082 ceph-mon[32933]: pgmap v363: 97 pgs: 97 active+clean; 632 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:21:13.574 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:21:13 smithi138 ceph-mon[33110]: pgmap v363: 97 pgs: 97 active+clean; 632 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:21:13.708 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/713bc57a-0991-11ef-bc97-c7b262605968/mon.a/config 2024-05-03T21:21:14.474 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:21:14 smithi067 ceph-mon[29730]: from='client.15030 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-05-03T21:21:14.495 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:21:14 smithi082 ceph-mon[32933]: from='client.15030 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-05-03T21:21:14.574 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:21:14 smithi138 ceph-mon[33110]: from='client.15030 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-05-03T21:21:15.262 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:15.258+0000 7f64d7d4d700 1 -- 172.21.15.67:0/1075661598 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f64d0104710 msgr2=0x7f64d0104b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:21:15.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:15.258+0000 7f64d7d4d700 1 --2- 172.21.15.67:0/1075661598 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f64d0104710 0x7f64d0104b70 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f64c00099c0 tx=0x7f64c0009cd0 comp rx=0 tx=0).stop 2024-05-03T21:21:15.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:15.258+0000 7f64d7d4d700 1 -- 172.21.15.67:0/1075661598 shutdown_connections 2024-05-03T21:21:15.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:15.258+0000 7f64d7d4d700 1 --2- 172.21.15.67:0/1075661598 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f64d01050b0 0x7f64d010b960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:21:15.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:15.258+0000 7f64d7d4d700 1 --2- 172.21.15.67:0/1075661598 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f64d0103510 0x7f64d0103930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:21:15.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:15.258+0000 7f64d7d4d700 1 --2- 172.21.15.67:0/1075661598 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f64d0104710 0x7f64d0104b70 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:21:15.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:15.258+0000 7f64d7d4d700 1 -- 172.21.15.67:0/1075661598 >> 172.21.15.67:0/1075661598 conn(0x7f64d00fea70 msgr2=0x7f64d0100ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T21:21:15.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:15.258+0000 7f64d7d4d700 1 -- 172.21.15.67:0/1075661598 shutdown_connections 2024-05-03T21:21:15.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:15.258+0000 7f64d7d4d700 1 -- 172.21.15.67:0/1075661598 wait complete. 2024-05-03T21:21:15.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:15.259+0000 7f64d7d4d700 1 Processor -- start 2024-05-03T21:21:15.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:15.259+0000 7f64d7d4d700 1 -- start start 2024-05-03T21:21:15.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:15.259+0000 7f64d7d4d700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f64d0103510 0x7f64d0199810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:21:15.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:15.259+0000 7f64d7d4d700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f64d0104710 0x7f64d0199d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:21:15.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:15.259+0000 7f64d7d4d700 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f64d01050b0 0x7f64d01a0dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:21:15.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:15.259+0000 7f64d7d4d700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f64d010f280 con 0x7f64d0104710 2024-05-03T21:21:15.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:15.259+0000 7f64d7d4d700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f64d010f100 con 0x7f64d0103510 2024-05-03T21:21:15.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:15.259+0000 7f64d7d4d700 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f64d010f400 con 0x7f64d01050b0 2024-05-03T21:21:15.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:15.259+0000 7f64d52e8700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f64d0104710 0x7f64d0199d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:21:15.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:15.259+0000 7f64d52e8700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f64d0104710 0x7f64d0199d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:49372/0 (socket says 172.21.15.67:49372) 2024-05-03T21:21:15.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:15.259+0000 7f64d52e8700 1 -- 172.21.15.67:0/1757508496 learned_addr learned my addr 172.21.15.67:0/1757508496 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-03T21:21:15.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:15.259+0000 7f64d5ae9700 1 --2- 172.21.15.67:0/1757508496 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f64d0103510 0x7f64d0199810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:21:15.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:15.259+0000 7f64d52e8700 1 -- 172.21.15.67:0/1757508496 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f64d01050b0 msgr2=0x7f64d01a0dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:21:15.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:15.259+0000 7f64d62ea700 1 --2- 172.21.15.67:0/1757508496 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f64d01050b0 0x7f64d01a0dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:21:15.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:15.259+0000 7f64d52e8700 1 --2- 172.21.15.67:0/1757508496 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f64d01050b0 0x7f64d01a0dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:21:15.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:15.259+0000 7f64d52e8700 1 -- 172.21.15.67:0/1757508496 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f64d0103510 msgr2=0x7f64d0199810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:21:15.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:15.259+0000 7f64d52e8700 1 --2- 172.21.15.67:0/1757508496 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f64d0103510 0x7f64d0199810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:21:15.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:15.259+0000 7f64d52e8700 1 -- 172.21.15.67:0/1757508496 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f64d01a1470 con 0x7f64d0104710 2024-05-03T21:21:15.265 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:15.259+0000 7f64d5ae9700 1 --2- 172.21.15.67:0/1757508496 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f64d0103510 0x7f64d0199810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-03T21:21:15.265 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:15.259+0000 7f64d62ea700 1 --2- 172.21.15.67:0/1757508496 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f64d01050b0 0x7f64d01a0dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-03T21:21:15.265 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:15.260+0000 7f64d52e8700 1 --2- 172.21.15.67:0/1757508496 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f64d0104710 0x7f64d0199d50 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f64c0004380 tx=0x7f64c0003730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T21:21:15.265 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:15.260+0000 7f64c6ffd700 1 -- 172.21.15.67:0/1757508496 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64c0027070 con 0x7f64d0104710 2024-05-03T21:21:15.265 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:15.260+0000 7f64d7d4d700 1 -- 172.21.15.67:0/1757508496 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f64d01a1700 con 0x7f64d0104710 2024-05-03T21:21:15.265 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:15.260+0000 7f64c6ffd700 1 -- 172.21.15.67:0/1757508496 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(28 keys) ==== 1170+0+0 (secure 0 0 0) 0x7f64c0022470 con 0x7f64d0104710 2024-05-03T21:21:15.265 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:15.260+0000 7f64c6ffd700 1 -- 172.21.15.67:0/1757508496 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64c000f6c0 con 0x7f64d0104710 2024-05-03T21:21:15.265 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:15.260+0000 7f64d7d4d700 1 -- 172.21.15.67:0/1757508496 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f64d01a1b50 con 0x7f64d0104710 2024-05-03T21:21:15.266 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:15.261+0000 7f64d7d4d700 1 -- 172.21.15.67:0/1757508496 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f64d006a190 con 0x7f64d0104710 2024-05-03T21:21:15.266 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:15.262+0000 7f64c6ffd700 1 -- 172.21.15.67:0/1757508496 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 17) ==== 149703+0+0 (secure 0 0 0) 0x7f64c002a070 con 0x7f64d0104710 2024-05-03T21:21:15.266 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:15.263+0000 7f64c6ffd700 1 --2- 172.21.15.67:0/1757508496 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f64bc0b2910 0x7f64bc0b4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:21:15.266 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:15.263+0000 7f64d5ae9700 1 --2- 172.21.15.67:0/1757508496 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f64bc0b2910 0x7f64bc0b4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:21:15.266 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:15.263+0000 7f64c6ffd700 1 -- 172.21.15.67:0/1757508496 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 8211+0+0 (secure 0 0 0) 0x7f64c00e3880 con 0x7f64d0104710 2024-05-03T21:21:15.266 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:15.263+0000 7f64d5ae9700 1 --2- 172.21.15.67:0/1757508496 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f64bc0b2910 0x7f64bc0b4dd0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f64cc005fd0 tx=0x7f64cc005de0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T21:21:15.268 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:15.266+0000 7f64c6ffd700 1 -- 172.21.15.67:0/1757508496 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191839 (secure 0 0 0) 0x7f64c0013070 con 0x7f64d0104710 2024-05-03T21:21:15.471 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:21:15 smithi067 ceph-mon[29730]: pgmap v364: 97 pgs: 97 active+clean; 632 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:21:15.494 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:15.492+0000 7f64d7d4d700 1 -- 172.21.15.67:0/1757508496 --> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] -- mgr_command(tid 0: {"prefix": "orch device ls", "target": ["mon-mgr", ""]}) -- 0x7f64d00638e0 con 0x7f64bc0b2910 2024-05-03T21:21:15.495 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:21:15 smithi082 ceph-mon[32933]: pgmap v364: 97 pgs: 97 active+clean; 632 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:21:15.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:15.493+0000 7f64c6ffd700 1 -- 172.21.15.67:0/1757508496 <== mgr.14150 v2:172.21.15.67:6800/604451592 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+532 (secure 0 0 0) 0x7f64d00638e0 con 0x7f64bc0b2910 2024-05-03T21:21:15.496 INFO:teuthology.orchestra.run.smithi067.stdout:HOST PATH TYPE DEVICE ID SIZE AVAILABLE REFRESHED REJECT REASONS 2024-05-03T21:21:15.496 INFO:teuthology.orchestra.run.smithi067.stdout:smithi067 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_CVFT53440042400BGN 372G No 4m ago Has a FileSystem, LVM detected 2024-05-03T21:21:15.496 INFO:teuthology.orchestra.run.smithi067.stdout:smithi082 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_PHFT620400SD400BGN 372G No 3m ago Has a FileSystem, LVM detected 2024-05-03T21:21:15.496 INFO:teuthology.orchestra.run.smithi067.stdout:smithi138 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_CVFT6161003C400BGN 372G No 3m ago Has a FileSystem, LVM detected 2024-05-03T21:21:15.498 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:15.496+0000 7f64d7d4d700 1 -- 172.21.15.67:0/1757508496 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f64bc0b2910 msgr2=0x7f64bc0b4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:21:15.498 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:15.496+0000 7f64d7d4d700 1 --2- 172.21.15.67:0/1757508496 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f64bc0b2910 0x7f64bc0b4dd0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f64cc005fd0 tx=0x7f64cc005de0 comp rx=0 tx=0).stop 2024-05-03T21:21:15.498 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:15.496+0000 7f64d7d4d700 1 -- 172.21.15.67:0/1757508496 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f64d0104710 msgr2=0x7f64d0199d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:21:15.498 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:15.496+0000 7f64d7d4d700 1 --2- 172.21.15.67:0/1757508496 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f64d0104710 0x7f64d0199d50 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f64c0004380 tx=0x7f64c0003730 comp rx=0 tx=0).stop 2024-05-03T21:21:15.499 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:15.496+0000 7f64d7d4d700 1 -- 172.21.15.67:0/1757508496 shutdown_connections 2024-05-03T21:21:15.499 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:15.496+0000 7f64d7d4d700 1 --2- 172.21.15.67:0/1757508496 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f64bc0b2910 0x7f64bc0b4dd0 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:21:15.499 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:15.496+0000 7f64d7d4d700 1 --2- 172.21.15.67:0/1757508496 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f64d01050b0 0x7f64d01a0dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:21:15.499 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:15.496+0000 7f64d7d4d700 1 --2- 172.21.15.67:0/1757508496 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f64d0103510 0x7f64d0199810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:21:15.499 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:15.496+0000 7f64d7d4d700 1 --2- 172.21.15.67:0/1757508496 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f64d0104710 0x7f64d0199d50 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:21:15.499 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:15.496+0000 7f64d7d4d700 1 -- 172.21.15.67:0/1757508496 >> 172.21.15.67:0/1757508496 conn(0x7f64d00fea70 msgr2=0x7f64d0107a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T21:21:15.499 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:15.496+0000 7f64d7d4d700 1 -- 172.21.15.67:0/1757508496 shutdown_connections 2024-05-03T21:21:15.499 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:15.496+0000 7f64d7d4d700 1 -- 172.21.15.67:0/1757508496 wait complete. 2024-05-03T21:21:15.574 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:21:15 smithi138 ceph-mon[33110]: pgmap v364: 97 pgs: 97 active+clean; 632 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:21:16.157 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-05-03T21:21:16.167 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi067.front.sepia.ceph.com 2024-05-03T21:21:16.167 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9a047fe7599a549042d06a3fb5b6813a203eabea shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 713bc57a-0991-11ef-bc97-c7b262605968 -- bash -c 'ceph fs dump' 2024-05-03T21:21:16.419 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/713bc57a-0991-11ef-bc97-c7b262605968/mon.a/config 2024-05-03T21:21:17.496 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:21:17 smithi082 ceph-mon[32933]: from='client.15036 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-05-03T21:21:17.496 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:21:17 smithi082 ceph-mon[32933]: pgmap v365: 97 pgs: 97 active+clean; 632 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:21:17.499 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:21:17 smithi067 ceph-mon[29730]: from='client.15036 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-05-03T21:21:17.499 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:21:17 smithi067 ceph-mon[29730]: pgmap v365: 97 pgs: 97 active+clean; 632 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:21:17.574 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:21:17 smithi138 ceph-mon[33110]: from='client.15036 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-05-03T21:21:17.574 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:21:17 smithi138 ceph-mon[33110]: pgmap v365: 97 pgs: 97 active+clean; 632 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:21:18.038 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:18.033+0000 7f6736085700 1 -- 172.21.15.67:0/1283989507 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6730109210 msgr2=0x7f673010b680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:21:18.038 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:18.033+0000 7f6736085700 1 --2- 172.21.15.67:0/1283989507 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6730109210 0x7f673010b680 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f672800b0a0 tx=0x7f672800b3b0 comp rx=0 tx=0).stop 2024-05-03T21:21:18.038 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:18.033+0000 7f6736085700 1 -- 172.21.15.67:0/1283989507 shutdown_connections 2024-05-03T21:21:18.038 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:18.033+0000 7f6736085700 1 --2- 172.21.15.67:0/1283989507 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6730109210 0x7f673010b680 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:21:18.038 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:18.033+0000 7f6736085700 1 --2- 172.21.15.67:0/1283989507 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f6730103fb0 0x7f67301063a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:21:18.038 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:18.033+0000 7f6736085700 1 --2- 172.21.15.67:0/1283989507 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f67301068e0 0x7f6730108cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:21:18.038 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:18.033+0000 7f6736085700 1 -- 172.21.15.67:0/1283989507 >> 172.21.15.67:0/1283989507 conn(0x7f67300fd9c0 msgr2=0x7f67300ffe00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T21:21:18.038 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:18.033+0000 7f6736085700 1 -- 172.21.15.67:0/1283989507 shutdown_connections 2024-05-03T21:21:18.038 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:18.033+0000 7f6736085700 1 -- 172.21.15.67:0/1283989507 wait complete. 2024-05-03T21:21:18.038 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:18.033+0000 7f6736085700 1 Processor -- start 2024-05-03T21:21:18.039 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:18.034+0000 7f6736085700 1 -- start start 2024-05-03T21:21:18.039 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:18.034+0000 7f6736085700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f6730103fb0 0x7f6730199760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:21:18.039 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:18.034+0000 7f6736085700 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f67301068e0 0x7f6730199ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:21:18.039 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:18.034+0000 7f6736085700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6730109210 0x7f67301a0d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:21:18.039 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:18.034+0000 7f6736085700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f673010f230 con 0x7f6730109210 2024-05-03T21:21:18.039 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:18.034+0000 7f6736085700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f673010f0b0 con 0x7f6730103fb0 2024-05-03T21:21:18.039 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:18.034+0000 7f6736085700 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f673010f3b0 con 0x7f67301068e0 2024-05-03T21:21:18.039 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:18.034+0000 7f672ffff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6730109210 0x7f67301a0d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:21:18.039 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:18.034+0000 7f672ffff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6730109210 0x7f67301a0d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:41288/0 (socket says 172.21.15.67:41288) 2024-05-03T21:21:18.039 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:18.034+0000 7f672ffff700 1 -- 172.21.15.67:0/1713041358 learned_addr learned my addr 172.21.15.67:0/1713041358 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-03T21:21:18.039 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:18.034+0000 7f672effd700 1 --2- 172.21.15.67:0/1713041358 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f67301068e0 0x7f6730199ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:21:18.039 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:18.034+0000 7f672f7fe700 1 --2- 172.21.15.67:0/1713041358 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f6730103fb0 0x7f6730199760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:21:18.039 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:18.034+0000 7f672ffff700 1 -- 172.21.15.67:0/1713041358 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f67301068e0 msgr2=0x7f6730199ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:21:18.039 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:18.034+0000 7f672ffff700 1 --2- 172.21.15.67:0/1713041358 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f67301068e0 0x7f6730199ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:21:18.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:18.034+0000 7f672ffff700 1 -- 172.21.15.67:0/1713041358 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f6730103fb0 msgr2=0x7f6730199760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:21:18.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:18.034+0000 7f672ffff700 1 --2- 172.21.15.67:0/1713041358 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f6730103fb0 0x7f6730199760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:21:18.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:18.034+0000 7f672ffff700 1 -- 172.21.15.67:0/1713041358 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f67301a13c0 con 0x7f6730109210 2024-05-03T21:21:18.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:18.035+0000 7f672ffff700 1 --2- 172.21.15.67:0/1713041358 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6730109210 0x7f67301a0d20 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f6728007d10 tx=0x7f6728007d40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T21:21:18.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:18.035+0000 7f672cff9700 1 -- 172.21.15.67:0/1713041358 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6728021020 con 0x7f6730109210 2024-05-03T21:21:18.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:18.035+0000 7f672cff9700 1 -- 172.21.15.67:0/1713041358 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(28 keys) ==== 1170+0+0 (secure 0 0 0) 0x7f672801d970 con 0x7f6730109210 2024-05-03T21:21:18.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:18.035+0000 7f6736085700 1 -- 172.21.15.67:0/1713041358 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f67301a1650 con 0x7f6730109210 2024-05-03T21:21:18.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:18.035+0000 7f672cff9700 1 -- 172.21.15.67:0/1713041358 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67280054e0 con 0x7f6730109210 2024-05-03T21:21:18.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:18.035+0000 7f6736085700 1 -- 172.21.15.67:0/1713041358 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f67301a1ab0 con 0x7f6730109210 2024-05-03T21:21:18.041 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:18.036+0000 7f6736085700 1 -- 172.21.15.67:0/1713041358 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f673006a190 con 0x7f6730109210 2024-05-03T21:21:18.041 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:18.037+0000 7f672cff9700 1 -- 172.21.15.67:0/1713041358 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 17) ==== 149703+0+0 (secure 0 0 0) 0x7f6728024020 con 0x7f6730109210 2024-05-03T21:21:18.041 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:18.038+0000 7f672cff9700 1 --2- 172.21.15.67:0/1713041358 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f67180b2910 0x7f67180b4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:21:18.041 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:18.038+0000 7f672f7fe700 1 --2- 172.21.15.67:0/1713041358 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f67180b2910 0x7f67180b4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:21:18.042 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:18.038+0000 7f672cff9700 1 -- 172.21.15.67:0/1713041358 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 8211+0+0 (secure 0 0 0) 0x7f67280e3b10 con 0x7f6730109210 2024-05-03T21:21:18.042 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:18.038+0000 7f672f7fe700 1 --2- 172.21.15.67:0/1713041358 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f67180b2910 0x7f67180b4dd0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f6720007c60 tx=0x7f67200073d0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T21:21:18.043 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:18.041+0000 7f672cff9700 1 -- 172.21.15.67:0/1713041358 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191839 (secure 0 0 0) 0x7f672802b020 con 0x7f6730109210 2024-05-03T21:21:18.301 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:18.299+0000 7f6736085700 1 -- 172.21.15.67:0/1713041358 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump"} v 0) -- 0x7f673002e540 con 0x7f6730109210 2024-05-03T21:21:18.302 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:18.300+0000 7f672cff9700 1 -- 172.21.15.67:0/1713041358 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 25 v25) ==== 76+0+3048 (secure 0 0 0) 0x7f67280ab770 con 0x7f6730109210 2024-05-03T21:21:18.302 INFO:teuthology.orchestra.run.smithi067.stdout:e25 2024-05-03T21:21:18.302 INFO:teuthology.orchestra.run.smithi067.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-05-03T21:21:18.302 INFO:teuthology.orchestra.run.smithi067.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-05-03T21:21:18.302 INFO:teuthology.orchestra.run.smithi067.stdout:legacy client fscid: 1 2024-05-03T21:21:18.303 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-05-03T21:21:18.303 INFO:teuthology.orchestra.run.smithi067.stdout:Filesystem 'cephfs' (1) 2024-05-03T21:21:18.303 INFO:teuthology.orchestra.run.smithi067.stdout:fs_name cephfs 2024-05-03T21:21:18.303 INFO:teuthology.orchestra.run.smithi067.stdout:epoch 25 2024-05-03T21:21:18.303 INFO:teuthology.orchestra.run.smithi067.stdout:flags 32 joinable allow_snaps allow_multimds_snaps allow_standby_replay 2024-05-03T21:21:18.303 INFO:teuthology.orchestra.run.smithi067.stdout:created 2024-05-03T21:18:44.665825+0000 2024-05-03T21:21:18.303 INFO:teuthology.orchestra.run.smithi067.stdout:modified 2024-05-03T21:19:03.775546+0000 2024-05-03T21:21:18.303 INFO:teuthology.orchestra.run.smithi067.stdout:tableserver 0 2024-05-03T21:21:18.303 INFO:teuthology.orchestra.run.smithi067.stdout:root 0 2024-05-03T21:21:18.303 INFO:teuthology.orchestra.run.smithi067.stdout:session_timeout 300 2024-05-03T21:21:18.303 INFO:teuthology.orchestra.run.smithi067.stdout:session_autoclose 300 2024-05-03T21:21:18.303 INFO:teuthology.orchestra.run.smithi067.stdout:max_file_size 1099511627776 2024-05-03T21:21:18.303 INFO:teuthology.orchestra.run.smithi067.stdout:max_xattr_size 65536 2024-05-03T21:21:18.303 INFO:teuthology.orchestra.run.smithi067.stdout:required_client_features {} 2024-05-03T21:21:18.303 INFO:teuthology.orchestra.run.smithi067.stdout:last_failure 0 2024-05-03T21:21:18.304 INFO:teuthology.orchestra.run.smithi067.stdout:last_failure_osd_epoch 0 2024-05-03T21:21:18.304 INFO:teuthology.orchestra.run.smithi067.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-05-03T21:21:18.304 INFO:teuthology.orchestra.run.smithi067.stdout:max_mds 3 2024-05-03T21:21:18.304 INFO:teuthology.orchestra.run.smithi067.stdout:in 0,1,2 2024-05-03T21:21:18.304 INFO:teuthology.orchestra.run.smithi067.stdout:up {0=24473,1=24449,2=24425} 2024-05-03T21:21:18.304 INFO:teuthology.orchestra.run.smithi067.stdout:failed 2024-05-03T21:21:18.304 INFO:teuthology.orchestra.run.smithi067.stdout:damaged 2024-05-03T21:21:18.304 INFO:teuthology.orchestra.run.smithi067.stdout:stopped 2024-05-03T21:21:18.304 INFO:teuthology.orchestra.run.smithi067.stdout:data_pools [3] 2024-05-03T21:21:18.304 INFO:teuthology.orchestra.run.smithi067.stdout:metadata_pool 2 2024-05-03T21:21:18.304 INFO:teuthology.orchestra.run.smithi067.stdout:inline_data disabled 2024-05-03T21:21:18.304 INFO:teuthology.orchestra.run.smithi067.stdout:balancer 2024-05-03T21:21:18.304 INFO:teuthology.orchestra.run.smithi067.stdout:bal_rank_mask -1 2024-05-03T21:21:18.304 INFO:teuthology.orchestra.run.smithi067.stdout:standby_count_wanted 0 2024-05-03T21:21:18.304 INFO:teuthology.orchestra.run.smithi067.stdout:qdb_cluster leader: 24473 members: 24449,24473,24425 2024-05-03T21:21:18.305 INFO:teuthology.orchestra.run.smithi067.stdout:[mds.b{0:24473} state up:active seq 5 addr [v2:172.21.15.82:6838/3644499578,v1:172.21.15.82:6839/3644499578] compat {c=[1],r=[1],i=[1fff]}] 2024-05-03T21:21:18.305 INFO:teuthology.orchestra.run.smithi067.stdout:[mds.i{0:24418} state up:standby-replay seq 1 addr [v2:172.21.15.138:6838/334382443,v1:172.21.15.138:6839/334382443] compat {c=[1],r=[1],i=[1fff]}] 2024-05-03T21:21:18.305 INFO:teuthology.orchestra.run.smithi067.stdout:[mds.h{1:24449} state up:active seq 11 addr [v2:172.21.15.82:6836/1275987852,v1:172.21.15.82:6837/1275987852] compat {c=[1],r=[1],i=[1fff]}] 2024-05-03T21:21:18.305 INFO:teuthology.orchestra.run.smithi067.stdout:[mds.k{1:24407} state up:standby-replay seq 1 addr [v2:172.21.15.82:6832/1127287628,v1:172.21.15.82:6833/1127287628] compat {c=[1],r=[1],i=[1fff]}] 2024-05-03T21:21:18.305 INFO:teuthology.orchestra.run.smithi067.stdout:[mds.e{2:24425} state up:active seq 14 addr [v2:172.21.15.82:6834/280518761,v1:172.21.15.82:6835/280518761] compat {c=[1],r=[1],i=[1fff]}] 2024-05-03T21:21:18.305 INFO:teuthology.orchestra.run.smithi067.stdout:[mds.c{2:24406} state up:standby-replay seq 1 addr [v2:172.21.15.138:6836/2203273249,v1:172.21.15.138:6837/2203273249] compat {c=[1],r=[1],i=[1fff]}] 2024-05-03T21:21:18.305 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-05-03T21:21:18.305 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-05-03T21:21:18.305 INFO:teuthology.orchestra.run.smithi067.stdout:Standby daemons: 2024-05-03T21:21:18.305 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-05-03T21:21:18.305 INFO:teuthology.orchestra.run.smithi067.stdout:[mds.j{-1:14541} state up:standby seq 1 addr [v2:172.21.15.67:6834/4224826546,v1:172.21.15.67:6835/4224826546] compat {c=[1],r=[1],i=[1fff]}] 2024-05-03T21:21:18.305 INFO:teuthology.orchestra.run.smithi067.stdout:[mds.d{-1:14553} state up:standby seq 1 addr [v2:172.21.15.67:6836/2485231621,v1:172.21.15.67:6837/2485231621] compat {c=[1],r=[1],i=[1fff]}] 2024-05-03T21:21:18.305 INFO:teuthology.orchestra.run.smithi067.stdout:[mds.g{-1:14598} state up:standby seq 1 addr [v2:172.21.15.67:6838/2106078262,v1:172.21.15.67:6839/2106078262] compat {c=[1],r=[1],i=[1fff]}] 2024-05-03T21:21:18.305 INFO:teuthology.orchestra.run.smithi067.stdout:[mds.a{-1:14610} state up:standby seq 1 addr [v2:172.21.15.67:6840/1559196215,v1:172.21.15.67:6841/1559196215] compat {c=[1],r=[1],i=[1fff]}] 2024-05-03T21:21:18.305 INFO:teuthology.orchestra.run.smithi067.stdout:[mds.l{-1:24376} state up:standby seq 1 addr [v2:172.21.15.138:6832/3028022602,v1:172.21.15.138:6833/3028022602] compat {c=[1],r=[1],i=[1fff]}] 2024-05-03T21:21:18.306 INFO:teuthology.orchestra.run.smithi067.stdout:[mds.f{-1:24400} state up:standby seq 1 addr [v2:172.21.15.138:6834/3342864697,v1:172.21.15.138:6835/3342864697] compat {c=[1],r=[1],i=[1fff]}] 2024-05-03T21:21:18.306 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:18.302+0000 7f6736085700 1 -- 172.21.15.67:0/1713041358 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f67180b2910 msgr2=0x7f67180b4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:21:18.306 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:18.302+0000 7f6736085700 1 --2- 172.21.15.67:0/1713041358 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f67180b2910 0x7f67180b4dd0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f6720007c60 tx=0x7f67200073d0 comp rx=0 tx=0).stop 2024-05-03T21:21:18.306 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:18.302+0000 7f6736085700 1 -- 172.21.15.67:0/1713041358 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6730109210 msgr2=0x7f67301a0d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:21:18.307 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:18.302+0000 7f6736085700 1 --2- 172.21.15.67:0/1713041358 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6730109210 0x7f67301a0d20 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f6728007d10 tx=0x7f6728007d40 comp rx=0 tx=0).stop 2024-05-03T21:21:18.307 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:18.302+0000 7f6736085700 1 -- 172.21.15.67:0/1713041358 shutdown_connections 2024-05-03T21:21:18.307 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:18.302+0000 7f6736085700 1 --2- 172.21.15.67:0/1713041358 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f67180b2910 0x7f67180b4dd0 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:21:18.307 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:18.302+0000 7f6736085700 1 --2- 172.21.15.67:0/1713041358 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6730109210 0x7f67301a0d20 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:21:18.307 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:18.302+0000 7f6736085700 1 --2- 172.21.15.67:0/1713041358 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f6730103fb0 0x7f6730199760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:21:18.307 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:18.302+0000 7f6736085700 1 --2- 172.21.15.67:0/1713041358 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f67301068e0 0x7f6730199ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:21:18.307 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:18.302+0000 7f6736085700 1 -- 172.21.15.67:0/1713041358 >> 172.21.15.67:0/1713041358 conn(0x7f67300fd9c0 msgr2=0x7f67301048f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T21:21:18.307 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:18.302+0000 7f6736085700 1 -- 172.21.15.67:0/1713041358 shutdown_connections 2024-05-03T21:21:18.307 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:18.302+0000 7f6736085700 1 -- 172.21.15.67:0/1713041358 wait complete. 2024-05-03T21:21:18.307 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 25 2024-05-03T21:21:18.984 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9a047fe7599a549042d06a3fb5b6813a203eabea shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 713bc57a-0991-11ef-bc97-c7b262605968 -- bash -c 'ceph osd dump' 2024-05-03T21:21:19.235 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/713bc57a-0991-11ef-bc97-c7b262605968/mon.a/config 2024-05-03T21:21:19.493 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:21:19 smithi067 ceph-mon[29730]: pgmap v366: 97 pgs: 97 active+clean; 632 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 6 op/s 2024-05-03T21:21:19.493 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:21:19 smithi067 ceph-mon[29730]: from='client.? 172.21.15.67:0/1713041358' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-05-03T21:21:19.495 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:21:19 smithi082 ceph-mon[32933]: pgmap v366: 97 pgs: 97 active+clean; 632 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 6 op/s 2024-05-03T21:21:19.495 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:21:19 smithi082 ceph-mon[32933]: from='client.? 172.21.15.67:0/1713041358' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-05-03T21:21:19.575 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:21:19 smithi138 ceph-mon[33110]: pgmap v366: 97 pgs: 97 active+clean; 632 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 6 op/s 2024-05-03T21:21:19.575 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:21:19 smithi138 ceph-mon[33110]: from='client.? 172.21.15.67:0/1713041358' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-05-03T21:21:20.805 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:20.803+0000 7f0236e9d700 1 -- 172.21.15.67:0/2646190003 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0230103fc0 msgr2=0x7f023010a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:21:20.805 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:20.803+0000 7f0236e9d700 1 --2- 172.21.15.67:0/2646190003 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0230103fc0 0x7f023010a870 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f022800b0d0 tx=0x7f022800b3e0 comp rx=0 tx=0).stop 2024-05-03T21:21:20.805 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:20.803+0000 7f0236e9d700 1 -- 172.21.15.67:0/2646190003 shutdown_connections 2024-05-03T21:21:20.805 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:20.803+0000 7f0236e9d700 1 --2- 172.21.15.67:0/2646190003 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0230103fc0 0x7f023010a870 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:21:20.805 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:20.803+0000 7f0236e9d700 1 --2- 172.21.15.67:0/2646190003 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f0230102420 0x7f0230102840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:21:20.806 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:20.803+0000 7f0236e9d700 1 --2- 172.21.15.67:0/2646190003 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f0230103620 0x7f0230103a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:21:20.806 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:20.803+0000 7f0236e9d700 1 -- 172.21.15.67:0/2646190003 >> 172.21.15.67:0/2646190003 conn(0x7f02300fd9c0 msgr2=0x7f02300ffe00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T21:21:20.806 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:20.803+0000 7f0236e9d700 1 -- 172.21.15.67:0/2646190003 shutdown_connections 2024-05-03T21:21:20.806 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:20.803+0000 7f0236e9d700 1 -- 172.21.15.67:0/2646190003 wait complete. 2024-05-03T21:21:20.808 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:20.803+0000 7f0236e9d700 1 Processor -- start 2024-05-03T21:21:20.808 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:20.803+0000 7f0236e9d700 1 -- start start 2024-05-03T21:21:20.808 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:20.804+0000 7f0236e9d700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0230102420 0x7f02301063b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:21:20.808 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:20.804+0000 7f0236e9d700 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f0230103620 0x7f0230104a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:21:20.808 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:20.804+0000 7f0236e9d700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f0230103fc0 0x7f0230104f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:21:20.808 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:20.804+0000 7f0236e9d700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f023010e140 con 0x7f0230102420 2024-05-03T21:21:20.808 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:20.804+0000 7f0236e9d700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f023010dfc0 con 0x7f0230103fc0 2024-05-03T21:21:20.808 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:20.804+0000 7f0236e9d700 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f023010e2c0 con 0x7f0230103620 2024-05-03T21:21:20.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:20.804+0000 7f0234c39700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0230102420 0x7f02301063b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:21:20.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:20.804+0000 7f0234c39700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0230102420 0x7f02301063b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:41310/0 (socket says 172.21.15.67:41310) 2024-05-03T21:21:20.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:20.804+0000 7f0234c39700 1 -- 172.21.15.67:0/793294113 learned_addr learned my addr 172.21.15.67:0/793294113 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-03T21:21:20.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:20.804+0000 7f022ffff700 1 --2- 172.21.15.67:0/793294113 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f0230103620 0x7f0230104a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:21:20.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:20.804+0000 7f023543a700 1 --2- 172.21.15.67:0/793294113 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f0230103fc0 0x7f0230104f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:21:20.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:20.804+0000 7f0234c39700 1 -- 172.21.15.67:0/793294113 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f0230103620 msgr2=0x7f0230104a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:21:20.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:20.804+0000 7f0234c39700 1 --2- 172.21.15.67:0/793294113 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f0230103620 0x7f0230104a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:21:20.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:20.804+0000 7f0234c39700 1 -- 172.21.15.67:0/793294113 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f0230103fc0 msgr2=0x7f0230104f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:21:20.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:20.804+0000 7f0234c39700 1 --2- 172.21.15.67:0/793294113 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f0230103fc0 0x7f0230104f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:21:20.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:20.804+0000 7f0234c39700 1 -- 172.21.15.67:0/793294113 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f02301057f0 con 0x7f0230102420 2024-05-03T21:21:20.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:20.804+0000 7f0234c39700 1 --2- 172.21.15.67:0/793294113 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0230102420 0x7f02301063b0 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f022000d980 tx=0x7f022000dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T21:21:20.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:20.804+0000 7f022dffb700 1 -- 172.21.15.67:0/793294113 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0220017070 con 0x7f0230102420 2024-05-03T21:21:20.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:20.805+0000 7f0236e9d700 1 -- 172.21.15.67:0/793294113 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f02301a3900 con 0x7f0230102420 2024-05-03T21:21:20.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:20.805+0000 7f022dffb700 1 -- 172.21.15.67:0/793294113 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(28 keys) ==== 1170+0+0 (secure 0 0 0) 0x7f022000fd90 con 0x7f0230102420 2024-05-03T21:21:20.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:20.805+0000 7f022dffb700 1 -- 172.21.15.67:0/793294113 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f022000e610 con 0x7f0230102420 2024-05-03T21:21:20.810 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:20.805+0000 7f0236e9d700 1 -- 172.21.15.67:0/793294113 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f02301a3dc0 con 0x7f0230102420 2024-05-03T21:21:20.810 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:20.806+0000 7f0236e9d700 1 -- 172.21.15.67:0/793294113 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f023006a190 con 0x7f0230102420 2024-05-03T21:21:20.810 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:20.807+0000 7f022dffb700 1 -- 172.21.15.67:0/793294113 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 17) ==== 149703+0+0 (secure 0 0 0) 0x7f0220010460 con 0x7f0230102420 2024-05-03T21:21:20.811 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:20.808+0000 7f022dffb700 1 --2- 172.21.15.67:0/793294113 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f02180b2910 0x7f02180b4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:21:20.811 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:20.808+0000 7f022ffff700 1 --2- 172.21.15.67:0/793294113 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f02180b2910 0x7f02180b4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:21:20.811 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:20.808+0000 7f022dffb700 1 -- 172.21.15.67:0/793294113 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 8211+0+0 (secure 0 0 0) 0x7f02200e1e90 con 0x7f0230102420 2024-05-03T21:21:20.811 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:20.808+0000 7f022ffff700 1 --2- 172.21.15.67:0/793294113 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f02180b2910 0x7f02180b4dd0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f0224005950 tx=0x7f022400b410 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T21:21:20.814 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:20.812+0000 7f022dffb700 1 -- 172.21.15.67:0/793294113 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191839 (secure 0 0 0) 0x7f02200a9a60 con 0x7f0230102420 2024-05-03T21:21:21.031 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:21.028+0000 7f0236e9d700 1 -- 172.21.15.67:0/793294113 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump"} v 0) -- 0x7f02301a4150 con 0x7f0230102420 2024-05-03T21:21:21.031 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:21.029+0000 7f022dffb700 1 -- 172.21.15.67:0/793294113 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump"}]=0 v84) ==== 56+0+5276 (secure 0 0 0) 0x7f02200a9360 con 0x7f0230102420 2024-05-03T21:21:21.031 INFO:teuthology.orchestra.run.smithi067.stdout:epoch 84 2024-05-03T21:21:21.032 INFO:teuthology.orchestra.run.smithi067.stdout:fsid 713bc57a-0991-11ef-bc97-c7b262605968 2024-05-03T21:21:21.032 INFO:teuthology.orchestra.run.smithi067.stdout:created 2024-05-03T21:10:34.070178+0000 2024-05-03T21:21:21.032 INFO:teuthology.orchestra.run.smithi067.stdout:modified 2024-05-03T21:19:34.967839+0000 2024-05-03T21:21:21.032 INFO:teuthology.orchestra.run.smithi067.stdout:flags sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit 2024-05-03T21:21:21.032 INFO:teuthology.orchestra.run.smithi067.stdout:crush_version 26 2024-05-03T21:21:21.032 INFO:teuthology.orchestra.run.smithi067.stdout:full_ratio 0.95 2024-05-03T21:21:21.032 INFO:teuthology.orchestra.run.smithi067.stdout:backfillfull_ratio 0.9 2024-05-03T21:21:21.032 INFO:teuthology.orchestra.run.smithi067.stdout:nearfull_ratio 0.85 2024-05-03T21:21:21.032 INFO:teuthology.orchestra.run.smithi067.stdout:require_min_compat_client luminous 2024-05-03T21:21:21.032 INFO:teuthology.orchestra.run.smithi067.stdout:min_compat_client luminous 2024-05-03T21:21:21.032 INFO:teuthology.orchestra.run.smithi067.stdout:require_osd_release squid 2024-05-03T21:21:21.033 INFO:teuthology.orchestra.run.smithi067.stdout:stretch_mode_enabled false 2024-05-03T21:21:21.033 INFO:teuthology.orchestra.run.smithi067.stdout:pool 1 '.mgr' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode off last_change 22 flags hashpspool stripe_width 0 pg_num_max 32 pg_num_min 1 application mgr read_balance_score 12.00 2024-05-03T21:21:21.033 INFO:teuthology.orchestra.run.smithi067.stdout:pool 2 'cephfs_metadata' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 82 flags hashpspool stripe_width 0 pg_autoscale_bias 4 pg_num_min 16 recovery_priority 5 application cephfs read_balance_score 1.88 2024-05-03T21:21:21.033 INFO:teuthology.orchestra.run.smithi067.stdout:pool 3 'cephfs_data' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 64 pgp_num 64 autoscale_mode off last_change 82 flags hashpspool stripe_width 0 pg_num_min 64 target_size_ratio 0.9 application cephfs read_balance_score 1.50 2024-05-03T21:21:21.033 INFO:teuthology.orchestra.run.smithi067.stdout:max_osd 12 2024-05-03T21:21:21.033 INFO:teuthology.orchestra.run.smithi067.stdout:osd.0 up in weight 1 up_from 8 up_thru 79 down_at 0 last_clean_interval [0,0) [v2:172.21.15.67:6802/3692483807,v1:172.21.15.67:6803/3692483807] [v2:172.21.15.67:6804/3692483807,v1:172.21.15.67:6805/3692483807] exists,up 7939f31c-6fb7-42bd-af96-88a448d11bd2 2024-05-03T21:21:21.033 INFO:teuthology.orchestra.run.smithi067.stdout:osd.1 up in weight 1 up_from 13 up_thru 79 down_at 0 last_clean_interval [0,0) [v2:172.21.15.82:6800/1139376397,v1:172.21.15.82:6801/1139376397] [v2:172.21.15.82:6802/1139376397,v1:172.21.15.82:6803/1139376397] exists,up 71858193-05a6-4c2c-9bb7-bc3d811d634e 2024-05-03T21:21:21.033 INFO:teuthology.orchestra.run.smithi067.stdout:osd.2 up in weight 1 up_from 18 up_thru 83 down_at 0 last_clean_interval [0,0) [v2:172.21.15.138:6800/2774330123,v1:172.21.15.138:6801/2774330123] [v2:172.21.15.138:6802/2774330123,v1:172.21.15.138:6803/2774330123] exists,up f9ece78d-6a19-4d46-9022-6ed184074745 2024-05-03T21:21:21.033 INFO:teuthology.orchestra.run.smithi067.stdout:osd.3 up in weight 1 up_from 27 up_thru 79 down_at 0 last_clean_interval [0,0) [v2:172.21.15.67:6810/3915934652,v1:172.21.15.67:6811/3915934652] [v2:172.21.15.67:6812/3915934652,v1:172.21.15.67:6813/3915934652] exists,up 3616df71-e7b3-4687-aaba-37b0b7766a3c 2024-05-03T21:21:21.033 INFO:teuthology.orchestra.run.smithi067.stdout:osd.4 up in weight 1 up_from 33 up_thru 79 down_at 0 last_clean_interval [0,0) [v2:172.21.15.82:6808/4031818951,v1:172.21.15.82:6809/4031818951] [v2:172.21.15.82:6810/4031818951,v1:172.21.15.82:6811/4031818951] exists,up c3fd0ffd-e23f-4279-b679-9d95a144cf32 2024-05-03T21:21:21.033 INFO:teuthology.orchestra.run.smithi067.stdout:osd.5 up in weight 1 up_from 39 up_thru 79 down_at 0 last_clean_interval [0,0) [v2:172.21.15.138:6808/2521889499,v1:172.21.15.138:6809/2521889499] [v2:172.21.15.138:6810/2521889499,v1:172.21.15.138:6811/2521889499] exists,up 540ef0d9-a6dc-43c6-b09e-e87340ba24cf 2024-05-03T21:21:21.033 INFO:teuthology.orchestra.run.smithi067.stdout:osd.6 up in weight 1 up_from 45 up_thru 83 down_at 0 last_clean_interval [0,0) [v2:172.21.15.67:6818/3269682731,v1:172.21.15.67:6819/3269682731] [v2:172.21.15.67:6820/3269682731,v1:172.21.15.67:6821/3269682731] exists,up 49a89fd0-958a-4be9-a8db-6022f946153e 2024-05-03T21:21:21.034 INFO:teuthology.orchestra.run.smithi067.stdout:osd.7 up in weight 1 up_from 50 up_thru 83 down_at 0 last_clean_interval [0,0) [v2:172.21.15.82:6816/3008589976,v1:172.21.15.82:6817/3008589976] [v2:172.21.15.82:6818/3008589976,v1:172.21.15.82:6819/3008589976] exists,up 8fa4cd64-ca6b-48ae-8db0-d369ce6102ac 2024-05-03T21:21:21.034 INFO:teuthology.orchestra.run.smithi067.stdout:osd.8 up in weight 1 up_from 57 up_thru 83 down_at 0 last_clean_interval [0,0) [v2:172.21.15.138:6816/2223920070,v1:172.21.15.138:6817/2223920070] [v2:172.21.15.138:6818/2223920070,v1:172.21.15.138:6819/2223920070] exists,up fae39087-6c38-4771-a2b9-3f4fecb08edd 2024-05-03T21:21:21.034 INFO:teuthology.orchestra.run.smithi067.stdout:osd.9 up in weight 1 up_from 63 up_thru 83 down_at 0 last_clean_interval [0,0) [v2:172.21.15.67:6826/3939417129,v1:172.21.15.67:6827/3939417129] [v2:172.21.15.67:6828/3939417129,v1:172.21.15.67:6829/3939417129] exists,up 3d387dbf-0163-41b7-98af-4fc45ed73ad4 2024-05-03T21:21:21.034 INFO:teuthology.orchestra.run.smithi067.stdout:osd.10 up in weight 1 up_from 68 up_thru 79 down_at 0 last_clean_interval [0,0) [v2:172.21.15.82:6824/2505507029,v1:172.21.15.82:6825/2505507029] [v2:172.21.15.82:6826/2505507029,v1:172.21.15.82:6827/2505507029] exists,up db0305a0-a83b-474b-837a-fbc239ac1738 2024-05-03T21:21:21.034 INFO:teuthology.orchestra.run.smithi067.stdout:osd.11 up in weight 1 up_from 74 up_thru 83 down_at 0 last_clean_interval [0,0) [v2:172.21.15.138:6824/2528685018,v1:172.21.15.138:6825/2528685018] [v2:172.21.15.138:6826/2528685018,v1:172.21.15.138:6827/2528685018] exists,up 479c2747-f28e-4704-87e4-d0c31782e2de 2024-05-03T21:21:21.034 INFO:teuthology.orchestra.run.smithi067.stdout:pg_upmap_items 2.0 [8,4] 2024-05-03T21:21:21.034 INFO:teuthology.orchestra.run.smithi067.stdout:pg_upmap_items 2.a [2,11] 2024-05-03T21:21:21.034 INFO:teuthology.orchestra.run.smithi067.stdout:pg_upmap_items 2.b [2,4] 2024-05-03T21:21:21.034 INFO:teuthology.orchestra.run.smithi067.stdout:pg_upmap_items 2.10 [8,11] 2024-05-03T21:21:21.034 INFO:teuthology.orchestra.run.smithi067.stdout:pg_upmap_items 3.1 [4,9] 2024-05-03T21:21:21.034 INFO:teuthology.orchestra.run.smithi067.stdout:pg_upmap_items 3.1b [5,9] 2024-05-03T21:21:21.034 INFO:teuthology.orchestra.run.smithi067.stdout:pg_upmap_items 3.21 [4,8] 2024-05-03T21:21:21.035 INFO:teuthology.orchestra.run.smithi067.stdout:pg_upmap_items 3.3c [4,9] 2024-05-03T21:21:21.035 INFO:teuthology.orchestra.run.smithi067.stdout:blocklist 172.21.15.67:0/1239643865 expires 2024-05-04T21:11:33.721416+0000 2024-05-03T21:21:21.035 INFO:teuthology.orchestra.run.smithi067.stdout:blocklist 172.21.15.67:6801/2576113038 expires 2024-05-04T21:11:33.721416+0000 2024-05-03T21:21:21.035 INFO:teuthology.orchestra.run.smithi067.stdout:blocklist 172.21.15.67:6800/2576113038 expires 2024-05-04T21:11:33.721416+0000 2024-05-03T21:21:21.035 INFO:teuthology.orchestra.run.smithi067.stdout:blocklist 172.21.15.67:6800/1698171509 expires 2024-05-04T21:11:07.333651+0000 2024-05-03T21:21:21.035 INFO:teuthology.orchestra.run.smithi067.stdout:blocklist 172.21.15.67:0/2217893923 expires 2024-05-04T21:11:33.721416+0000 2024-05-03T21:21:21.035 INFO:teuthology.orchestra.run.smithi067.stdout:blocklist 172.21.15.67:0/20864811 expires 2024-05-04T21:11:07.333651+0000 2024-05-03T21:21:21.035 INFO:teuthology.orchestra.run.smithi067.stdout:blocklist 172.21.15.67:6801/1698171509 expires 2024-05-04T21:11:07.333651+0000 2024-05-03T21:21:21.035 INFO:teuthology.orchestra.run.smithi067.stdout:blocklist 172.21.15.67:0/773376553 expires 2024-05-04T21:11:33.721416+0000 2024-05-03T21:21:21.035 INFO:teuthology.orchestra.run.smithi067.stdout:blocklist 172.21.15.67:0/847549114 expires 2024-05-04T21:11:07.333651+0000 2024-05-03T21:21:21.035 INFO:teuthology.orchestra.run.smithi067.stdout:blocklist 172.21.15.67:0/1291778035 expires 2024-05-04T21:11:07.333651+0000 2024-05-03T21:21:21.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:21.031+0000 7f0236e9d700 1 -- 172.21.15.67:0/793294113 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f02180b2910 msgr2=0x7f02180b4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:21:21.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:21.031+0000 7f0236e9d700 1 --2- 172.21.15.67:0/793294113 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f02180b2910 0x7f02180b4dd0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f0224005950 tx=0x7f022400b410 comp rx=0 tx=0).stop 2024-05-03T21:21:21.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:21.031+0000 7f0236e9d700 1 -- 172.21.15.67:0/793294113 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0230102420 msgr2=0x7f02301063b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:21:21.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:21.031+0000 7f0236e9d700 1 --2- 172.21.15.67:0/793294113 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0230102420 0x7f02301063b0 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f022000d980 tx=0x7f022000dc90 comp rx=0 tx=0).stop 2024-05-03T21:21:21.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:21.031+0000 7f0236e9d700 1 -- 172.21.15.67:0/793294113 shutdown_connections 2024-05-03T21:21:21.037 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:21.031+0000 7f0236e9d700 1 --2- 172.21.15.67:0/793294113 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f02180b2910 0x7f02180b4dd0 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:21:21.037 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:21.031+0000 7f0236e9d700 1 --2- 172.21.15.67:0/793294113 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0230102420 0x7f02301063b0 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:21:21.037 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:21.031+0000 7f0236e9d700 1 --2- 172.21.15.67:0/793294113 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f0230103fc0 0x7f0230104f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:21:21.037 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:21.031+0000 7f0236e9d700 1 --2- 172.21.15.67:0/793294113 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f0230103620 0x7f0230104a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:21:21.037 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:21.031+0000 7f0236e9d700 1 -- 172.21.15.67:0/793294113 >> 172.21.15.67:0/793294113 conn(0x7f02300fd9c0 msgr2=0x7f02300ff340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T21:21:21.037 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:21.032+0000 7f0236e9d700 1 -- 172.21.15.67:0/793294113 shutdown_connections 2024-05-03T21:21:21.037 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:21.032+0000 7f0236e9d700 1 -- 172.21.15.67:0/793294113 wait complete. 2024-05-03T21:21:21.404 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:21:21 smithi067 ceph-mon[29730]: pgmap v367: 97 pgs: 97 active+clean; 632 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-05-03T21:21:21.405 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:21:21 smithi067 ceph-mon[29730]: from='client.? 172.21.15.67:0/793294113' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-05-03T21:21:21.495 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:21:21 smithi082 ceph-mon[32933]: pgmap v367: 97 pgs: 97 active+clean; 632 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-05-03T21:21:21.495 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:21:21 smithi082 ceph-mon[32933]: from='client.? 172.21.15.67:0/793294113' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-05-03T21:21:21.574 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:21:21 smithi138 ceph-mon[33110]: pgmap v367: 97 pgs: 97 active+clean; 632 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-05-03T21:21:21.575 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:21:21 smithi138 ceph-mon[33110]: from='client.? 172.21.15.67:0/793294113' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-05-03T21:21:21.736 INFO:teuthology.run_tasks:Running task fs.ready... 2024-05-03T21:21:21.747 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9a047fe7599a549042d06a3fb5b6813a203eabea shell --fsid 713bc57a-0991-11ef-bc97-c7b262605968 -- ceph fs dump --format=json 2024-05-03T21:21:21.997 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/713bc57a-0991-11ef-bc97-c7b262605968/mon.a/config 2024-05-03T21:21:23.471 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:21:23 smithi067 ceph-mon[29730]: pgmap v368: 97 pgs: 97 active+clean; 632 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-05-03T21:21:23.495 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:21:23 smithi082 ceph-mon[32933]: pgmap v368: 97 pgs: 97 active+clean; 632 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-05-03T21:21:23.574 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:21:23 smithi138 ceph-mon[33110]: pgmap v368: 97 pgs: 97 active+clean; 632 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-05-03T21:21:23.721 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:23.719+0000 7f8c27fff700 1 -- 172.21.15.67:0/3497198347 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8c281086f0 msgr2=0x7f8c281102c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:21:23.721 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:23.719+0000 7f8c27fff700 1 --2- 172.21.15.67:0/3497198347 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8c281086f0 0x7f8c281102c0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f8c2000b0d0 tx=0x7f8c2000b3e0 comp rx=0 tx=0).stop 2024-05-03T21:21:23.722 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:23.719+0000 7f8c27fff700 1 -- 172.21.15.67:0/3497198347 shutdown_connections 2024-05-03T21:21:23.722 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:23.719+0000 7f8c27fff700 1 --2- 172.21.15.67:0/3497198347 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8c281086f0 0x7f8c281102c0 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:21:23.722 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:23.719+0000 7f8c27fff700 1 --2- 172.21.15.67:0/3497198347 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f8c28103610 0x7f8c281039f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:21:23.722 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:23.719+0000 7f8c27fff700 1 --2- 172.21.15.67:0/3497198347 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f8c28103fc0 0x7f8c28108010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:21:23.722 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:23.719+0000 7f8c27fff700 1 -- 172.21.15.67:0/3497198347 >> 172.21.15.67:0/3497198347 conn(0x7f8c280feec0 msgr2=0x7f8c281012e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T21:21:23.722 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:23.719+0000 7f8c27fff700 1 -- 172.21.15.67:0/3497198347 shutdown_connections 2024-05-03T21:21:23.722 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:23.719+0000 7f8c27fff700 1 -- 172.21.15.67:0/3497198347 wait complete. 2024-05-03T21:21:23.724 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:23.720+0000 7f8c27fff700 1 Processor -- start 2024-05-03T21:21:23.724 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:23.720+0000 7f8c27fff700 1 -- start start 2024-05-03T21:21:23.724 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:23.720+0000 7f8c27fff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8c28103610 0x7f8c2819f8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:21:23.724 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:23.720+0000 7f8c27fff700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f8c28103fc0 0x7f8c2819fdf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:21:23.724 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:23.720+0000 7f8c27fff700 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f8c281086f0 0x7f8c28199a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:21:23.724 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:23.720+0000 7f8c27fff700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f8c28113890 con 0x7f8c28103610 2024-05-03T21:21:23.724 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:23.720+0000 7f8c27fff700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f8c28113710 con 0x7f8c28103fc0 2024-05-03T21:21:23.725 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:23.720+0000 7f8c27fff700 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f8c28113a10 con 0x7f8c281086f0 2024-05-03T21:21:23.725 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:23.720+0000 7f8c26ffd700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8c28103610 0x7f8c2819f8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:21:23.725 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:23.720+0000 7f8c26ffd700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8c28103610 0x7f8c2819f8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:41318/0 (socket says 172.21.15.67:41318) 2024-05-03T21:21:23.725 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:23.720+0000 7f8c26ffd700 1 -- 172.21.15.67:0/3693736378 learned_addr learned my addr 172.21.15.67:0/3693736378 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-03T21:21:23.725 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:23.721+0000 7f8c267fc700 1 --2- 172.21.15.67:0/3693736378 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f8c28103fc0 0x7f8c2819fdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:21:23.725 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:23.721+0000 7f8c26ffd700 1 -- 172.21.15.67:0/3693736378 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f8c281086f0 msgr2=0x7f8c28199a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:21:23.725 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:23.721+0000 7f8c26ffd700 1 --2- 172.21.15.67:0/3693736378 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f8c281086f0 0x7f8c28199a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:21:23.725 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:23.721+0000 7f8c26ffd700 1 -- 172.21.15.67:0/3693736378 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f8c28103fc0 msgr2=0x7f8c2819fdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:21:23.725 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:23.721+0000 7f8c26ffd700 1 --2- 172.21.15.67:0/3693736378 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f8c28103fc0 0x7f8c2819fdf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:21:23.725 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:23.721+0000 7f8c26ffd700 1 -- 172.21.15.67:0/3693736378 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8c2819a1d0 con 0x7f8c28103610 2024-05-03T21:21:23.725 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:23.721+0000 7f8c26ffd700 1 --2- 172.21.15.67:0/3693736378 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8c28103610 0x7f8c2819f8b0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f8c1800d980 tx=0x7f8c1800dd40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T21:21:23.725 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:23.721+0000 7f8c2ca02700 1 -- 172.21.15.67:0/3693736378 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c18016070 con 0x7f8c28103610 2024-05-03T21:21:23.725 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:23.721+0000 7f8c27fff700 1 -- 172.21.15.67:0/3693736378 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8c2819a4c0 con 0x7f8c28103610 2024-05-03T21:21:23.726 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:23.721+0000 7f8c2ca02700 1 -- 172.21.15.67:0/3693736378 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(28 keys) ==== 1170+0+0 (secure 0 0 0) 0x7f8c18004520 con 0x7f8c28103610 2024-05-03T21:21:23.726 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:23.721+0000 7f8c2ca02700 1 -- 172.21.15.67:0/3693736378 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c18004e20 con 0x7f8c28103610 2024-05-03T21:21:23.726 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:23.721+0000 7f8c27fff700 1 -- 172.21.15.67:0/3693736378 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8c281a6780 con 0x7f8c28103610 2024-05-03T21:21:23.726 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:23.722+0000 7f8c27fff700 1 -- 172.21.15.67:0/3693736378 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8c2806a190 con 0x7f8c28103610 2024-05-03T21:21:23.726 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:23.724+0000 7f8c2ca02700 1 -- 172.21.15.67:0/3693736378 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 17) ==== 149703+0+0 (secure 0 0 0) 0x7f8c18004020 con 0x7f8c28103610 2024-05-03T21:21:23.727 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:23.725+0000 7f8c2ca02700 1 --2- 172.21.15.67:0/3693736378 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f8c100b2910 0x7f8c100b4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:21:23.727 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:23.725+0000 7f8c267fc700 1 --2- 172.21.15.67:0/3693736378 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f8c100b2910 0x7f8c100b4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:21:23.727 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:23.725+0000 7f8c2ca02700 1 -- 172.21.15.67:0/3693736378 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 8211+0+0 (secure 0 0 0) 0x7f8c180e2720 con 0x7f8c28103610 2024-05-03T21:21:23.727 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:23.725+0000 7f8c267fc700 1 --2- 172.21.15.67:0/3693736378 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f8c100b2910 0x7f8c100b4dd0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f8c1c006fd0 tx=0x7f8c1c008040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T21:21:23.730 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:23.728+0000 7f8c2ca02700 1 -- 172.21.15.67:0/3693736378 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191839 (secure 0 0 0) 0x7f8c1801f020 con 0x7f8c28103610 2024-05-03T21:21:23.993 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:23.991+0000 7f8c27fff700 1 -- 172.21.15.67:0/3693736378 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f8c281a69b0 con 0x7f8c28103610 2024-05-03T21:21:23.993 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:23.991+0000 7f8c2ca02700 1 -- 172.21.15.67:0/3693736378 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+11868 (secure 0 0 0) 0x7f8c18055e20 con 0x7f8c28103610 2024-05-03T21:21:23.994 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-05-03T21:21:23.994 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14541,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6835/4224826546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":4224826546},{"type":"v1","addr":"172.21.15.67:6835","nonce":4224826546}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14553,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6837/2485231621","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6836","nonce":2485231621},{"type":"v1","addr":"172.21.15.67:6837","nonce":2485231621}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14598,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6839/2106078262","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6838","nonce":2106078262},{"type":"v1","addr":"172.21.15.67:6839","nonce":2106078262}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14610,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6841/1559196215","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6840","nonce":1559196215},{"type":"v1","addr":"172.21.15.67:6841","nonce":1559196215}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24376,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.138:6833/3028022602","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6832","nonce":3028022602},{"type":"v1","addr":"172.21.15.138:6833","nonce":3028022602}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24400,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.138:6835/3342864697","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6834","nonce":3342864697},{"type":"v1","addr":"172.21.15.138:6835","nonce":3342864697}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7}],"filesystems":[{"mdsmap":{"epoch":25,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-05-03T21:18:44.665825+0000","modified":"2024-05-03T21:19:03.775546+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24473,"mds_1":24449,"mds_2":24425},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24406":{"gid":24406,"name":"c","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.138:6837/2203273249","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6836","nonce":2203273249},{"type":"v1","addr":"172.21.15.138:6837","nonce":2203273249}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24407":{"gid":24407,"name":"k","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.82:6833/1127287628","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6832","nonce":1127287628},{"type":"v1","addr":"172.21.15.82:6833","nonce":1127287628}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24418":{"gid":24418,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.138:6839/334382443","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6838","nonce":334382443},{"type":"v1","addr":"172.21.15.138:6839","nonce":334382443}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24425":{"gid":24425,"name":"e","rank":2,"incarnation":21,"state":"up:active","state_seq":14,"addr":"172.21.15.82:6835/280518761","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6834","nonce":280518761},{"type":"v1","addr":"172.21.15.82:6835","nonce":280518761}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24449":{"gid":24449,"name":"h","rank":1,"incarnation":19,"state":"up:active","state_seq":11,"addr":"172.21.15.82:6837/1275987852","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6836","nonce":1275987852},{"type":"v1","addr":"172.21.15.82:6837","nonce":1275987852}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24473":{"gid":24473,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":5,"addr":"172.21.15.82:6839/3644499578","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6838","nonce":3644499578},{"type":"v1","addr":"172.21.15.82:6839","nonce":3644499578}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24473,"qdb_cluster":[24449,24473,24425]},"id":1}]} 2024-05-03T21:21:23.996 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:23.993+0000 7f8c27fff700 1 -- 172.21.15.67:0/3693736378 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f8c100b2910 msgr2=0x7f8c100b4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:21:23.996 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:23.993+0000 7f8c27fff700 1 --2- 172.21.15.67:0/3693736378 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f8c100b2910 0x7f8c100b4dd0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f8c1c006fd0 tx=0x7f8c1c008040 comp rx=0 tx=0).stop 2024-05-03T21:21:23.996 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:23.993+0000 7f8c27fff700 1 -- 172.21.15.67:0/3693736378 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8c28103610 msgr2=0x7f8c2819f8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:21:23.996 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:23.993+0000 7f8c27fff700 1 --2- 172.21.15.67:0/3693736378 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8c28103610 0x7f8c2819f8b0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f8c1800d980 tx=0x7f8c1800dd40 comp rx=0 tx=0).stop 2024-05-03T21:21:23.996 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:23.993+0000 7f8c27fff700 1 -- 172.21.15.67:0/3693736378 shutdown_connections 2024-05-03T21:21:23.996 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:23.993+0000 7f8c27fff700 1 --2- 172.21.15.67:0/3693736378 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f8c100b2910 0x7f8c100b4dd0 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:21:23.996 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:23.993+0000 7f8c27fff700 1 --2- 172.21.15.67:0/3693736378 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8c28103610 0x7f8c2819f8b0 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:21:23.997 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:23.993+0000 7f8c27fff700 1 --2- 172.21.15.67:0/3693736378 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f8c281086f0 0x7f8c28199a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:21:23.997 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:23.993+0000 7f8c27fff700 1 --2- 172.21.15.67:0/3693736378 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f8c28103fc0 0x7f8c2819fdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:21:23.997 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:23.993+0000 7f8c27fff700 1 -- 172.21.15.67:0/3693736378 >> 172.21.15.67:0/3693736378 conn(0x7f8c280feec0 msgr2=0x7f8c280ff870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T21:21:23.997 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:23.994+0000 7f8c27fff700 1 -- 172.21.15.67:0/3693736378 shutdown_connections 2024-05-03T21:21:23.997 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:23.994+0000 7f8c27fff700 1 -- 172.21.15.67:0/3693736378 wait complete. 2024-05-03T21:21:23.997 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 25 2024-05-03T21:21:24.574 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:21:24 smithi138 ceph-mon[33110]: from='client.? 172.21.15.67:0/3693736378' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-03T21:21:24.721 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:21:24 smithi067 ceph-mon[29730]: from='client.? 172.21.15.67:0/3693736378' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-03T21:21:24.745 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:21:24 smithi082 ceph-mon[32933]: from='client.? 172.21.15.67:0/3693736378' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-03T21:21:24.800 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9a047fe7599a549042d06a3fb5b6813a203eabea shell --fsid 713bc57a-0991-11ef-bc97-c7b262605968 -- ceph fs dump --format=json 2024-05-03T21:21:25.052 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/713bc57a-0991-11ef-bc97-c7b262605968/mon.a/config 2024-05-03T21:21:25.471 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:21:25 smithi067 ceph-mon[29730]: pgmap v369: 97 pgs: 97 active+clean; 632 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 6 op/s 2024-05-03T21:21:25.495 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:21:25 smithi082 ceph-mon[32933]: pgmap v369: 97 pgs: 97 active+clean; 632 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 6 op/s 2024-05-03T21:21:25.574 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:21:25 smithi138 ceph-mon[33110]: pgmap v369: 97 pgs: 97 active+clean; 632 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 6 op/s 2024-05-03T21:21:26.751 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:26.746+0000 7f55d904f700 1 -- 172.21.15.67:0/3723467228 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f55d4104100 msgr2=0x7f55d4108120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:21:26.752 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:26.746+0000 7f55d904f700 1 --2- 172.21.15.67:0/3723467228 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f55d4104100 0x7f55d4108120 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f55c80099c0 tx=0x7f55c8009cd0 comp rx=0 tx=0).stop 2024-05-03T21:21:26.752 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:26.746+0000 7f55d904f700 1 -- 172.21.15.67:0/3723467228 shutdown_connections 2024-05-03T21:21:26.752 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:26.746+0000 7f55d904f700 1 --2- 172.21.15.67:0/3723467228 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f55d4108850 0x7f55d41103d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:21:26.752 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:26.746+0000 7f55d904f700 1 --2- 172.21.15.67:0/3723467228 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f55d4103750 0x7f55d4103b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:21:26.752 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:26.746+0000 7f55d904f700 1 --2- 172.21.15.67:0/3723467228 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f55d4104100 0x7f55d4108120 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:21:26.752 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:26.746+0000 7f55d904f700 1 -- 172.21.15.67:0/3723467228 >> 172.21.15.67:0/3723467228 conn(0x7f55d407a600 msgr2=0x7f55d407aa10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T21:21:26.752 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:26.747+0000 7f55d904f700 1 -- 172.21.15.67:0/3723467228 shutdown_connections 2024-05-03T21:21:26.752 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:26.747+0000 7f55d904f700 1 -- 172.21.15.67:0/3723467228 wait complete. 2024-05-03T21:21:26.752 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:26.747+0000 7f55d904f700 1 Processor -- start 2024-05-03T21:21:26.752 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:26.748+0000 7f55d904f700 1 -- start start 2024-05-03T21:21:26.752 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:26.748+0000 7f55d904f700 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f55d4103750 0x7f55d40757a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:21:26.752 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:26.748+0000 7f55d904f700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f55d4104100 0x7f55d4075ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:21:26.753 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:26.748+0000 7f55d904f700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f55d4108850 0x7f55d4076220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:21:26.753 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:26.748+0000 7f55d904f700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f55d4113920 con 0x7f55d4104100 2024-05-03T21:21:26.753 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:26.748+0000 7f55d904f700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f55d41137a0 con 0x7f55d4108850 2024-05-03T21:21:26.753 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:26.748+0000 7f55d904f700 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f55d4113aa0 con 0x7f55d4103750 2024-05-03T21:21:26.753 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:26.748+0000 7f55d259c700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f55d4104100 0x7f55d4075ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:21:26.753 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:26.748+0000 7f55d259c700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f55d4104100 0x7f55d4075ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:48630/0 (socket says 172.21.15.67:48630) 2024-05-03T21:21:26.753 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:26.748+0000 7f55d259c700 1 -- 172.21.15.67:0/3019151759 learned_addr learned my addr 172.21.15.67:0/3019151759 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-03T21:21:26.753 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:26.748+0000 7f55d2d9d700 1 --2- 172.21.15.67:0/3019151759 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f55d4103750 0x7f55d40757a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:21:26.753 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:26.748+0000 7f55d359e700 1 --2- 172.21.15.67:0/3019151759 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f55d4108850 0x7f55d4076220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:21:26.753 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:26.748+0000 7f55d259c700 1 -- 172.21.15.67:0/3019151759 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f55d4103750 msgr2=0x7f55d40757a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:21:26.753 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:26.748+0000 7f55d259c700 1 --2- 172.21.15.67:0/3019151759 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f55d4103750 0x7f55d40757a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:21:26.753 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:26.748+0000 7f55d259c700 1 -- 172.21.15.67:0/3019151759 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f55d4108850 msgr2=0x7f55d4076220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:21:26.753 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:26.748+0000 7f55d259c700 1 --2- 172.21.15.67:0/3019151759 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f55d4108850 0x7f55d4076220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:21:26.753 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:26.748+0000 7f55d259c700 1 -- 172.21.15.67:0/3019151759 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f55d41a8250 con 0x7f55d4104100 2024-05-03T21:21:26.754 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:26.748+0000 7f55d359e700 1 --2- 172.21.15.67:0/3019151759 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f55d4108850 0x7f55d4076220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-03T21:21:26.754 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:26.748+0000 7f55d2d9d700 1 --2- 172.21.15.67:0/3019151759 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f55d4103750 0x7f55d40757a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-03T21:21:26.754 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:26.748+0000 7f55d259c700 1 --2- 172.21.15.67:0/3019151759 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f55d4104100 0x7f55d4075ce0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f55c80098e0 tx=0x7f55c80046e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T21:21:26.754 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:26.749+0000 7f55bbfff700 1 -- 172.21.15.67:0/3019151759 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55c8027070 con 0x7f55d4104100 2024-05-03T21:21:26.754 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:26.749+0000 7f55d904f700 1 -- 172.21.15.67:0/3019151759 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f55d41a84e0 con 0x7f55d4104100 2024-05-03T21:21:26.754 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:26.749+0000 7f55bbfff700 1 -- 172.21.15.67:0/3019151759 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(28 keys) ==== 1170+0+0 (secure 0 0 0) 0x7f55c8022470 con 0x7f55d4104100 2024-05-03T21:21:26.754 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:26.749+0000 7f55bbfff700 1 -- 172.21.15.67:0/3019151759 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55c800f6a0 con 0x7f55d4104100 2024-05-03T21:21:26.754 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:26.749+0000 7f55d904f700 1 -- 172.21.15.67:0/3019151759 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f55d41a8990 con 0x7f55d4104100 2024-05-03T21:21:26.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:26.750+0000 7f55d904f700 1 -- 172.21.15.67:0/3019151759 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f55d406a190 con 0x7f55d4104100 2024-05-03T21:21:26.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:26.751+0000 7f55bbfff700 1 -- 172.21.15.67:0/3019151759 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 17) ==== 149703+0+0 (secure 0 0 0) 0x7f55c802a070 con 0x7f55d4104100 2024-05-03T21:21:26.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:26.752+0000 7f55bbfff700 1 --2- 172.21.15.67:0/3019151759 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f55bc0b2990 0x7f55bc0b4e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:21:26.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:26.752+0000 7f55d2d9d700 1 --2- 172.21.15.67:0/3019151759 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f55bc0b2990 0x7f55bc0b4e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:21:26.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:26.752+0000 7f55bbfff700 1 -- 172.21.15.67:0/3019151759 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 8211+0+0 (secure 0 0 0) 0x7f55c80e31c0 con 0x7f55d4104100 2024-05-03T21:21:26.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:26.752+0000 7f55d2d9d700 1 --2- 172.21.15.67:0/3019151759 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f55bc0b2990 0x7f55bc0b4e50 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f55c40099d0 tx=0x7f55c4008040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T21:21:26.759 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:26.756+0000 7f55bbfff700 1 -- 172.21.15.67:0/3019151759 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191839 (secure 0 0 0) 0x7f55c8013070 con 0x7f55d4104100 2024-05-03T21:21:27.028 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:27.026+0000 7f55d904f700 1 -- 172.21.15.67:0/3019151759 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f55d410ba90 con 0x7f55d4104100 2024-05-03T21:21:27.029 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:27.026+0000 7f55bbfff700 1 -- 172.21.15.67:0/3019151759 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+11868 (secure 0 0 0) 0x7f55c80aae20 con 0x7f55d4104100 2024-05-03T21:21:27.029 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-05-03T21:21:27.029 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14541,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6835/4224826546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":4224826546},{"type":"v1","addr":"172.21.15.67:6835","nonce":4224826546}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14553,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6837/2485231621","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6836","nonce":2485231621},{"type":"v1","addr":"172.21.15.67:6837","nonce":2485231621}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14598,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6839/2106078262","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6838","nonce":2106078262},{"type":"v1","addr":"172.21.15.67:6839","nonce":2106078262}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14610,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6841/1559196215","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6840","nonce":1559196215},{"type":"v1","addr":"172.21.15.67:6841","nonce":1559196215}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24376,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.138:6833/3028022602","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6832","nonce":3028022602},{"type":"v1","addr":"172.21.15.138:6833","nonce":3028022602}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24400,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.138:6835/3342864697","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6834","nonce":3342864697},{"type":"v1","addr":"172.21.15.138:6835","nonce":3342864697}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7}],"filesystems":[{"mdsmap":{"epoch":25,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-05-03T21:18:44.665825+0000","modified":"2024-05-03T21:19:03.775546+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24473,"mds_1":24449,"mds_2":24425},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24406":{"gid":24406,"name":"c","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.138:6837/2203273249","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6836","nonce":2203273249},{"type":"v1","addr":"172.21.15.138:6837","nonce":2203273249}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24407":{"gid":24407,"name":"k","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.82:6833/1127287628","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6832","nonce":1127287628},{"type":"v1","addr":"172.21.15.82:6833","nonce":1127287628}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24418":{"gid":24418,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.138:6839/334382443","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6838","nonce":334382443},{"type":"v1","addr":"172.21.15.138:6839","nonce":334382443}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24425":{"gid":24425,"name":"e","rank":2,"incarnation":21,"state":"up:active","state_seq":14,"addr":"172.21.15.82:6835/280518761","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6834","nonce":280518761},{"type":"v1","addr":"172.21.15.82:6835","nonce":280518761}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24449":{"gid":24449,"name":"h","rank":1,"incarnation":19,"state":"up:active","state_seq":11,"addr":"172.21.15.82:6837/1275987852","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6836","nonce":1275987852},{"type":"v1","addr":"172.21.15.82:6837","nonce":1275987852}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24473":{"gid":24473,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":5,"addr":"172.21.15.82:6839/3644499578","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6838","nonce":3644499578},{"type":"v1","addr":"172.21.15.82:6839","nonce":3644499578}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24473,"qdb_cluster":[24449,24473,24425]},"id":1}]} 2024-05-03T21:21:27.031 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:27.028+0000 7f55d904f700 1 -- 172.21.15.67:0/3019151759 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f55bc0b2990 msgr2=0x7f55bc0b4e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:21:27.031 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:27.028+0000 7f55d904f700 1 --2- 172.21.15.67:0/3019151759 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f55bc0b2990 0x7f55bc0b4e50 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f55c40099d0 tx=0x7f55c4008040 comp rx=0 tx=0).stop 2024-05-03T21:21:27.031 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:27.028+0000 7f55d904f700 1 -- 172.21.15.67:0/3019151759 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f55d4104100 msgr2=0x7f55d4075ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:21:27.031 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:27.028+0000 7f55d904f700 1 --2- 172.21.15.67:0/3019151759 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f55d4104100 0x7f55d4075ce0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f55c80098e0 tx=0x7f55c80046e0 comp rx=0 tx=0).stop 2024-05-03T21:21:27.031 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:27.029+0000 7f55d904f700 1 -- 172.21.15.67:0/3019151759 shutdown_connections 2024-05-03T21:21:27.031 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:27.029+0000 7f55d904f700 1 --2- 172.21.15.67:0/3019151759 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f55bc0b2990 0x7f55bc0b4e50 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:21:27.031 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:27.029+0000 7f55d904f700 1 --2- 172.21.15.67:0/3019151759 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f55d4108850 0x7f55d4076220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:21:27.031 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:27.029+0000 7f55d904f700 1 --2- 172.21.15.67:0/3019151759 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f55d4103750 0x7f55d40757a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:21:27.031 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:27.029+0000 7f55d904f700 1 --2- 172.21.15.67:0/3019151759 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f55d4104100 0x7f55d4075ce0 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:21:27.031 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:27.029+0000 7f55d904f700 1 -- 172.21.15.67:0/3019151759 >> 172.21.15.67:0/3019151759 conn(0x7f55d407a600 msgr2=0x7f55d4101540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T21:21:27.032 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:27.029+0000 7f55d904f700 1 -- 172.21.15.67:0/3019151759 shutdown_connections 2024-05-03T21:21:27.032 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:27.029+0000 7f55d904f700 1 -- 172.21.15.67:0/3019151759 wait complete. 2024-05-03T21:21:27.032 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 25 2024-05-03T21:21:27.416 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:21:27 smithi067 ceph-mon[29730]: pgmap v370: 97 pgs: 97 active+clean; 632 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-05-03T21:21:27.416 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:21:27 smithi067 ceph-mon[29730]: from='client.? 172.21.15.67:0/3019151759' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-03T21:21:27.575 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:21:27 smithi138 ceph-mon[33110]: pgmap v370: 97 pgs: 97 active+clean; 632 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-05-03T21:21:27.575 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:21:27 smithi138 ceph-mon[33110]: from='client.? 172.21.15.67:0/3019151759' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-03T21:21:27.735 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9a047fe7599a549042d06a3fb5b6813a203eabea shell --fsid 713bc57a-0991-11ef-bc97-c7b262605968 -- ceph osd dump --format=json 2024-05-03T21:21:27.745 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:21:27 smithi082 ceph-mon[32933]: pgmap v370: 97 pgs: 97 active+clean; 632 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-05-03T21:21:27.745 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:21:27 smithi082 ceph-mon[32933]: from='client.? 172.21.15.67:0/3019151759' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-03T21:21:27.992 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/713bc57a-0991-11ef-bc97-c7b262605968/mon.a/config 2024-05-03T21:21:29.388 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:21:29 smithi067 ceph-mon[29730]: pgmap v371: 97 pgs: 97 active+clean; 632 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 6 op/s 2024-05-03T21:21:29.574 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:21:29 smithi138 ceph-mon[33110]: pgmap v371: 97 pgs: 97 active+clean; 632 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 6 op/s 2024-05-03T21:21:29.628 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:29.623+0000 7fd66ff3e700 1 -- 172.21.15.67:0/2562173182 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd668103780 msgr2=0x7fd668103b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:21:29.628 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:29.623+0000 7fd66ff3e700 1 --2- 172.21.15.67:0/2562173182 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd668103780 0x7fd668103b60 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7fd6580099c0 tx=0x7fd658009cd0 comp rx=0 tx=0).stop 2024-05-03T21:21:29.628 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:29.623+0000 7fd66ff3e700 1 -- 172.21.15.67:0/2562173182 shutdown_connections 2024-05-03T21:21:29.628 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:29.623+0000 7fd66ff3e700 1 --2- 172.21.15.67:0/2562173182 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd668103780 0x7fd668103b60 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:21:29.628 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:29.623+0000 7fd66ff3e700 1 --2- 172.21.15.67:0/2562173182 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fd668108870 0x7fd6681103f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:21:29.628 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:29.623+0000 7fd66ff3e700 1 --2- 172.21.15.67:0/2562173182 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fd668104130 0x7fd668108140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:21:29.629 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:29.623+0000 7fd66ff3e700 1 -- 172.21.15.67:0/2562173182 >> 172.21.15.67:0/2562173182 conn(0x7fd66807a610 msgr2=0x7fd66807aa20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T21:21:29.629 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:29.624+0000 7fd66ff3e700 1 -- 172.21.15.67:0/2562173182 shutdown_connections 2024-05-03T21:21:29.629 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:29.624+0000 7fd66ff3e700 1 -- 172.21.15.67:0/2562173182 wait complete. 2024-05-03T21:21:29.629 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:29.624+0000 7fd66ff3e700 1 Processor -- start 2024-05-03T21:21:29.629 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:29.624+0000 7fd66ff3e700 1 -- start start 2024-05-03T21:21:29.629 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:29.624+0000 7fd66ff3e700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd668103780 0x7fd6680757a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:21:29.629 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:29.624+0000 7fd66ff3e700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fd668104130 0x7fd668075ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:21:29.629 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:29.624+0000 7fd66ff3e700 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fd668108870 0x7fd668076220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:21:29.629 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:29.625+0000 7fd66ff3e700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fd6681139c0 con 0x7fd668103780 2024-05-03T21:21:29.629 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:29.625+0000 7fd66ff3e700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7fd668113840 con 0x7fd668104130 2024-05-03T21:21:29.629 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:29.625+0000 7fd66ff3e700 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7fd668113b40 con 0x7fd668108870 2024-05-03T21:21:29.629 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:29.625+0000 7fd66dcda700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd668103780 0x7fd6680757a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:21:29.629 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:29.625+0000 7fd66dcda700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd668103780 0x7fd6680757a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:48648/0 (socket says 172.21.15.67:48648) 2024-05-03T21:21:29.630 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:29.625+0000 7fd66dcda700 1 -- 172.21.15.67:0/795650593 learned_addr learned my addr 172.21.15.67:0/795650593 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-03T21:21:29.630 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:29.625+0000 7fd66d4d9700 1 --2- 172.21.15.67:0/795650593 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fd668104130 0x7fd668075ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:21:29.630 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:29.625+0000 7fd66e4db700 1 --2- 172.21.15.67:0/795650593 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fd668108870 0x7fd668076220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:21:29.630 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:29.625+0000 7fd66dcda700 1 -- 172.21.15.67:0/795650593 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fd668108870 msgr2=0x7fd668076220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:21:29.630 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:29.625+0000 7fd66dcda700 1 --2- 172.21.15.67:0/795650593 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fd668108870 0x7fd668076220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:21:29.630 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:29.625+0000 7fd66dcda700 1 -- 172.21.15.67:0/795650593 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fd668104130 msgr2=0x7fd668075ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:21:29.630 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:29.625+0000 7fd66dcda700 1 --2- 172.21.15.67:0/795650593 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fd668104130 0x7fd668075ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:21:29.630 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:29.625+0000 7fd66dcda700 1 -- 172.21.15.67:0/795650593 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd6681a8300 con 0x7fd668103780 2024-05-03T21:21:29.630 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:29.625+0000 7fd66d4d9700 1 --2- 172.21.15.67:0/795650593 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fd668104130 0x7fd668075ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-03T21:21:29.630 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:29.625+0000 7fd66dcda700 1 --2- 172.21.15.67:0/795650593 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd668103780 0x7fd6680757a0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7fd658009fd0 tx=0x7fd65800bde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T21:21:29.630 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:29.625+0000 7fd65effd700 1 -- 172.21.15.67:0/795650593 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd658027070 con 0x7fd668103780 2024-05-03T21:21:29.630 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:29.625+0000 7fd66ff3e700 1 -- 172.21.15.67:0/795650593 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd6681a8590 con 0x7fd668103780 2024-05-03T21:21:29.630 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:29.625+0000 7fd65effd700 1 -- 172.21.15.67:0/795650593 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(28 keys) ==== 1170+0+0 (secure 0 0 0) 0x7fd658004180 con 0x7fd668103780 2024-05-03T21:21:29.630 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:29.625+0000 7fd65effd700 1 -- 172.21.15.67:0/795650593 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd65800f460 con 0x7fd668103780 2024-05-03T21:21:29.631 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:29.625+0000 7fd66ff3e700 1 -- 172.21.15.67:0/795650593 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd6681a8a40 con 0x7fd668103780 2024-05-03T21:21:29.631 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:29.626+0000 7fd66ff3e700 1 -- 172.21.15.67:0/795650593 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd66806a190 con 0x7fd668103780 2024-05-03T21:21:29.631 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:29.628+0000 7fd65effd700 1 -- 172.21.15.67:0/795650593 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 17) ==== 149703+0+0 (secure 0 0 0) 0x7fd65802a070 con 0x7fd668103780 2024-05-03T21:21:29.631 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:29.628+0000 7fd65effd700 1 --2- 172.21.15.67:0/795650593 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7fd6540b2910 0x7fd6540b4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:21:29.632 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:29.628+0000 7fd66d4d9700 1 --2- 172.21.15.67:0/795650593 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7fd6540b2910 0x7fd6540b4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:21:29.632 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:29.628+0000 7fd65effd700 1 -- 172.21.15.67:0/795650593 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 8211+0+0 (secure 0 0 0) 0x7fd6580e36d0 con 0x7fd668103780 2024-05-03T21:21:29.632 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:29.629+0000 7fd66d4d9700 1 --2- 172.21.15.67:0/795650593 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7fd6540b2910 0x7fd6540b4dd0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fd6681080f0 tx=0x7fd664008040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T21:21:29.634 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:29.632+0000 7fd65effd700 1 -- 172.21.15.67:0/795650593 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191839 (secure 0 0 0) 0x7fd658013070 con 0x7fd668103780 2024-05-03T21:21:29.745 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:21:29 smithi082 ceph-mon[32933]: pgmap v371: 97 pgs: 97 active+clean; 632 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 6 op/s 2024-05-03T21:21:29.849 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:29.847+0000 7fd66ff3e700 1 -- 172.21.15.67:0/795650593 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fd6681a8d40 con 0x7fd668103780 2024-05-03T21:21:29.851 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:29.849+0000 7fd65effd700 1 -- 172.21.15.67:0/795650593 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v84) ==== 74+0+22637 (secure 0 0 0) 0x7fd6580ab330 con 0x7fd668103780 2024-05-03T21:21:29.851 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-05-03T21:21:29.851 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":84,"fsid":"713bc57a-0991-11ef-bc97-c7b262605968","created":"2024-05-03T21:10:34.070178+0000","modified":"2024-05-03T21:19:34.967839+0000","last_up_change":"2024-05-03T21:17:39.993919+0000","last_in_change":"2024-05-03T21:17:19.978727+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-03T21:13:59.810446+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-05-03T21:18:37.953919+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"82","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-05-03T21:18:41.571707+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"82","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7939f31c-6fb7-42bd-af96-88a448d11bd2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":3692483807},{"type":"v1","addr":"172.21.15.67:6803","nonce":3692483807}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":3692483807},{"type":"v1","addr":"172.21.15.67:6805","nonce":3692483807}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":3692483807},{"type":"v1","addr":"172.21.15.67:6809","nonce":3692483807}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":3692483807},{"type":"v1","addr":"172.21.15.67:6807","nonce":3692483807}]},"public_addr":"172.21.15.67:6803/3692483807","cluster_addr":"172.21.15.67:6805/3692483807","heartbeat_back_addr":"172.21.15.67:6809/3692483807","heartbeat_front_addr":"172.21.15.67:6807/3692483807","state":["exists","up"]},{"osd":1,"uuid":"71858193-05a6-4c2c-9bb7-bc3d811d634e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6800","nonce":1139376397},{"type":"v1","addr":"172.21.15.82:6801","nonce":1139376397}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6802","nonce":1139376397},{"type":"v1","addr":"172.21.15.82:6803","nonce":1139376397}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6806","nonce":1139376397},{"type":"v1","addr":"172.21.15.82:6807","nonce":1139376397}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6804","nonce":1139376397},{"type":"v1","addr":"172.21.15.82:6805","nonce":1139376397}]},"public_addr":"172.21.15.82:6801/1139376397","cluster_addr":"172.21.15.82:6803/1139376397","heartbeat_back_addr":"172.21.15.82:6807/1139376397","heartbeat_front_addr":"172.21.15.82:6805/1139376397","state":["exists","up"]},{"osd":2,"uuid":"f9ece78d-6a19-4d46-9022-6ed184074745","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6800","nonce":2774330123},{"type":"v1","addr":"172.21.15.138:6801","nonce":2774330123}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6802","nonce":2774330123},{"type":"v1","addr":"172.21.15.138:6803","nonce":2774330123}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6806","nonce":2774330123},{"type":"v1","addr":"172.21.15.138:6807","nonce":2774330123}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6804","nonce":2774330123},{"type":"v1","addr":"172.21.15.138:6805","nonce":2774330123}]},"public_addr":"172.21.15.138:6801/2774330123","cluster_addr":"172.21.15.138:6803/2774330123","heartbeat_back_addr":"172.21.15.138:6807/2774330123","heartbeat_front_addr":"172.21.15.138:6805/2774330123","state":["exists","up"]},{"osd":3,"uuid":"3616df71-e7b3-4687-aaba-37b0b7766a3c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":27,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":3915934652},{"type":"v1","addr":"172.21.15.67:6811","nonce":3915934652}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":3915934652},{"type":"v1","addr":"172.21.15.67:6813","nonce":3915934652}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":3915934652},{"type":"v1","addr":"172.21.15.67:6817","nonce":3915934652}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":3915934652},{"type":"v1","addr":"172.21.15.67:6815","nonce":3915934652}]},"public_addr":"172.21.15.67:6811/3915934652","cluster_addr":"172.21.15.67:6813/3915934652","heartbeat_back_addr":"172.21.15.67:6817/3915934652","heartbeat_front_addr":"172.21.15.67:6815/3915934652","state":["exists","up"]},{"osd":4,"uuid":"c3fd0ffd-e23f-4279-b679-9d95a144cf32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6808","nonce":4031818951},{"type":"v1","addr":"172.21.15.82:6809","nonce":4031818951}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6810","nonce":4031818951},{"type":"v1","addr":"172.21.15.82:6811","nonce":4031818951}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6814","nonce":4031818951},{"type":"v1","addr":"172.21.15.82:6815","nonce":4031818951}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6812","nonce":4031818951},{"type":"v1","addr":"172.21.15.82:6813","nonce":4031818951}]},"public_addr":"172.21.15.82:6809/4031818951","cluster_addr":"172.21.15.82:6811/4031818951","heartbeat_back_addr":"172.21.15.82:6815/4031818951","heartbeat_front_addr":"172.21.15.82:6813/4031818951","state":["exists","up"]},{"osd":5,"uuid":"540ef0d9-a6dc-43c6-b09e-e87340ba24cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6808","nonce":2521889499},{"type":"v1","addr":"172.21.15.138:6809","nonce":2521889499}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6810","nonce":2521889499},{"type":"v1","addr":"172.21.15.138:6811","nonce":2521889499}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6814","nonce":2521889499},{"type":"v1","addr":"172.21.15.138:6815","nonce":2521889499}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6812","nonce":2521889499},{"type":"v1","addr":"172.21.15.138:6813","nonce":2521889499}]},"public_addr":"172.21.15.138:6809/2521889499","cluster_addr":"172.21.15.138:6811/2521889499","heartbeat_back_addr":"172.21.15.138:6815/2521889499","heartbeat_front_addr":"172.21.15.138:6813/2521889499","state":["exists","up"]},{"osd":6,"uuid":"49a89fd0-958a-4be9-a8db-6022f946153e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6818","nonce":3269682731},{"type":"v1","addr":"172.21.15.67:6819","nonce":3269682731}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":3269682731},{"type":"v1","addr":"172.21.15.67:6821","nonce":3269682731}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":3269682731},{"type":"v1","addr":"172.21.15.67:6825","nonce":3269682731}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":3269682731},{"type":"v1","addr":"172.21.15.67:6823","nonce":3269682731}]},"public_addr":"172.21.15.67:6819/3269682731","cluster_addr":"172.21.15.67:6821/3269682731","heartbeat_back_addr":"172.21.15.67:6825/3269682731","heartbeat_front_addr":"172.21.15.67:6823/3269682731","state":["exists","up"]},{"osd":7,"uuid":"8fa4cd64-ca6b-48ae-8db0-d369ce6102ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":50,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6816","nonce":3008589976},{"type":"v1","addr":"172.21.15.82:6817","nonce":3008589976}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6818","nonce":3008589976},{"type":"v1","addr":"172.21.15.82:6819","nonce":3008589976}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6822","nonce":3008589976},{"type":"v1","addr":"172.21.15.82:6823","nonce":3008589976}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6820","nonce":3008589976},{"type":"v1","addr":"172.21.15.82:6821","nonce":3008589976}]},"public_addr":"172.21.15.82:6817/3008589976","cluster_addr":"172.21.15.82:6819/3008589976","heartbeat_back_addr":"172.21.15.82:6823/3008589976","heartbeat_front_addr":"172.21.15.82:6821/3008589976","state":["exists","up"]},{"osd":8,"uuid":"fae39087-6c38-4771-a2b9-3f4fecb08edd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":57,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6816","nonce":2223920070},{"type":"v1","addr":"172.21.15.138:6817","nonce":2223920070}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6818","nonce":2223920070},{"type":"v1","addr":"172.21.15.138:6819","nonce":2223920070}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6822","nonce":2223920070},{"type":"v1","addr":"172.21.15.138:6823","nonce":2223920070}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6820","nonce":2223920070},{"type":"v1","addr":"172.21.15.138:6821","nonce":2223920070}]},"public_addr":"172.21.15.138:6817/2223920070","cluster_addr":"172.21.15.138:6819/2223920070","heartbeat_back_addr":"172.21.15.138:6823/2223920070","heartbeat_front_addr":"172.21.15.138:6821/2223920070","state":["exists","up"]},{"osd":9,"uuid":"3d387dbf-0163-41b7-98af-4fc45ed73ad4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":63,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":3939417129},{"type":"v1","addr":"172.21.15.67:6827","nonce":3939417129}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":3939417129},{"type":"v1","addr":"172.21.15.67:6829","nonce":3939417129}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":3939417129},{"type":"v1","addr":"172.21.15.67:6833","nonce":3939417129}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":3939417129},{"type":"v1","addr":"172.21.15.67:6831","nonce":3939417129}]},"public_addr":"172.21.15.67:6827/3939417129","cluster_addr":"172.21.15.67:6829/3939417129","heartbeat_back_addr":"172.21.15.67:6833/3939417129","heartbeat_front_addr":"172.21.15.67:6831/3939417129","state":["exists","up"]},{"osd":10,"uuid":"db0305a0-a83b-474b-837a-fbc239ac1738","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6824","nonce":2505507029},{"type":"v1","addr":"172.21.15.82:6825","nonce":2505507029}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6826","nonce":2505507029},{"type":"v1","addr":"172.21.15.82:6827","nonce":2505507029}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6830","nonce":2505507029},{"type":"v1","addr":"172.21.15.82:6831","nonce":2505507029}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6828","nonce":2505507029},{"type":"v1","addr":"172.21.15.82:6829","nonce":2505507029}]},"public_addr":"172.21.15.82:6825/2505507029","cluster_addr":"172.21.15.82:6827/2505507029","heartbeat_back_addr":"172.21.15.82:6831/2505507029","heartbeat_front_addr":"172.21.15.82:6829/2505507029","state":["exists","up"]},{"osd":11,"uuid":"479c2747-f28e-4704-87e4-d0c31782e2de","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":74,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6824","nonce":2528685018},{"type":"v1","addr":"172.21.15.138:6825","nonce":2528685018}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6826","nonce":2528685018},{"type":"v1","addr":"172.21.15.138:6827","nonce":2528685018}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6830","nonce":2528685018},{"type":"v1","addr":"172.21.15.138:6831","nonce":2528685018}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6828","nonce":2528685018},{"type":"v1","addr":"172.21.15.138:6829","nonce":2528685018}]},"public_addr":"172.21.15.138:6825/2528685018","cluster_addr":"172.21.15.138:6827/2528685018","heartbeat_back_addr":"172.21.15.138:6831/2528685018","heartbeat_front_addr":"172.21.15.138:6829/2528685018","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-03T21:13:07.768773+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-03T21:13:31.870232+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-03T21:13:55.394206+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-03T21:14:20.540281+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-03T21:14:44.318523+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-03T21:15:08.490829+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-03T21:15:34.144734+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-03T21:15:58.717240+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-03T21:16:22.743013+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-03T21:16:49.099588+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-03T21:17:13.321126+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-03T21:17:38.001978+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.0","mappings":[{"from":8,"to":4}]},{"pgid":"2.a","mappings":[{"from":2,"to":11}]},{"pgid":"2.b","mappings":[{"from":2,"to":4}]},{"pgid":"2.10","mappings":[{"from":8,"to":11}]},{"pgid":"3.1","mappings":[{"from":4,"to":9}]},{"pgid":"3.1b","mappings":[{"from":5,"to":9}]},{"pgid":"3.21","mappings":[{"from":4,"to":8}]},{"pgid":"3.3c","mappings":[{"from":4,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.67:0/1239643865":"2024-05-04T21:11:33.721416+0000","172.21.15.67:6801/2576113038":"2024-05-04T21:11:33.721416+0000","172.21.15.67:6800/2576113038":"2024-05-04T21:11:33.721416+0000","172.21.15.67:6800/1698171509":"2024-05-04T21:11:07.333651+0000","172.21.15.67:0/2217893923":"2024-05-04T21:11:33.721416+0000","172.21.15.67:0/20864811":"2024-05-04T21:11:07.333651+0000","172.21.15.67:6801/1698171509":"2024-05-04T21:11:07.333651+0000","172.21.15.67:0/773376553":"2024-05-04T21:11:33.721416+0000","172.21.15.67:0/847549114":"2024-05-04T21:11:07.333651+0000","172.21.15.67:0/1291778035":"2024-05-04T21:11:07.333651+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-03T21:21:29.853 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:29.851+0000 7fd66ff3e700 1 -- 172.21.15.67:0/795650593 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7fd6540b2910 msgr2=0x7fd6540b4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:21:29.853 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:29.851+0000 7fd66ff3e700 1 --2- 172.21.15.67:0/795650593 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7fd6540b2910 0x7fd6540b4dd0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fd6681080f0 tx=0x7fd664008040 comp rx=0 tx=0).stop 2024-05-03T21:21:29.853 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:29.851+0000 7fd66ff3e700 1 -- 172.21.15.67:0/795650593 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd668103780 msgr2=0x7fd6680757a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:21:29.853 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:29.851+0000 7fd66ff3e700 1 --2- 172.21.15.67:0/795650593 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd668103780 0x7fd6680757a0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7fd658009fd0 tx=0x7fd65800bde0 comp rx=0 tx=0).stop 2024-05-03T21:21:29.854 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:29.851+0000 7fd66ff3e700 1 -- 172.21.15.67:0/795650593 shutdown_connections 2024-05-03T21:21:29.854 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:29.851+0000 7fd66ff3e700 1 --2- 172.21.15.67:0/795650593 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7fd6540b2910 0x7fd6540b4dd0 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:21:29.854 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:29.851+0000 7fd66ff3e700 1 --2- 172.21.15.67:0/795650593 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd668103780 0x7fd6680757a0 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:21:29.854 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:29.851+0000 7fd66ff3e700 1 --2- 172.21.15.67:0/795650593 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fd668108870 0x7fd668076220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:21:29.854 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:29.851+0000 7fd66ff3e700 1 --2- 172.21.15.67:0/795650593 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fd668104130 0x7fd668075ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:21:29.854 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:29.851+0000 7fd66ff3e700 1 -- 172.21.15.67:0/795650593 >> 172.21.15.67:0/795650593 conn(0x7fd66807a610 msgr2=0x7fd6681014f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T21:21:29.854 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:29.851+0000 7fd66ff3e700 1 -- 172.21.15.67:0/795650593 shutdown_connections 2024-05-03T21:21:29.854 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:29.851+0000 7fd66ff3e700 1 -- 172.21.15.67:0/795650593 wait complete. 2024-05-03T21:21:30.471 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:21:30 smithi067 ceph-mon[29730]: from='client.? 172.21.15.67:0/795650593' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-03T21:21:30.557 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 25, 'flags': 50, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-05-03T21:18:44.665825+0000', 'modified': '2024-05-03T21:19:03.775546+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 3, 'in': [0, 1, 2], 'up': {'mds_0': 24473, 'mds_1': 24449, 'mds_2': 24425}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24406': {'gid': 24406, 'name': 'c', 'rank': 2, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.138:6837/2203273249', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.138:6836', 'nonce': 2203273249}, {'type': 'v1', 'addr': '172.21.15.138:6837', 'nonce': 2203273249}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24407': {'gid': 24407, 'name': 'k', 'rank': 1, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.82:6833/1127287628', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.82:6832', 'nonce': 1127287628}, {'type': 'v1', 'addr': '172.21.15.82:6833', 'nonce': 1127287628}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24418': {'gid': 24418, 'name': 'i', 'rank': 0, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.138:6839/334382443', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.138:6838', 'nonce': 334382443}, {'type': 'v1', 'addr': '172.21.15.138:6839', 'nonce': 334382443}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24425': {'gid': 24425, 'name': 'e', 'rank': 2, 'incarnation': 21, 'state': 'up:active', 'state_seq': 14, 'addr': '172.21.15.82:6835/280518761', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.82:6834', 'nonce': 280518761}, {'type': 'v1', 'addr': '172.21.15.82:6835', 'nonce': 280518761}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24449': {'gid': 24449, 'name': 'h', 'rank': 1, 'incarnation': 19, 'state': 'up:active', 'state_seq': 11, 'addr': '172.21.15.82:6837/1275987852', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.82:6836', 'nonce': 1275987852}, {'type': 'v1', 'addr': '172.21.15.82:6837', 'nonce': 1275987852}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24473': {'gid': 24473, 'name': 'b', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 5, 'addr': '172.21.15.82:6839/3644499578', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.82:6838', 'nonce': 3644499578}, {'type': 'v1', 'addr': '172.21.15.82:6839', 'nonce': 3644499578}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 24473, 'qdb_cluster': [24449, 24473, 24425]} 2024-05-03T21:21:30.557 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 3/3 2024-05-03T21:21:30.557 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9a047fe7599a549042d06a3fb5b6813a203eabea shell --fsid 713bc57a-0991-11ef-bc97-c7b262605968 -- ceph tell mds.e status 2024-05-03T21:21:30.574 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:21:30 smithi138 ceph-mon[33110]: from='client.? 172.21.15.67:0/795650593' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-03T21:21:30.745 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:21:30 smithi082 ceph-mon[32933]: from='client.? 172.21.15.67:0/795650593' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-03T21:21:30.808 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/713bc57a-0991-11ef-bc97-c7b262605968/mon.a/config 2024-05-03T21:21:31.471 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:21:31 smithi067 ceph-mon[29730]: pgmap v372: 97 pgs: 97 active+clean; 632 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T21:21:31.574 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:21:31 smithi138 ceph-mon[33110]: pgmap v372: 97 pgs: 97 active+clean; 632 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T21:21:31.745 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:21:31 smithi082 ceph-mon[32933]: pgmap v372: 97 pgs: 97 active+clean; 632 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T21:21:32.429 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:32.424+0000 7f106c2c9700 1 -- 172.21.15.67:0/1029639654 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f106410fed0 msgr2=0x7f10641122c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:21:32.429 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:32.424+0000 7f106c2c9700 1 --2- 172.21.15.67:0/1029639654 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f106410fed0 0x7f10641122c0 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f105c00b0a0 tx=0x7f105c00b3b0 comp rx=0 tx=0).stop 2024-05-03T21:21:32.429 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:32.424+0000 7f106c2c9700 1 -- 172.21.15.67:0/1029639654 shutdown_connections 2024-05-03T21:21:32.429 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:32.424+0000 7f106c2c9700 1 --2- 172.21.15.67:0/1029639654 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f106410fed0 0x7f10641122c0 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:21:32.429 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:32.424+0000 7f106c2c9700 1 --2- 172.21.15.67:0/1029639654 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f10640756d0 0x7f1064075ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:21:32.429 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:32.424+0000 7f106c2c9700 1 --2- 172.21.15.67:0/1029639654 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f1064075ff0 0x7f106410f990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:21:32.429 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:32.424+0000 7f106c2c9700 1 -- 172.21.15.67:0/1029639654 >> 172.21.15.67:0/1029639654 conn(0x7f106407a610 msgr2=0x7f106407aa20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T21:21:32.429 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:32.425+0000 7f106c2c9700 1 -- 172.21.15.67:0/1029639654 shutdown_connections 2024-05-03T21:21:32.429 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:32.425+0000 7f106c2c9700 1 -- 172.21.15.67:0/1029639654 wait complete. 2024-05-03T21:21:32.429 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:32.425+0000 7f106c2c9700 1 Processor -- start 2024-05-03T21:21:32.429 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:32.425+0000 7f106c2c9700 1 -- start start 2024-05-03T21:21:32.430 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:32.425+0000 7f106c2c9700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f10640756d0 0x7f106419c060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:21:32.430 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:32.425+0000 7f106c2c9700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f1064075ff0 0x7f106419c5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:21:32.430 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:32.425+0000 7f106c2c9700 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f106410fed0 0x7f10641a0930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:21:32.430 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:32.425+0000 7f106c2c9700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f1064115c30 con 0x7f10640756d0 2024-05-03T21:21:32.430 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:32.425+0000 7f106c2c9700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f1064115ab0 con 0x7f1064075ff0 2024-05-03T21:21:32.430 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:32.425+0000 7f106c2c9700 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f1064115db0 con 0x7f106410fed0 2024-05-03T21:21:32.430 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:32.425+0000 7f106a065700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f10640756d0 0x7f106419c060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:21:32.430 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:32.426+0000 7f106a065700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f10640756d0 0x7f106419c060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:48662/0 (socket says 172.21.15.67:48662) 2024-05-03T21:21:32.430 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:32.426+0000 7f106a065700 1 -- 172.21.15.67:0/4282601850 learned_addr learned my addr 172.21.15.67:0/4282601850 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-03T21:21:32.430 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:32.426+0000 7f1069864700 1 --2- 172.21.15.67:0/4282601850 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f1064075ff0 0x7f106419c5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:21:32.430 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:32.426+0000 7f106a866700 1 --2- 172.21.15.67:0/4282601850 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f106410fed0 0x7f10641a0930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:21:32.430 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:32.426+0000 7f106a065700 1 -- 172.21.15.67:0/4282601850 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f106410fed0 msgr2=0x7f10641a0930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:21:32.430 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:32.426+0000 7f106a065700 1 --2- 172.21.15.67:0/4282601850 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f106410fed0 0x7f10641a0930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:21:32.430 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:32.426+0000 7f106a065700 1 -- 172.21.15.67:0/4282601850 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f1064075ff0 msgr2=0x7f106419c5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:21:32.431 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:32.426+0000 7f106a065700 1 --2- 172.21.15.67:0/4282601850 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f1064075ff0 0x7f106419c5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:21:32.431 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:32.426+0000 7f106a065700 1 -- 172.21.15.67:0/4282601850 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f10641a1070 con 0x7f10640756d0 2024-05-03T21:21:32.431 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:32.426+0000 7f106a866700 1 --2- 172.21.15.67:0/4282601850 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f106410fed0 0x7f10641a0930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-03T21:21:32.431 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:32.426+0000 7f1069864700 1 --2- 172.21.15.67:0/4282601850 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f1064075ff0 0x7f106419c5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-03T21:21:32.431 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:32.426+0000 7f106a065700 1 --2- 172.21.15.67:0/4282601850 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f10640756d0 0x7f106419c060 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f106000b530 tx=0x7f106000b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T21:21:32.431 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:32.426+0000 7f105b7fe700 1 -- 172.21.15.67:0/4282601850 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1060015020 con 0x7f10640756d0 2024-05-03T21:21:32.431 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:32.426+0000 7f106c2c9700 1 -- 172.21.15.67:0/4282601850 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f10641a1360 con 0x7f10640756d0 2024-05-03T21:21:32.431 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:32.426+0000 7f105b7fe700 1 -- 172.21.15.67:0/4282601850 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(28 keys) ==== 1170+0+0 (secure 0 0 0) 0x7f1060011d30 con 0x7f10640756d0 2024-05-03T21:21:32.431 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:32.426+0000 7f105b7fe700 1 -- 172.21.15.67:0/4282601850 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f106000f2e0 con 0x7f10640756d0 2024-05-03T21:21:32.431 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:32.426+0000 7f106c2c9700 1 -- 172.21.15.67:0/4282601850 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f10641a8be0 con 0x7f10640756d0 2024-05-03T21:21:32.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:32.429+0000 7f105b7fe700 1 -- 172.21.15.67:0/4282601850 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 17) ==== 149703+0+0 (secure 0 0 0) 0x7f1060018020 con 0x7f10640756d0 2024-05-03T21:21:32.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:32.429+0000 7f105b7fe700 1 --2- 172.21.15.67:0/4282601850 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f10500b2910 0x7f10500b4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:21:32.434 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:32.429+0000 7f1069864700 1 --2- 172.21.15.67:0/4282601850 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f10500b2910 0x7f10500b4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:21:32.434 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:32.429+0000 7f105b7fe700 1 -- 172.21.15.67:0/4282601850 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 8211+0+0 (secure 0 0 0) 0x7f10600e1b40 con 0x7f10640756d0 2024-05-03T21:21:32.434 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:32.430+0000 7f1069864700 1 --2- 172.21.15.67:0/4282601850 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f10500b2910 0x7f10500b4dd0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f1064111a90 tx=0x7f1054008040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T21:21:32.434 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:32.430+0000 7f1076b5cb80 1 Processor -- start 2024-05-03T21:21:32.434 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:32.430+0000 7f1076b5cb80 1 -- start start 2024-05-03T21:21:32.434 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:32.430+0000 7f1076b5cb80 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5555da66e740 0x5555da66eb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:21:32.434 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:32.430+0000 7f1076b5cb80 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5555da66f060 0x5555da669f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:21:32.434 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:32.430+0000 7f1076b5cb80 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5555da66a480 0x5555da66a8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:21:32.434 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:32.430+0000 7f1076b5cb80 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x5555da5bc8f0 con 0x5555da66f060 2024-05-03T21:21:32.434 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:32.430+0000 7f1076b5cb80 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x5555da521d50 con 0x5555da66e740 2024-05-03T21:21:32.434 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:32.430+0000 7f1076b5cb80 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x5555da5b88b0 con 0x5555da66a480 2024-05-03T21:21:32.434 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:32.430+0000 7f1069864700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5555da66f060 0x5555da669f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:21:32.434 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:32.430+0000 7f1069864700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5555da66f060 0x5555da669f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:48678/0 (socket says 172.21.15.67:48678) 2024-05-03T21:21:32.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:32.430+0000 7f1069864700 1 -- 172.21.15.67:0/1175101131 learned_addr learned my addr 172.21.15.67:0/1175101131 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-03T21:21:32.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:32.431+0000 7f106a065700 1 --2- 172.21.15.67:0/1175101131 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5555da66e740 0x5555da66eb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:21:32.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:32.431+0000 7f106a866700 1 --2- 172.21.15.67:0/1175101131 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5555da66a480 0x5555da66a8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:21:32.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:32.431+0000 7f1069864700 1 -- 172.21.15.67:0/1175101131 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5555da66a480 msgr2=0x5555da66a8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-03T21:21:32.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:32.431+0000 7f1069864700 1 --2- 172.21.15.67:0/1175101131 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5555da66a480 0x5555da66a8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:21:32.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:32.431+0000 7f1069864700 1 -- 172.21.15.67:0/1175101131 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5555da66e740 msgr2=0x5555da66eb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-03T21:21:32.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:32.431+0000 7f1069864700 1 --2- 172.21.15.67:0/1175101131 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5555da66e740 0x5555da66eb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:21:32.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:32.431+0000 7f1069864700 1 -- 172.21.15.67:0/1175101131 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5555da509880 con 0x5555da66f060 2024-05-03T21:21:32.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:32.431+0000 7f106a065700 1 --2- 172.21.15.67:0/1175101131 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5555da66e740 0x5555da66eb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-03T21:21:32.436 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:32.431+0000 7f1069864700 1 --2- 172.21.15.67:0/1175101131 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5555da66f060 0x5555da669f40 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f105400d780 tx=0x7f1054010000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c85b99d3ca350fa4 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T21:21:32.436 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:32.431+0000 7f1058d70700 1 -- 172.21.15.67:0/1175101131 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1054024070 con 0x5555da66f060 2024-05-03T21:21:32.436 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:32.431+0000 7f1058d70700 1 -- 172.21.15.67:0/1175101131 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(28 keys) ==== 1170+0+0 (secure 0 0 0) 0x7f1054014b40 con 0x5555da66f060 2024-05-03T21:21:32.436 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:32.431+0000 7f1058d70700 1 -- 172.21.15.67:0/1175101131 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f105401fcc0 con 0x5555da66f060 2024-05-03T21:21:32.439 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:32.432+0000 7f1076b5cb80 1 -- 172.21.15.67:0/1175101131 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5555da66f060 msgr2=0x5555da669f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:21:32.440 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:32.432+0000 7f1076b5cb80 1 --2- 172.21.15.67:0/1175101131 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5555da66f060 0x5555da669f40 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f105400d780 tx=0x7f1054010000 comp rx=0 tx=0).stop 2024-05-03T21:21:32.440 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:32.432+0000 7f1076b5cb80 1 -- 172.21.15.67:0/1175101131 shutdown_connections 2024-05-03T21:21:32.440 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:32.432+0000 7f1076b5cb80 1 --2- 172.21.15.67:0/1175101131 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5555da66a480 0x5555da66a8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:21:32.440 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:32.432+0000 7f1076b5cb80 1 --2- 172.21.15.67:0/1175101131 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5555da66e740 0x5555da66eb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:21:32.440 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:32.432+0000 7f1076b5cb80 1 --2- 172.21.15.67:0/1175101131 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5555da66f060 0x5555da669f40 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:21:32.440 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:32.432+0000 7f1076b5cb80 1 -- 172.21.15.67:0/1175101131 >> 172.21.15.67:0/1175101131 conn(0x5555da596b20 msgr2=0x5555da6649a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T21:21:32.440 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:32.432+0000 7f1076b5cb80 1 -- 172.21.15.67:0/1175101131 shutdown_connections 2024-05-03T21:21:32.440 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:32.432+0000 7f1076b5cb80 1 -- 172.21.15.67:0/1175101131 wait complete. 2024-05-03T21:21:32.440 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:32.432+0000 7f1076b5cb80 1 -- start start 2024-05-03T21:21:32.440 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:32.432+0000 7f1076b5cb80 1 Processor -- start 2024-05-03T21:21:32.440 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:32.433+0000 7f1076b5cb80 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5555da66a480 0x5555da675dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:21:32.440 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:32.433+0000 7f1076b5cb80 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5555da66e740 0x5555da673170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:21:32.440 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:32.433+0000 7f1076b5cb80 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5555da66f060 0x5555da6736b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:21:32.440 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:32.433+0000 7f1076b5cb80 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x5555da676ba0 con 0x5555da66a480 2024-05-03T21:21:32.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:32.433+0000 7f1076b5cb80 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x5555da575e80 con 0x5555da66f060 2024-05-03T21:21:32.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:32.433+0000 7f1076b5cb80 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x5555da676d20 con 0x5555da66e740 2024-05-03T21:21:32.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:32.433+0000 7f1076b5cb80 10 client.0 _ll_register_callbacks cb 0x5555da550560 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-03T21:21:32.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:32.433+0000 7f106a065700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5555da66a480 0x5555da675dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:21:32.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:32.433+0000 7f106a065700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5555da66a480 0x5555da675dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:48694/0 (socket says 172.21.15.67:48694) 2024-05-03T21:21:32.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:32.433+0000 7f106a065700 1 -- 172.21.15.67:0/3531502196 learned_addr learned my addr 172.21.15.67:0/3531502196 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-03T21:21:32.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:32.433+0000 7f1069864700 1 --2- 172.21.15.67:0/3531502196 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5555da66e740 0x5555da673170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:21:32.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:32.433+0000 7f106a866700 1 --2- 172.21.15.67:0/3531502196 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5555da66f060 0x5555da6736b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:21:32.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:32.433+0000 7f106a065700 1 -- 172.21.15.67:0/3531502196 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5555da66e740 msgr2=0x5555da673170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-03T21:21:32.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:32.433+0000 7f106a065700 1 --2- 172.21.15.67:0/3531502196 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5555da66e740 0x5555da673170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:21:32.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:32.433+0000 7f106a065700 1 -- 172.21.15.67:0/3531502196 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5555da66f060 msgr2=0x5555da6736b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-03T21:21:32.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:32.433+0000 7f106a065700 1 --2- 172.21.15.67:0/3531502196 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5555da66f060 0x5555da6736b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:21:32.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:32.433+0000 7f106a065700 1 -- 172.21.15.67:0/3531502196 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5555da573710 con 0x5555da66a480 2024-05-03T21:21:32.442 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:32.433+0000 7f106a866700 1 --2- 172.21.15.67:0/3531502196 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5555da66f060 0x5555da6736b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-03T21:21:32.442 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:32.433+0000 7f106a065700 1 --2- 172.21.15.67:0/3531502196 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5555da66a480 0x5555da675dd0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f10600a90b0 tx=0x7f10600df980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fcb639b64c38e79c server_cookie=0 in_seq=0 out_seq=0 2024-05-03T21:21:32.442 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:32.433+0000 7f104dffb700 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-05-03T21:21:32.442 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:32.433+0000 7f104dffb700 1 -- 172.21.15.67:0/3531502196 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10600a9120 con 0x5555da66a480 2024-05-03T21:21:32.442 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:32.434+0000 7f104dffb700 1 -- 172.21.15.67:0/3531502196 <== mon.0 v2:172.21.15.67:3300/0 2 ==== osd_map(84..84 src has 1..84) ==== 8211+0+0 (secure 0 0 0) 0x7f10600ad170 con 0x5555da66a480 2024-05-03T21:21:32.442 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:32.434+0000 7f1076b5cb80 1 -- 172.21.15.67:0/3531502196 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1054014b40 con 0x5555da66a480 2024-05-03T21:21:32.442 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:32.434+0000 7f104dffb700 1 -- 172.21.15.67:0/3531502196 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f10600ad4d0 con 0x5555da66a480 2024-05-03T21:21:32.442 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:32.434+0000 7f1076b5cb80 10 client.15078 fetch_fsmap learned FSMap version 25 2024-05-03T21:21:32.442 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:32.434+0000 7f1076b5cb80 1 -- 172.21.15.67:0/3531502196 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x5555da681430 con 0x5555da66a480 2024-05-03T21:21:32.442 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:32.434+0000 7f104dffb700 1 -- 172.21.15.67:0/3531502196 <== mon.0 v2:172.21.15.67:3300/0 4 ==== config(28 keys) ==== 1170+0+0 (secure 0 0 0) 0x7f1054014b40 con 0x5555da66a480 2024-05-03T21:21:32.442 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:32.434+0000 7f104dffb700 1 -- 172.21.15.67:0/3531502196 <== mon.0 v2:172.21.15.67:3300/0 5 ==== fsmap(e 25) ==== 8338+0+0 (secure 0 0 0) 0x7f10600dfb10 con 0x5555da66a480 2024-05-03T21:21:32.442 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:32.434+0000 7f1076b5cb80 10 client.15078 fetch_fsmap finished waiting for FSMap version 25 2024-05-03T21:21:32.442 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:32.434+0000 7f1076b5cb80 10 client.15078 resolve_mds: resolved name 'e' to daemon mds.e 2024-05-03T21:21:32.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:32.434+0000 7f1076b5cb80 20 client.15078 populate_metadata read hostname 'smithi067' 2024-05-03T21:21:32.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:32.434+0000 7f1076b5cb80 1 --2- 172.21.15.67:0/3531502196 >> [v2:172.21.15.82:6834/280518761,v1:172.21.15.82:6835/280518761] conn(0x5555da6839b0 0x5555da684e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:21:32.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:32.434+0000 7f1076b5cb80 4 client.15078 mds_command: new command op to 24425 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-03T21:21:32.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:32.434+0000 7f1076b5cb80 1 -- 172.21.15.67:0/3531502196 --> [v2:172.21.15.82:6834/280518761,v1:172.21.15.82:6835/280518761] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5555da5ca2d0 con 0x5555da6839b0 2024-05-03T21:21:32.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:32.434+0000 7f1069864700 1 --2- 172.21.15.67:0/3531502196 >> [v2:172.21.15.82:6834/280518761,v1:172.21.15.82:6835/280518761] conn(0x5555da6839b0 0x5555da684e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:21:32.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:32.435+0000 7f1069864700 1 --2- 172.21.15.67:0/3531502196 >> [v2:172.21.15.82:6834/280518761,v1:172.21.15.82:6835/280518761] conn(0x5555da6839b0 0x5555da684e20 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=7d8452887e7d7df3 server_cookie=6a5d8e2c9ece171 in_seq=0 out_seq=0 2024-05-03T21:21:32.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:32.435+0000 7f104dffb700 10 client.15078 ms_handle_connect on v2:172.21.15.82:6834/280518761 2024-05-03T21:21:32.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:32.436+0000 7f104dffb700 1 -- 172.21.15.67:0/3531502196 <== mds.2 v2:172.21.15.82:6834/280518761 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x5555da5ca2d0 con 0x5555da6839b0 2024-05-03T21:21:32.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:32.436+0000 7f104dffb700 10 client.15078 handle_command_reply: tid=0 2024-05-03T21:21:32.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:32.436+0000 7f1076b5cb80 1 client.15078 shutdown 2024-05-03T21:21:32.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:32.436+0000 7f1076b5cb80 1 -- 172.21.15.67:0/3531502196 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5555da66a480 msgr2=0x5555da675dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:21:32.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:32.436+0000 7f1076b5cb80 1 --2- 172.21.15.67:0/3531502196 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5555da66a480 0x5555da675dd0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f10600a90b0 tx=0x7f10600df980 comp rx=0 tx=0).stop 2024-05-03T21:21:32.444 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:32.436+0000 7f1076b5cb80 1 -- 172.21.15.67:0/3531502196 shutdown_connections 2024-05-03T21:21:32.444 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:32.436+0000 7f1076b5cb80 1 --2- 172.21.15.67:0/3531502196 >> [v2:172.21.15.82:6834/280518761,v1:172.21.15.82:6835/280518761] conn(0x5555da6839b0 0x5555da684e20 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:21:32.444 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:32.436+0000 7f1076b5cb80 1 --2- 172.21.15.67:0/3531502196 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5555da66a480 0x5555da675dd0 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:21:32.444 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:32.436+0000 7f1076b5cb80 1 --2- 172.21.15.67:0/3531502196 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5555da66f060 0x5555da6736b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:21:32.444 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:32.436+0000 7f1076b5cb80 1 --2- 172.21.15.67:0/3531502196 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5555da66e740 0x5555da673170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:21:32.444 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:32.436+0000 7f1076b5cb80 1 -- 172.21.15.67:0/3531502196 >> 172.21.15.67:0/3531502196 conn(0x5555da596b20 msgr2=0x5555da66fd90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T21:21:32.444 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:32.436+0000 7f104dffb700 0 client.15078 ms_handle_reset on v2:172.21.15.82:6834/280518761 2024-05-03T21:21:32.444 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:32.436+0000 7f1076b5cb80 1 -- 172.21.15.67:0/3531502196 shutdown_connections 2024-05-03T21:21:32.444 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:32.436+0000 7f1076b5cb80 1 -- 172.21.15.67:0/3531502196 wait complete. 2024-05-03T21:21:32.444 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:32.436+0000 7f1076b5cb80 20 client.15078 trim_cache size 0 max 16384 2024-05-03T21:21:32.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:32.457+0000 7f1076b5cb80 1 Processor -- start 2024-05-03T21:21:32.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:32.457+0000 7f1076b5cb80 1 -- start start 2024-05-03T21:21:32.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:32.457+0000 7f1076b5cb80 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5555da66a480 0x5555da675610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:21:32.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:32.457+0000 7f1076b5cb80 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5555da66e740 0x5555da664810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:21:32.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:32.457+0000 7f1076b5cb80 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5555da66f060 0x5555da673130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:21:32.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:32.457+0000 7f1076b5cb80 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x5555da683160 con 0x5555da66e740 2024-05-03T21:21:32.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:32.457+0000 7f1076b5cb80 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x5555da683830 con 0x5555da66a480 2024-05-03T21:21:32.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:32.457+0000 7f1076b5cb80 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x5555da683f00 con 0x5555da66f060 2024-05-03T21:21:32.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:32.457+0000 7f1069864700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5555da66e740 0x5555da664810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:21:32.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:32.457+0000 7f1069864700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5555da66e740 0x5555da664810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:48696/0 (socket says 172.21.15.67:48696) 2024-05-03T21:21:32.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:32.457+0000 7f1069864700 1 -- 172.21.15.67:0/2906948038 learned_addr learned my addr 172.21.15.67:0/2906948038 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-03T21:21:32.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:32.457+0000 7f106a065700 1 --2- 172.21.15.67:0/2906948038 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5555da66a480 0x5555da675610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:21:32.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:32.458+0000 7f106a866700 1 --2- 172.21.15.67:0/2906948038 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5555da66f060 0x5555da673130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:21:32.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:32.458+0000 7f1069864700 1 -- 172.21.15.67:0/2906948038 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5555da66f060 msgr2=0x5555da673130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-03T21:21:32.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:32.458+0000 7f1069864700 1 --2- 172.21.15.67:0/2906948038 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5555da66f060 0x5555da673130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:21:32.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:32.458+0000 7f1069864700 1 -- 172.21.15.67:0/2906948038 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5555da66a480 msgr2=0x5555da675610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-03T21:21:32.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:32.458+0000 7f1069864700 1 --2- 172.21.15.67:0/2906948038 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5555da66a480 0x5555da675610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:21:32.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:32.458+0000 7f1069864700 1 -- 172.21.15.67:0/2906948038 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5555da676cd0 con 0x5555da66e740 2024-05-03T21:21:32.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:32.458+0000 7f1069864700 1 --2- 172.21.15.67:0/2906948038 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5555da66e740 0x5555da664810 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f1054006fd0 tx=0x7f105401f420 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7c6885607278b740 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T21:21:32.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:32.458+0000 7f104ffff700 1 -- 172.21.15.67:0/2906948038 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1054024040 con 0x5555da66e740 2024-05-03T21:21:32.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:32.458+0000 7f104ffff700 1 -- 172.21.15.67:0/2906948038 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(28 keys) ==== 1170+0+0 (secure 0 0 0) 0x7f105401fe10 con 0x5555da66e740 2024-05-03T21:21:32.462 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:32.458+0000 7f104ffff700 1 -- 172.21.15.67:0/2906948038 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f105402f170 con 0x5555da66e740 2024-05-03T21:21:32.465 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:32.459+0000 7f1076b5cb80 1 -- 172.21.15.67:0/2906948038 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5555da66e740 msgr2=0x5555da664810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:21:32.465 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:32.459+0000 7f1076b5cb80 1 --2- 172.21.15.67:0/2906948038 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5555da66e740 0x5555da664810 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f1054006fd0 tx=0x7f105401f420 comp rx=0 tx=0).stop 2024-05-03T21:21:32.465 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:32.459+0000 7f1076b5cb80 1 -- 172.21.15.67:0/2906948038 shutdown_connections 2024-05-03T21:21:32.465 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:32.459+0000 7f1076b5cb80 1 --2- 172.21.15.67:0/2906948038 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5555da66f060 0x5555da673130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:21:32.465 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:32.459+0000 7f1076b5cb80 1 --2- 172.21.15.67:0/2906948038 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5555da66a480 0x5555da675610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:21:32.466 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:32.459+0000 7f1076b5cb80 1 --2- 172.21.15.67:0/2906948038 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5555da66e740 0x5555da664810 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:21:32.466 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:32.459+0000 7f1076b5cb80 1 -- 172.21.15.67:0/2906948038 >> 172.21.15.67:0/2906948038 conn(0x5555da596b20 msgr2=0x5555da66dd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T21:21:32.466 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:32.459+0000 7f1076b5cb80 1 -- 172.21.15.67:0/2906948038 shutdown_connections 2024-05-03T21:21:32.466 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:32.459+0000 7f1076b5cb80 1 -- 172.21.15.67:0/2906948038 wait complete. 2024-05-03T21:21:32.466 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:32.459+0000 7f1076b5cb80 1 -- start start 2024-05-03T21:21:32.466 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:32.459+0000 7f1076b5cb80 1 Processor -- start 2024-05-03T21:21:32.466 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:32.459+0000 7f1076b5cb80 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5555da66a480 0x5555da68d470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:21:32.467 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:32.460+0000 7f1076b5cb80 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5555da66e740 0x5555da68d9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:21:32.467 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:32.460+0000 7f1076b5cb80 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5555da66f060 0x5555da68def0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:21:32.467 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:32.460+0000 7f1076b5cb80 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x5555da684ca0 con 0x5555da66e740 2024-05-03T21:21:32.467 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:32.460+0000 7f1076b5cb80 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x5555da6845d0 con 0x5555da66f060 2024-05-03T21:21:32.467 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:32.460+0000 7f1076b5cb80 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x5555da685370 con 0x5555da66a480 2024-05-03T21:21:32.467 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:32.460+0000 7f1076b5cb80 10 client.0 _ll_register_callbacks cb 0x5555da550560 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-03T21:21:32.467 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:32.460+0000 7f1069864700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5555da66e740 0x5555da68d9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:21:32.467 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:32.460+0000 7f1069864700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5555da66e740 0x5555da68d9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:48706/0 (socket says 172.21.15.67:48706) 2024-05-03T21:21:32.467 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:32.460+0000 7f1069864700 1 -- 172.21.15.67:0/1171562185 learned_addr learned my addr 172.21.15.67:0/1171562185 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-03T21:21:32.467 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:32.460+0000 7f106a065700 1 --2- 172.21.15.67:0/1171562185 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5555da66a480 0x5555da68d470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:21:32.467 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:32.460+0000 7f106a866700 1 --2- 172.21.15.67:0/1171562185 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5555da66f060 0x5555da68def0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:21:32.467 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:32.460+0000 7f1069864700 1 -- 172.21.15.67:0/1171562185 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5555da66a480 msgr2=0x5555da68d470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-03T21:21:32.467 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:32.460+0000 7f1069864700 1 --2- 172.21.15.67:0/1171562185 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5555da66a480 0x5555da68d470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:21:32.467 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:32.460+0000 7f1069864700 1 -- 172.21.15.67:0/1171562185 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5555da66f060 msgr2=0x5555da68def0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-03T21:21:32.468 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:32.460+0000 7f1069864700 1 --2- 172.21.15.67:0/1171562185 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5555da66f060 0x5555da68def0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:21:32.468 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:32.460+0000 7f1069864700 1 -- 172.21.15.67:0/1171562185 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5555da68e730 con 0x5555da66e740 2024-05-03T21:21:32.468 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:32.460+0000 7f106a866700 1 --2- 172.21.15.67:0/1171562185 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5555da66f060 0x5555da68def0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-03T21:21:32.468 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:32.460+0000 7f1069864700 1 --2- 172.21.15.67:0/1171562185 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5555da66e740 0x5555da68d9b0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f1054022040 tx=0x7f105401f550 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8468a989d504b19b server_cookie=0 in_seq=0 out_seq=0 2024-05-03T21:21:32.468 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:32.460+0000 7f104dffb700 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-05-03T21:21:32.468 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:32.460+0000 7f104dffb700 1 -- 172.21.15.67:0/1171562185 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f105403e020 con 0x5555da66e740 2024-05-03T21:21:32.468 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:32.460+0000 7f104dffb700 1 -- 172.21.15.67:0/1171562185 <== mon.0 v2:172.21.15.67:3300/0 2 ==== osd_map(84..84 src has 1..84) ==== 8211+0+0 (secure 0 0 0) 0x7f105401f910 con 0x5555da66e740 2024-05-03T21:21:32.468 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:32.460+0000 7f1076b5cb80 1 -- 172.21.15.67:0/1171562185 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f105401fe10 con 0x5555da66e740 2024-05-03T21:21:32.468 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:32.461+0000 7f104dffb700 1 -- 172.21.15.67:0/1171562185 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f1054024040 con 0x5555da66e740 2024-05-03T21:21:32.468 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:32.461+0000 7f1076b5cb80 10 client.15084 fetch_fsmap learned FSMap version 25 2024-05-03T21:21:32.468 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:32.461+0000 7f1076b5cb80 1 -- 172.21.15.67:0/1171562185 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x5555da670040 con 0x5555da66e740 2024-05-03T21:21:32.468 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:32.461+0000 7f104dffb700 1 -- 172.21.15.67:0/1171562185 <== mon.0 v2:172.21.15.67:3300/0 4 ==== config(28 keys) ==== 1170+0+0 (secure 0 0 0) 0x7f105401fe10 con 0x5555da66e740 2024-05-03T21:21:32.468 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:32.461+0000 7f104dffb700 1 -- 172.21.15.67:0/1171562185 <== mon.0 v2:172.21.15.67:3300/0 5 ==== fsmap(e 25) ==== 8338+0+0 (secure 0 0 0) 0x7f1054013070 con 0x5555da66e740 2024-05-03T21:21:32.468 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:32.461+0000 7f1076b5cb80 10 client.15084 fetch_fsmap finished waiting for FSMap version 25 2024-05-03T21:21:32.469 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:32.461+0000 7f1076b5cb80 10 client.15084 resolve_mds: resolved name 'e' to daemon mds.e 2024-05-03T21:21:32.469 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:32.461+0000 7f1076b5cb80 20 client.15084 populate_metadata read hostname 'smithi067' 2024-05-03T21:21:32.469 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:32.461+0000 7f1076b5cb80 1 --2- 172.21.15.67:0/1171562185 >> [v2:172.21.15.82:6834/280518761,v1:172.21.15.82:6835/280518761] conn(0x5555da6f3ef0 0x5555da6f2c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:21:32.469 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:32.461+0000 7f1076b5cb80 4 client.15084 mds_command: new command op to 24425 tid=0 multi_id=0 [{"prefix": "status"}] 2024-05-03T21:21:32.469 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:32.461+0000 7f1076b5cb80 1 -- 172.21.15.67:0/1171562185 --> [v2:172.21.15.82:6834/280518761,v1:172.21.15.82:6835/280518761] -- command(tid 0: {"prefix": "status"}) -- 0x5555da5c4370 con 0x5555da6f3ef0 2024-05-03T21:21:32.469 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:32.461+0000 7f106a065700 1 --2- 172.21.15.67:0/1171562185 >> [v2:172.21.15.82:6834/280518761,v1:172.21.15.82:6835/280518761] conn(0x5555da6f3ef0 0x5555da6f2c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:21:32.469 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:32.462+0000 7f106a065700 1 --2- 172.21.15.67:0/1171562185 >> [v2:172.21.15.82:6834/280518761,v1:172.21.15.82:6835/280518761] conn(0x5555da6f3ef0 0x5555da6f2c00 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=b9692e3aeb12596e server_cookie=7f984927cf7668b0 in_seq=0 out_seq=0 2024-05-03T21:21:32.469 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:32.462+0000 7f104dffb700 10 client.15084 ms_handle_connect on v2:172.21.15.82:6834/280518761 2024-05-03T21:21:32.469 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:32.462+0000 7f104dffb700 1 -- 172.21.15.67:0/1171562185 <== mds.2 v2:172.21.15.82:6834/280518761 1 ==== command_reply(tid 0: 0 ) ==== 8+0+327 (crc 0 0 0) 0x5555da5c4370 con 0x5555da6f3ef0 2024-05-03T21:21:32.469 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:32.462+0000 7f104dffb700 10 client.15084 handle_command_reply: tid=0 2024-05-03T21:21:32.469 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:32.462+0000 7f1076b5cb80 1 client.15084 shutdown 2024-05-03T21:21:32.469 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:32.462+0000 7f1076b5cb80 1 -- 172.21.15.67:0/1171562185 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5555da66e740 msgr2=0x5555da68d9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:21:32.469 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:32.462+0000 7f1076b5cb80 1 --2- 172.21.15.67:0/1171562185 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5555da66e740 0x5555da68d9b0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f1054022040 tx=0x7f105401f550 comp rx=0 tx=0).stop 2024-05-03T21:21:32.469 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:32.462+0000 7f1076b5cb80 1 -- 172.21.15.67:0/1171562185 shutdown_connections 2024-05-03T21:21:32.470 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:32.462+0000 7f1076b5cb80 1 --2- 172.21.15.67:0/1171562185 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5555da66f060 0x5555da68def0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:21:32.470 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:32.462+0000 7f1076b5cb80 1 --2- 172.21.15.67:0/1171562185 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5555da66a480 0x5555da68d470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:21:32.470 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:32.462+0000 7f1076b5cb80 1 --2- 172.21.15.67:0/1171562185 >> [v2:172.21.15.82:6834/280518761,v1:172.21.15.82:6835/280518761] conn(0x5555da6f3ef0 0x5555da6f2c00 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:21:32.470 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:32.462+0000 7f1076b5cb80 1 --2- 172.21.15.67:0/1171562185 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5555da66e740 0x5555da68d9b0 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:21:32.470 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:32.462+0000 7f104dffb700 0 client.15084 ms_handle_reset on v2:172.21.15.82:6834/280518761 2024-05-03T21:21:32.470 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:32.462+0000 7f1076b5cb80 1 -- 172.21.15.67:0/1171562185 >> 172.21.15.67:0/1171562185 conn(0x5555da596b20 msgr2=0x5555da67f870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T21:21:32.470 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:32.462+0000 7f1076b5cb80 1 -- 172.21.15.67:0/1171562185 shutdown_connections 2024-05-03T21:21:32.470 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:32.462+0000 7f1076b5cb80 1 -- 172.21.15.67:0/1171562185 wait complete. 2024-05-03T21:21:32.470 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:32.462+0000 7f1076b5cb80 20 client.15084 trim_cache size 0 max 16384 2024-05-03T21:21:32.471 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-05-03T21:21:32.471 INFO:teuthology.orchestra.run.smithi067.stdout: "cluster_fsid": "713bc57a-0991-11ef-bc97-c7b262605968", 2024-05-03T21:21:32.471 INFO:teuthology.orchestra.run.smithi067.stdout: "whoami": 2, 2024-05-03T21:21:32.471 INFO:teuthology.orchestra.run.smithi067.stdout: "id": 24425, 2024-05-03T21:21:32.471 INFO:teuthology.orchestra.run.smithi067.stdout: "want_state": "up:active", 2024-05-03T21:21:32.471 INFO:teuthology.orchestra.run.smithi067.stdout: "state": "up:active", 2024-05-03T21:21:32.471 INFO:teuthology.orchestra.run.smithi067.stdout: "fs_name": "cephfs", 2024-05-03T21:21:32.471 INFO:teuthology.orchestra.run.smithi067.stdout: "rank_uptime": 154.598423787, 2024-05-03T21:21:32.471 INFO:teuthology.orchestra.run.smithi067.stdout: "mdsmap_epoch": 25, 2024-05-03T21:21:32.471 INFO:teuthology.orchestra.run.smithi067.stdout: "osdmap_epoch": 84, 2024-05-03T21:21:32.473 INFO:teuthology.orchestra.run.smithi067.stdout: "osdmap_epoch_barrier": 82, 2024-05-03T21:21:32.473 INFO:teuthology.orchestra.run.smithi067.stdout: "uptime": 203.09495421099999 2024-05-03T21:21:32.473 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-05-03T21:21:32.474 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:32.463+0000 7f106c2c9700 1 -- 172.21.15.67:0/4282601850 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f10500b2910 msgr2=0x7f10500b4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:21:32.474 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:32.463+0000 7f106c2c9700 1 --2- 172.21.15.67:0/4282601850 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f10500b2910 0x7f10500b4dd0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f1064111a90 tx=0x7f1054008040 comp rx=0 tx=0).stop 2024-05-03T21:21:32.474 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:32.463+0000 7f106c2c9700 1 -- 172.21.15.67:0/4282601850 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f10640756d0 msgr2=0x7f106419c060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:21:32.475 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:32.463+0000 7f106c2c9700 1 --2- 172.21.15.67:0/4282601850 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f10640756d0 0x7f106419c060 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f106000b530 tx=0x7f106000b8f0 comp rx=0 tx=0).stop 2024-05-03T21:21:32.475 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:32.463+0000 7f106c2c9700 1 -- 172.21.15.67:0/4282601850 shutdown_connections 2024-05-03T21:21:32.475 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:32.463+0000 7f106c2c9700 1 --2- 172.21.15.67:0/4282601850 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f10500b2910 0x7f10500b4dd0 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:21:32.475 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:32.463+0000 7f106c2c9700 1 --2- 172.21.15.67:0/4282601850 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f10640756d0 0x7f106419c060 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:21:32.475 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:32.463+0000 7f106c2c9700 1 --2- 172.21.15.67:0/4282601850 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f106410fed0 0x7f10641a0930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:21:32.475 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:32.463+0000 7f106c2c9700 1 --2- 172.21.15.67:0/4282601850 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f1064075ff0 0x7f106419c5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:21:32.475 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:32.463+0000 7f106c2c9700 1 -- 172.21.15.67:0/4282601850 >> 172.21.15.67:0/4282601850 conn(0x7f106407a610 msgr2=0x7f1064100640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T21:21:32.475 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:32.463+0000 7f106c2c9700 1 -- 172.21.15.67:0/4282601850 shutdown_connections 2024-05-03T21:21:32.475 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:32.463+0000 7f106c2c9700 1 -- 172.21.15.67:0/4282601850 wait complete. 2024-05-03T21:21:33.120 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9a047fe7599a549042d06a3fb5b6813a203eabea shell --fsid 713bc57a-0991-11ef-bc97-c7b262605968 -- ceph tell mds.h status 2024-05-03T21:21:33.369 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/713bc57a-0991-11ef-bc97-c7b262605968/mon.a/config 2024-05-03T21:21:33.574 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:21:33 smithi138 ceph-mon[33110]: pgmap v373: 97 pgs: 97 active+clean; 632 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:21:33.611 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:21:33 smithi067 ceph-mon[29730]: pgmap v373: 97 pgs: 97 active+clean; 632 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:21:33.745 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:21:33 smithi082 ceph-mon[32933]: pgmap v373: 97 pgs: 97 active+clean; 632 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:21:35.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:35.037+0000 7efe4dac8700 1 -- 172.21.15.67:0/3574428326 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efe48108850 msgr2=0x7efe481103d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:21:35.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:35.037+0000 7efe4dac8700 1 --2- 172.21.15.67:0/3574428326 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efe48108850 0x7efe481103d0 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7efe4000b0d0 tx=0x7efe4000b3e0 comp rx=0 tx=0).stop 2024-05-03T21:21:35.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:35.037+0000 7efe4dac8700 1 -- 172.21.15.67:0/3574428326 shutdown_connections 2024-05-03T21:21:35.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:35.037+0000 7efe4dac8700 1 --2- 172.21.15.67:0/3574428326 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efe48108850 0x7efe481103d0 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:21:35.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:35.037+0000 7efe4dac8700 1 --2- 172.21.15.67:0/3574428326 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7efe48103750 0x7efe48103b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:21:35.041 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:35.037+0000 7efe4dac8700 1 --2- 172.21.15.67:0/3574428326 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7efe48104100 0x7efe48108120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:21:35.041 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:35.037+0000 7efe4dac8700 1 -- 172.21.15.67:0/3574428326 >> 172.21.15.67:0/3574428326 conn(0x7efe4807a600 msgr2=0x7efe4807aa10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T21:21:35.041 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:35.037+0000 7efe4dac8700 1 -- 172.21.15.67:0/3574428326 shutdown_connections 2024-05-03T21:21:35.041 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:35.037+0000 7efe4dac8700 1 -- 172.21.15.67:0/3574428326 wait complete. 2024-05-03T21:21:35.041 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:35.038+0000 7efe4dac8700 1 Processor -- start 2024-05-03T21:21:35.043 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:35.038+0000 7efe4dac8700 1 -- start start 2024-05-03T21:21:35.043 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:35.038+0000 7efe4dac8700 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7efe48103750 0x7efe4819bdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:21:35.043 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:35.038+0000 7efe4dac8700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7efe48104100 0x7efe4819c300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:21:35.043 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:35.039+0000 7efe4dac8700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efe48108850 0x7efe481a0690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:21:35.043 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:35.039+0000 7efe4dac8700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7efe481138b0 con 0x7efe48108850 2024-05-03T21:21:35.043 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:35.039+0000 7efe4dac8700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7efe48113730 con 0x7efe48104100 2024-05-03T21:21:35.043 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:35.039+0000 7efe4dac8700 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7efe48113a30 con 0x7efe48103750 2024-05-03T21:21:35.043 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:35.039+0000 7efe47fff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efe48108850 0x7efe481a0690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:21:35.043 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:35.039+0000 7efe47fff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efe48108850 0x7efe481a0690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:48724/0 (socket says 172.21.15.67:48724) 2024-05-03T21:21:35.043 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:35.039+0000 7efe47fff700 1 -- 172.21.15.67:0/3572268826 learned_addr learned my addr 172.21.15.67:0/3572268826 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-03T21:21:35.044 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:35.039+0000 7efe477fe700 1 --2- 172.21.15.67:0/3572268826 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7efe48103750 0x7efe4819bdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:21:35.044 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:35.039+0000 7efe46ffd700 1 --2- 172.21.15.67:0/3572268826 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7efe48104100 0x7efe4819c300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:21:35.044 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:35.039+0000 7efe47fff700 1 -- 172.21.15.67:0/3572268826 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7efe48103750 msgr2=0x7efe4819bdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:21:35.044 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:35.039+0000 7efe47fff700 1 --2- 172.21.15.67:0/3572268826 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7efe48103750 0x7efe4819bdc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:21:35.044 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:35.039+0000 7efe47fff700 1 -- 172.21.15.67:0/3572268826 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7efe48104100 msgr2=0x7efe4819c300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:21:35.044 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:35.039+0000 7efe47fff700 1 --2- 172.21.15.67:0/3572268826 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7efe48104100 0x7efe4819c300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:21:35.044 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:35.039+0000 7efe47fff700 1 -- 172.21.15.67:0/3572268826 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efe481a0dd0 con 0x7efe48108850 2024-05-03T21:21:35.044 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:35.039+0000 7efe477fe700 1 --2- 172.21.15.67:0/3572268826 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7efe48103750 0x7efe4819bdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-03T21:21:35.044 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:35.039+0000 7efe46ffd700 1 --2- 172.21.15.67:0/3572268826 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7efe48104100 0x7efe4819c300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-03T21:21:35.044 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:35.039+0000 7efe47fff700 1 --2- 172.21.15.67:0/3572268826 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efe48108850 0x7efe481a0690 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7efe4000d010 tx=0x7efe40007ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T21:21:35.044 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:35.039+0000 7efe44ff9700 1 -- 172.21.15.67:0/3572268826 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe4001f070 con 0x7efe48108850 2024-05-03T21:21:35.044 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:35.039+0000 7efe44ff9700 1 -- 172.21.15.67:0/3572268826 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(28 keys) ==== 1170+0+0 (secure 0 0 0) 0x7efe40003c00 con 0x7efe48108850 2024-05-03T21:21:35.044 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:35.039+0000 7efe4dac8700 1 -- 172.21.15.67:0/3572268826 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efe481a1060 con 0x7efe48108850 2024-05-03T21:21:35.045 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:35.039+0000 7efe44ff9700 1 -- 172.21.15.67:0/3572268826 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe4001b840 con 0x7efe48108850 2024-05-03T21:21:35.045 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:35.039+0000 7efe4dac8700 1 -- 172.21.15.67:0/3572268826 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efe481a8940 con 0x7efe48108850 2024-05-03T21:21:35.047 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:35.042+0000 7efe44ff9700 1 -- 172.21.15.67:0/3572268826 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 17) ==== 149703+0+0 (secure 0 0 0) 0x7efe40022070 con 0x7efe48108850 2024-05-03T21:21:35.048 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:35.042+0000 7efe44ff9700 1 --2- 172.21.15.67:0/3572268826 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7efe300b2990 0x7efe300b4e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:21:35.048 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:35.043+0000 7efe477fe700 1 --2- 172.21.15.67:0/3572268826 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7efe300b2990 0x7efe300b4e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:21:35.048 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:35.043+0000 7efe44ff9700 1 -- 172.21.15.67:0/3572268826 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 8211+0+0 (secure 0 0 0) 0x7efe4000f020 con 0x7efe48108850 2024-05-03T21:21:35.048 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:35.043+0000 7efe477fe700 1 --2- 172.21.15.67:0/3572268826 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7efe300b2990 0x7efe300b4e50 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7efe3c005950 tx=0x7efe3c0058e0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T21:21:35.048 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:35.043+0000 7efe5835bb80 1 Processor -- start 2024-05-03T21:21:35.048 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:35.043+0000 7efe5835bb80 1 -- start start 2024-05-03T21:21:35.048 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:35.043+0000 7efe5835bb80 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55ee15978c10 0x55ee15978ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:21:35.048 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:35.043+0000 7efe5835bb80 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ee15979530 0x55ee1597ccf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:21:35.048 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:35.043+0000 7efe5835bb80 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55ee1597d230 0x55ee1597d690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:21:35.048 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:35.043+0000 7efe5835bb80 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x55ee158cb810 con 0x55ee15979530 2024-05-03T21:21:35.049 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:35.043+0000 7efe5835bb80 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x55ee1586a770 con 0x55ee1597d230 2024-05-03T21:21:35.049 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:35.043+0000 7efe5835bb80 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x55ee158c77d0 con 0x55ee15978c10 2024-05-03T21:21:35.049 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:35.043+0000 7efe46ffd700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ee15979530 0x55ee1597ccf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:21:35.049 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:35.043+0000 7efe46ffd700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ee15979530 0x55ee1597ccf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:48738/0 (socket says 172.21.15.67:48738) 2024-05-03T21:21:35.049 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:35.043+0000 7efe46ffd700 1 -- 172.21.15.67:0/99833962 learned_addr learned my addr 172.21.15.67:0/99833962 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-03T21:21:35.049 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:35.043+0000 7efe477fe700 1 --2- 172.21.15.67:0/99833962 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55ee15978c10 0x55ee15978ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:21:35.049 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:35.044+0000 7efe47fff700 1 --2- 172.21.15.67:0/99833962 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55ee1597d230 0x55ee1597d690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:21:35.049 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:35.044+0000 7efe46ffd700 1 -- 172.21.15.67:0/99833962 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55ee15978c10 msgr2=0x55ee15978ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-03T21:21:35.049 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:35.044+0000 7efe46ffd700 1 --2- 172.21.15.67:0/99833962 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55ee15978c10 0x55ee15978ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:21:35.049 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:35.044+0000 7efe46ffd700 1 -- 172.21.15.67:0/99833962 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55ee1597d230 msgr2=0x55ee1597d690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-03T21:21:35.049 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:35.044+0000 7efe46ffd700 1 --2- 172.21.15.67:0/99833962 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55ee1597d230 0x55ee1597d690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:21:35.049 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:35.044+0000 7efe46ffd700 1 -- 172.21.15.67:0/99833962 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ee15818880 con 0x55ee15979530 2024-05-03T21:21:35.049 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:35.044+0000 7efe46ffd700 1 --2- 172.21.15.67:0/99833962 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ee15979530 0x55ee1597ccf0 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7efe38009ca0 tx=0x7efe3800e2f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c49e57d8dcbc8840 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T21:21:35.049 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:35.044+0000 7efe2dffb700 1 -- 172.21.15.67:0/99833962 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe38016070 con 0x55ee15979530 2024-05-03T21:21:35.050 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:35.044+0000 7efe2dffb700 1 -- 172.21.15.67:0/99833962 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(28 keys) ==== 1170+0+0 (secure 0 0 0) 0x7efe38004d30 con 0x55ee15979530 2024-05-03T21:21:35.050 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:35.044+0000 7efe2dffb700 1 -- 172.21.15.67:0/99833962 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe38011470 con 0x55ee15979530 2024-05-03T21:21:35.050 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:35.045+0000 7efe5835bb80 1 -- 172.21.15.67:0/99833962 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ee15979530 msgr2=0x55ee1597ccf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:21:35.050 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:35.045+0000 7efe5835bb80 1 --2- 172.21.15.67:0/99833962 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ee15979530 0x55ee1597ccf0 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7efe38009ca0 tx=0x7efe3800e2f0 comp rx=0 tx=0).stop 2024-05-03T21:21:35.050 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:35.045+0000 7efe5835bb80 1 -- 172.21.15.67:0/99833962 shutdown_connections 2024-05-03T21:21:35.050 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:35.045+0000 7efe5835bb80 1 --2- 172.21.15.67:0/99833962 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55ee1597d230 0x55ee1597d690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:21:35.050 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:35.045+0000 7efe5835bb80 1 --2- 172.21.15.67:0/99833962 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55ee15978c10 0x55ee15978ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:21:35.050 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:35.045+0000 7efe5835bb80 1 --2- 172.21.15.67:0/99833962 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ee15979530 0x55ee1597ccf0 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:21:35.050 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:35.045+0000 7efe5835bb80 1 -- 172.21.15.67:0/99833962 >> 172.21.15.67:0/99833962 conn(0x55ee158a50b0 msgr2=0x55ee15973280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T21:21:35.050 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:35.045+0000 7efe5835bb80 1 -- 172.21.15.67:0/99833962 shutdown_connections 2024-05-03T21:21:35.051 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:35.045+0000 7efe5835bb80 1 -- 172.21.15.67:0/99833962 wait complete. 2024-05-03T21:21:35.051 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:35.045+0000 7efe5835bb80 1 -- start start 2024-05-03T21:21:35.051 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:35.046+0000 7efe5835bb80 1 Processor -- start 2024-05-03T21:21:35.054 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:35.046+0000 7efe5835bb80 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55ee15978c10 0x55ee15984650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:21:35.054 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:35.046+0000 7efe5835bb80 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55ee15979530 0x55ee15981a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:21:35.054 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:35.046+0000 7efe5835bb80 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ee1597d230 0x55ee15981f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:21:35.054 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:35.046+0000 7efe5835bb80 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x55ee15985490 con 0x55ee1597d230 2024-05-03T21:21:35.054 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:35.046+0000 7efe5835bb80 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x55ee15831be0 con 0x55ee15978c10 2024-05-03T21:21:35.055 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:35.046+0000 7efe5835bb80 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x55ee15985610 con 0x55ee15979530 2024-05-03T21:21:35.055 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:35.046+0000 7efe5835bb80 10 client.0 _ll_register_callbacks cb 0x55ee158638d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-03T21:21:35.055 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:35.046+0000 7efe47fff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ee1597d230 0x55ee15981f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:21:35.055 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:35.046+0000 7efe47fff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ee1597d230 0x55ee15981f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:48746/0 (socket says 172.21.15.67:48746) 2024-05-03T21:21:35.055 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:35.046+0000 7efe47fff700 1 -- 172.21.15.67:0/3150280091 learned_addr learned my addr 172.21.15.67:0/3150280091 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-03T21:21:35.055 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:35.046+0000 7efe46ffd700 1 --2- 172.21.15.67:0/3150280091 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55ee15979530 0x55ee15981a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:21:35.055 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:35.046+0000 7efe477fe700 1 --2- 172.21.15.67:0/3150280091 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55ee15978c10 0x55ee15984650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:21:35.055 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:35.046+0000 7efe47fff700 1 -- 172.21.15.67:0/3150280091 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55ee15979530 msgr2=0x55ee15981a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-03T21:21:35.055 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:35.046+0000 7efe47fff700 1 --2- 172.21.15.67:0/3150280091 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55ee15979530 0x55ee15981a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:21:35.055 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:35.046+0000 7efe47fff700 1 -- 172.21.15.67:0/3150280091 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55ee15978c10 msgr2=0x55ee15984650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-03T21:21:35.055 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:35.046+0000 7efe47fff700 1 --2- 172.21.15.67:0/3150280091 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55ee15978c10 0x55ee15984650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:21:35.055 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:35.046+0000 7efe47fff700 1 -- 172.21.15.67:0/3150280091 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ee15867640 con 0x55ee1597d230 2024-05-03T21:21:35.055 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:35.046+0000 7efe477fe700 1 --2- 172.21.15.67:0/3150280091 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55ee15978c10 0x55ee15984650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-03T21:21:35.055 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:35.047+0000 7efe47fff700 1 --2- 172.21.15.67:0/3150280091 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ee1597d230 0x55ee15981f80 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7efe40009850 tx=0x7efe400aa3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ccc10380f49968ce server_cookie=0 in_seq=0 out_seq=0 2024-05-03T21:21:35.056 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:35.047+0000 7efe26ffd700 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-05-03T21:21:35.056 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:35.047+0000 7efe26ffd700 1 -- 172.21.15.67:0/3150280091 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe400aa630 con 0x55ee1597d230 2024-05-03T21:21:35.056 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:35.047+0000 7efe26ffd700 1 -- 172.21.15.67:0/3150280091 <== mon.0 v2:172.21.15.67:3300/0 2 ==== osd_map(84..84 src has 1..84) ==== 8211+0+0 (secure 0 0 0) 0x7efe400b3460 con 0x55ee1597d230 2024-05-03T21:21:35.056 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:35.047+0000 7efe5835bb80 1 -- 172.21.15.67:0/3150280091 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7efe38004d30 con 0x55ee1597d230 2024-05-03T21:21:35.056 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:35.047+0000 7efe26ffd700 1 -- 172.21.15.67:0/3150280091 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7efe400ab220 con 0x55ee1597d230 2024-05-03T21:21:35.056 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:35.047+0000 7efe5835bb80 10 client.15096 fetch_fsmap learned FSMap version 25 2024-05-03T21:21:35.056 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:35.047+0000 7efe5835bb80 1 -- 172.21.15.67:0/3150280091 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55ee1598fcc0 con 0x55ee1597d230 2024-05-03T21:21:35.056 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:35.047+0000 7efe26ffd700 1 -- 172.21.15.67:0/3150280091 <== mon.0 v2:172.21.15.67:3300/0 4 ==== config(28 keys) ==== 1170+0+0 (secure 0 0 0) 0x7efe38004d30 con 0x55ee1597d230 2024-05-03T21:21:35.056 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:35.047+0000 7efe26ffd700 1 -- 172.21.15.67:0/3150280091 <== mon.0 v2:172.21.15.67:3300/0 5 ==== fsmap(e 25) ==== 8338+0+0 (secure 0 0 0) 0x7efe40011070 con 0x55ee1597d230 2024-05-03T21:21:35.056 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:35.047+0000 7efe5835bb80 10 client.15096 fetch_fsmap finished waiting for FSMap version 25 2024-05-03T21:21:35.056 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:35.048+0000 7efe5835bb80 10 client.15096 resolve_mds: resolved name 'h' to daemon mds.h 2024-05-03T21:21:35.056 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:35.048+0000 7efe5835bb80 20 client.15096 populate_metadata read hostname 'smithi067' 2024-05-03T21:21:35.056 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:35.048+0000 7efe5835bb80 1 --2- 172.21.15.67:0/3150280091 >> [v2:172.21.15.82:6836/1275987852,v1:172.21.15.82:6837/1275987852] conn(0x55ee15992360 0x55ee159937d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:21:35.056 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:35.048+0000 7efe5835bb80 4 client.15096 mds_command: new command op to 24449 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-03T21:21:35.056 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:35.048+0000 7efe5835bb80 1 -- 172.21.15.67:0/3150280091 --> [v2:172.21.15.82:6836/1275987852,v1:172.21.15.82:6837/1275987852] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ee158d91f0 con 0x55ee15992360 2024-05-03T21:21:35.057 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:35.048+0000 7efe477fe700 1 --2- 172.21.15.67:0/3150280091 >> [v2:172.21.15.82:6836/1275987852,v1:172.21.15.82:6837/1275987852] conn(0x55ee15992360 0x55ee159937d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:21:35.057 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:35.048+0000 7efe26ffd700 10 client.15096 ms_handle_connect on v2:172.21.15.82:6836/1275987852 2024-05-03T21:21:35.057 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:35.048+0000 7efe477fe700 1 --2- 172.21.15.67:0/3150280091 >> [v2:172.21.15.82:6836/1275987852,v1:172.21.15.82:6837/1275987852] conn(0x55ee15992360 0x55ee159937d0 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=80048be8d4eab990 server_cookie=6a371c87e7898c82 in_seq=0 out_seq=0 2024-05-03T21:21:35.057 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:35.049+0000 7efe26ffd700 1 -- 172.21.15.67:0/3150280091 <== mds.1 v2:172.21.15.82:6836/1275987852 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55ee158d91f0 con 0x55ee15992360 2024-05-03T21:21:35.057 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:35.049+0000 7efe26ffd700 10 client.15096 handle_command_reply: tid=0 2024-05-03T21:21:35.057 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:35.049+0000 7efe5835bb80 1 client.15096 shutdown 2024-05-03T21:21:35.057 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:35.049+0000 7efe5835bb80 1 -- 172.21.15.67:0/3150280091 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ee1597d230 msgr2=0x55ee15981f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:21:35.057 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:35.049+0000 7efe5835bb80 1 --2- 172.21.15.67:0/3150280091 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ee1597d230 0x55ee15981f80 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7efe40009850 tx=0x7efe400aa3c0 comp rx=0 tx=0).stop 2024-05-03T21:21:35.057 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:35.049+0000 7efe5835bb80 1 -- 172.21.15.67:0/3150280091 shutdown_connections 2024-05-03T21:21:35.058 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:35.049+0000 7efe5835bb80 1 --2- 172.21.15.67:0/3150280091 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ee1597d230 0x55ee15981f80 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:21:35.058 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:35.049+0000 7efe5835bb80 1 --2- 172.21.15.67:0/3150280091 >> [v2:172.21.15.82:6836/1275987852,v1:172.21.15.82:6837/1275987852] conn(0x55ee15992360 0x55ee159937d0 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:21:35.058 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:35.049+0000 7efe5835bb80 1 --2- 172.21.15.67:0/3150280091 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55ee15978c10 0x55ee15984650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:21:35.058 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:35.049+0000 7efe5835bb80 1 --2- 172.21.15.67:0/3150280091 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55ee15979530 0x55ee15981a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:21:35.058 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:35.049+0000 7efe26ffd700 0 client.15096 ms_handle_reset on v2:172.21.15.82:6836/1275987852 2024-05-03T21:21:35.058 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:35.049+0000 7efe5835bb80 1 -- 172.21.15.67:0/3150280091 >> 172.21.15.67:0/3150280091 conn(0x55ee158a50b0 msgr2=0x55ee159728f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T21:21:35.058 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:35.050+0000 7efe5835bb80 1 -- 172.21.15.67:0/3150280091 shutdown_connections 2024-05-03T21:21:35.058 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:35.050+0000 7efe5835bb80 1 -- 172.21.15.67:0/3150280091 wait complete. 2024-05-03T21:21:35.058 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:35.050+0000 7efe5835bb80 20 client.15096 trim_cache size 0 max 16384 2024-05-03T21:21:35.076 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:35.074+0000 7efe5835bb80 1 Processor -- start 2024-05-03T21:21:35.076 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:35.074+0000 7efe5835bb80 1 -- start start 2024-05-03T21:21:35.076 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:35.074+0000 7efe5835bb80 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55ee15978c10 0x55ee1596fb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:21:35.076 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:35.074+0000 7efe5835bb80 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55ee15979530 0x55ee1597c370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:21:35.076 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:35.074+0000 7efe5835bb80 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ee1597d230 0x55ee15973070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:21:35.076 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:35.074+0000 7efe5835bb80 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x55ee15991b10 con 0x55ee1597d230 2024-05-03T21:21:35.076 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:35.074+0000 7efe5835bb80 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x55ee159921e0 con 0x55ee15979530 2024-05-03T21:21:35.077 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:35.074+0000 7efe5835bb80 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x55ee159928b0 con 0x55ee15978c10 2024-05-03T21:21:35.077 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:35.074+0000 7efe47fff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ee1597d230 0x55ee15973070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:21:35.077 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:35.074+0000 7efe47fff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ee1597d230 0x55ee15973070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:48762/0 (socket says 172.21.15.67:48762) 2024-05-03T21:21:35.077 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:35.074+0000 7efe47fff700 1 -- 172.21.15.67:0/4075575278 learned_addr learned my addr 172.21.15.67:0/4075575278 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-03T21:21:35.077 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:35.074+0000 7efe46ffd700 1 --2- 172.21.15.67:0/4075575278 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55ee15979530 0x55ee1597c370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:21:35.077 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:35.074+0000 7efe47fff700 1 -- 172.21.15.67:0/4075575278 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55ee15978c10 msgr2=0x55ee1596fb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-03T21:21:35.077 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:35.074+0000 7efe47fff700 1 --2- 172.21.15.67:0/4075575278 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55ee15978c10 0x55ee1596fb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:21:35.077 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:35.074+0000 7efe47fff700 1 -- 172.21.15.67:0/4075575278 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55ee15979530 msgr2=0x55ee1597c370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-03T21:21:35.077 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:35.074+0000 7efe47fff700 1 --2- 172.21.15.67:0/4075575278 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55ee15979530 0x55ee1597c370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:21:35.077 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:35.074+0000 7efe47fff700 1 -- 172.21.15.67:0/4075575278 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ee15994730 con 0x55ee1597d230 2024-05-03T21:21:35.077 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:35.074+0000 7efe46ffd700 1 --2- 172.21.15.67:0/4075575278 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55ee15979530 0x55ee1597c370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-03T21:21:35.077 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:35.074+0000 7efe47fff700 1 --2- 172.21.15.67:0/4075575278 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ee1597d230 0x55ee15973070 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7efe400aaab0 tx=0x7efe400cd050 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bdc40c74ad0e48aa server_cookie=0 in_seq=0 out_seq=0 2024-05-03T21:21:35.077 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:35.074+0000 7efe2d7fa700 1 -- 172.21.15.67:0/4075575278 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe400aa630 con 0x55ee1597d230 2024-05-03T21:21:35.077 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:35.074+0000 7efe2d7fa700 1 -- 172.21.15.67:0/4075575278 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(28 keys) ==== 1170+0+0 (secure 0 0 0) 0x7efe400abe50 con 0x55ee1597d230 2024-05-03T21:21:35.078 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:35.074+0000 7efe2d7fa700 1 -- 172.21.15.67:0/4075575278 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe400aeed0 con 0x55ee1597d230 2024-05-03T21:21:35.081 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:35.074+0000 7efe5835bb80 1 -- 172.21.15.67:0/4075575278 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ee1597d230 msgr2=0x55ee15973070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:21:35.081 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:35.074+0000 7efe5835bb80 1 --2- 172.21.15.67:0/4075575278 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ee1597d230 0x55ee15973070 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7efe400aaab0 tx=0x7efe400cd050 comp rx=0 tx=0).stop 2024-05-03T21:21:35.081 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:35.074+0000 7efe5835bb80 1 -- 172.21.15.67:0/4075575278 shutdown_connections 2024-05-03T21:21:35.082 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:35.074+0000 7efe5835bb80 1 --2- 172.21.15.67:0/4075575278 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ee1597d230 0x55ee15973070 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:21:35.082 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:35.074+0000 7efe5835bb80 1 --2- 172.21.15.67:0/4075575278 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55ee15978c10 0x55ee1596fb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:21:35.082 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:35.074+0000 7efe5835bb80 1 --2- 172.21.15.67:0/4075575278 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55ee15979530 0x55ee1597c370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:21:35.082 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:35.074+0000 7efe5835bb80 1 -- 172.21.15.67:0/4075575278 >> 172.21.15.67:0/4075575278 conn(0x55ee158a50b0 msgr2=0x55ee1597bc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T21:21:35.082 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:35.074+0000 7efe5835bb80 1 -- 172.21.15.67:0/4075575278 shutdown_connections 2024-05-03T21:21:35.082 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:35.074+0000 7efe5835bb80 1 -- 172.21.15.67:0/4075575278 wait complete. 2024-05-03T21:21:35.082 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:35.075+0000 7efe5835bb80 1 -- start start 2024-05-03T21:21:35.082 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:35.075+0000 7efe5835bb80 1 Processor -- start 2024-05-03T21:21:35.082 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:35.075+0000 7efe5835bb80 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ee15978c10 0x55ee1599ce30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:21:35.082 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:35.075+0000 7efe5835bb80 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55ee15979530 0x55ee15997c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:21:35.082 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:35.075+0000 7efe5835bb80 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55ee1597d230 0x55ee15998140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:21:35.082 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:35.075+0000 7efe5835bb80 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x55ee15993650 con 0x55ee15978c10 2024-05-03T21:21:35.082 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:35.075+0000 7efe5835bb80 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x55ee15992f80 con 0x55ee15979530 2024-05-03T21:21:35.083 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:35.075+0000 7efe5835bb80 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x55ee15993d20 con 0x55ee1597d230 2024-05-03T21:21:35.083 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:35.075+0000 7efe5835bb80 10 client.0 _ll_register_callbacks cb 0x55ee158638d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-03T21:21:35.083 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:35.075+0000 7efe477fe700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ee15978c10 0x55ee1599ce30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:21:35.083 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:35.075+0000 7efe477fe700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ee15978c10 0x55ee1599ce30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:48768/0 (socket says 172.21.15.67:48768) 2024-05-03T21:21:35.083 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:35.075+0000 7efe477fe700 1 -- 172.21.15.67:0/3809775291 learned_addr learned my addr 172.21.15.67:0/3809775291 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-03T21:21:35.083 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:35.075+0000 7efe46ffd700 1 --2- 172.21.15.67:0/3809775291 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55ee15979530 0x55ee15997c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:21:35.083 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:35.075+0000 7efe47fff700 1 --2- 172.21.15.67:0/3809775291 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55ee1597d230 0x55ee15998140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:21:35.083 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:35.076+0000 7efe477fe700 1 -- 172.21.15.67:0/3809775291 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55ee1597d230 msgr2=0x55ee15998140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-03T21:21:35.083 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:35.076+0000 7efe477fe700 1 --2- 172.21.15.67:0/3809775291 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55ee1597d230 0x55ee15998140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:21:35.083 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:35.076+0000 7efe477fe700 1 -- 172.21.15.67:0/3809775291 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55ee15979530 msgr2=0x55ee15997c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-03T21:21:35.083 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:35.076+0000 7efe477fe700 1 --2- 172.21.15.67:0/3809775291 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55ee15979530 0x55ee15997c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:21:35.083 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:35.076+0000 7efe477fe700 1 -- 172.21.15.67:0/3809775291 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ee15998680 con 0x55ee15978c10 2024-05-03T21:21:35.083 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:35.076+0000 7efe477fe700 1 --2- 172.21.15.67:0/3809775291 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ee15978c10 0x55ee1599ce30 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7efe3c00b5d0 tx=0x7efe3c004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7672404833ed7033 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T21:21:35.083 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:35.076+0000 7efe26ffd700 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-05-03T21:21:35.084 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:35.076+0000 7efe26ffd700 1 -- 172.21.15.67:0/3809775291 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe3c02d020 con 0x55ee15978c10 2024-05-03T21:21:35.084 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:35.076+0000 7efe26ffd700 1 -- 172.21.15.67:0/3809775291 <== mon.0 v2:172.21.15.67:3300/0 2 ==== osd_map(84..84 src has 1..84) ==== 8211+0+0 (secure 0 0 0) 0x7efe3c003880 con 0x55ee15978c10 2024-05-03T21:21:35.084 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:35.076+0000 7efe5835bb80 1 -- 172.21.15.67:0/3809775291 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7efe400abe50 con 0x55ee15978c10 2024-05-03T21:21:35.084 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:35.076+0000 7efe26ffd700 1 -- 172.21.15.67:0/3809775291 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7efe3c003a80 con 0x55ee15978c10 2024-05-03T21:21:35.084 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:35.076+0000 7efe5835bb80 10 client.15102 fetch_fsmap learned FSMap version 25 2024-05-03T21:21:35.084 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:35.076+0000 7efe5835bb80 1 -- 172.21.15.67:0/3809775291 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55ee159fe3f0 con 0x55ee15978c10 2024-05-03T21:21:35.084 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:35.076+0000 7efe26ffd700 1 -- 172.21.15.67:0/3809775291 <== mon.0 v2:172.21.15.67:3300/0 4 ==== config(28 keys) ==== 1170+0+0 (secure 0 0 0) 0x7efe400abe50 con 0x55ee15978c10 2024-05-03T21:21:35.084 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:35.077+0000 7efe26ffd700 1 -- 172.21.15.67:0/3809775291 <== mon.0 v2:172.21.15.67:3300/0 5 ==== fsmap(e 25) ==== 8338+0+0 (secure 0 0 0) 0x7efe3c011440 con 0x55ee15978c10 2024-05-03T21:21:35.084 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:35.077+0000 7efe5835bb80 10 client.15102 fetch_fsmap finished waiting for FSMap version 25 2024-05-03T21:21:35.084 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:35.077+0000 7efe5835bb80 10 client.15102 resolve_mds: resolved name 'h' to daemon mds.h 2024-05-03T21:21:35.084 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:35.077+0000 7efe5835bb80 20 client.15102 populate_metadata read hostname 'smithi067' 2024-05-03T21:21:35.084 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:35.077+0000 7efe5835bb80 1 --2- 172.21.15.67:0/3809775291 >> [v2:172.21.15.82:6836/1275987852,v1:172.21.15.82:6837/1275987852] conn(0x55ee15a03050 0x55ee15a01d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:21:35.084 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:35.077+0000 7efe5835bb80 4 client.15102 mds_command: new command op to 24449 tid=0 multi_id=0 [{"prefix": "status"}] 2024-05-03T21:21:35.084 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:35.077+0000 7efe5835bb80 1 -- 172.21.15.67:0/3809775291 --> [v2:172.21.15.82:6836/1275987852,v1:172.21.15.82:6837/1275987852] -- command(tid 0: {"prefix": "status"}) -- 0x55ee158d3290 con 0x55ee15a03050 2024-05-03T21:21:35.084 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:35.077+0000 7efe46ffd700 1 --2- 172.21.15.67:0/3809775291 >> [v2:172.21.15.82:6836/1275987852,v1:172.21.15.82:6837/1275987852] conn(0x55ee15a03050 0x55ee15a01d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:21:35.085 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:35.077+0000 7efe46ffd700 1 --2- 172.21.15.67:0/3809775291 >> [v2:172.21.15.82:6836/1275987852,v1:172.21.15.82:6837/1275987852] conn(0x55ee15a03050 0x55ee15a01d60 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=66c56ebebcbf303a server_cookie=37277f06276bc741 in_seq=0 out_seq=0 2024-05-03T21:21:35.085 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:35.077+0000 7efe26ffd700 10 client.15102 ms_handle_connect on v2:172.21.15.82:6836/1275987852 2024-05-03T21:21:35.085 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:35.078+0000 7efe26ffd700 1 -- 172.21.15.67:0/3809775291 <== mds.1 v2:172.21.15.82:6836/1275987852 1 ==== command_reply(tid 0: 0 ) ==== 8+0+327 (crc 0 0 0) 0x55ee158d3290 con 0x55ee15a03050 2024-05-03T21:21:35.085 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:35.078+0000 7efe26ffd700 10 client.15102 handle_command_reply: tid=0 2024-05-03T21:21:35.085 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:35.078+0000 7efe5835bb80 1 client.15102 shutdown 2024-05-03T21:21:35.085 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:35.078+0000 7efe5835bb80 1 -- 172.21.15.67:0/3809775291 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ee15978c10 msgr2=0x55ee1599ce30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:21:35.085 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:35.078+0000 7efe5835bb80 1 --2- 172.21.15.67:0/3809775291 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ee15978c10 0x55ee1599ce30 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7efe3c00b5d0 tx=0x7efe3c004750 comp rx=0 tx=0).stop 2024-05-03T21:21:35.085 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:35.078+0000 7efe5835bb80 1 -- 172.21.15.67:0/3809775291 shutdown_connections 2024-05-03T21:21:35.085 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:35.078+0000 7efe5835bb80 1 --2- 172.21.15.67:0/3809775291 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ee15978c10 0x55ee1599ce30 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:21:35.085 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:35.078+0000 7efe5835bb80 1 --2- 172.21.15.67:0/3809775291 >> [v2:172.21.15.82:6836/1275987852,v1:172.21.15.82:6837/1275987852] conn(0x55ee15a03050 0x55ee15a01d60 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:21:35.085 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:35.078+0000 7efe5835bb80 1 --2- 172.21.15.67:0/3809775291 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55ee1597d230 0x55ee15998140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:21:35.085 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:35.078+0000 7efe5835bb80 1 --2- 172.21.15.67:0/3809775291 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55ee15979530 0x55ee15997c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:21:35.085 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:35.078+0000 7efe5835bb80 1 -- 172.21.15.67:0/3809775291 >> 172.21.15.67:0/3809775291 conn(0x55ee158a50b0 msgr2=0x55ee15983030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T21:21:35.085 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:35.078+0000 7efe26ffd700 0 client.15102 ms_handle_reset on v2:172.21.15.82:6836/1275987852 2024-05-03T21:21:35.085 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:35.078+0000 7efe5835bb80 1 -- 172.21.15.67:0/3809775291 shutdown_connections 2024-05-03T21:21:35.085 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:35.078+0000 7efe5835bb80 1 -- 172.21.15.67:0/3809775291 wait complete. 2024-05-03T21:21:35.086 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:35.078+0000 7efe5835bb80 20 client.15102 trim_cache size 0 max 16384 2024-05-03T21:21:35.086 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-05-03T21:21:35.086 INFO:teuthology.orchestra.run.smithi067.stdout: "cluster_fsid": "713bc57a-0991-11ef-bc97-c7b262605968", 2024-05-03T21:21:35.086 INFO:teuthology.orchestra.run.smithi067.stdout: "whoami": 1, 2024-05-03T21:21:35.086 INFO:teuthology.orchestra.run.smithi067.stdout: "id": 24449, 2024-05-03T21:21:35.086 INFO:teuthology.orchestra.run.smithi067.stdout: "want_state": "up:active", 2024-05-03T21:21:35.086 INFO:teuthology.orchestra.run.smithi067.stdout: "state": "up:active", 2024-05-03T21:21:35.086 INFO:teuthology.orchestra.run.smithi067.stdout: "fs_name": "cephfs", 2024-05-03T21:21:35.086 INFO:teuthology.orchestra.run.smithi067.stdout: "rank_uptime": 158.22369494200001, 2024-05-03T21:21:35.086 INFO:teuthology.orchestra.run.smithi067.stdout: "mdsmap_epoch": 25, 2024-05-03T21:21:35.087 INFO:teuthology.orchestra.run.smithi067.stdout: "osdmap_epoch": 84, 2024-05-03T21:21:35.087 INFO:teuthology.orchestra.run.smithi067.stdout: "osdmap_epoch_barrier": 82, 2024-05-03T21:21:35.087 INFO:teuthology.orchestra.run.smithi067.stdout: "uptime": 197.640821883 2024-05-03T21:21:35.087 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-05-03T21:21:35.087 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:35.078+0000 7efe4dac8700 1 -- 172.21.15.67:0/3572268826 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7efe300b2990 msgr2=0x7efe300b4e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:21:35.087 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:35.078+0000 7efe4dac8700 1 --2- 172.21.15.67:0/3572268826 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7efe300b2990 0x7efe300b4e50 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7efe3c005950 tx=0x7efe3c0058e0 comp rx=0 tx=0).stop 2024-05-03T21:21:35.087 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:35.078+0000 7efe4dac8700 1 -- 172.21.15.67:0/3572268826 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efe48108850 msgr2=0x7efe481a0690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:21:35.087 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:35.078+0000 7efe4dac8700 1 --2- 172.21.15.67:0/3572268826 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efe48108850 0x7efe481a0690 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7efe4000d010 tx=0x7efe40007ad0 comp rx=0 tx=0).stop 2024-05-03T21:21:35.088 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:35.079+0000 7efe4dac8700 1 -- 172.21.15.67:0/3572268826 shutdown_connections 2024-05-03T21:21:35.088 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:35.079+0000 7efe4dac8700 1 --2- 172.21.15.67:0/3572268826 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7efe300b2990 0x7efe300b4e50 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:21:35.088 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:35.079+0000 7efe4dac8700 1 --2- 172.21.15.67:0/3572268826 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efe48108850 0x7efe481a0690 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:21:35.088 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:35.079+0000 7efe4dac8700 1 --2- 172.21.15.67:0/3572268826 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7efe48103750 0x7efe4819bdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:21:35.088 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:35.079+0000 7efe4dac8700 1 --2- 172.21.15.67:0/3572268826 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7efe48104100 0x7efe4819c300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:21:35.088 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:35.079+0000 7efe4dac8700 1 -- 172.21.15.67:0/3572268826 >> 172.21.15.67:0/3572268826 conn(0x7efe4807a600 msgr2=0x7efe48102410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T21:21:35.088 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:35.079+0000 7efe4dac8700 1 -- 172.21.15.67:0/3572268826 shutdown_connections 2024-05-03T21:21:35.088 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:35.079+0000 7efe4dac8700 1 -- 172.21.15.67:0/3572268826 wait complete. 2024-05-03T21:21:35.358 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:21:35 smithi067 ceph-mon[29730]: pgmap v374: 97 pgs: 97 active+clean; 632 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:21:35.574 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:21:35 smithi138 ceph-mon[33110]: pgmap v374: 97 pgs: 97 active+clean; 632 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:21:35.745 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:21:35 smithi082 ceph-mon[32933]: pgmap v374: 97 pgs: 97 active+clean; 632 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:21:35.802 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9a047fe7599a549042d06a3fb5b6813a203eabea shell --fsid 713bc57a-0991-11ef-bc97-c7b262605968 -- ceph tell mds.b status 2024-05-03T21:21:36.057 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/713bc57a-0991-11ef-bc97-c7b262605968/mon.a/config 2024-05-03T21:21:37.448 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:21:37 smithi067 ceph-mon[29730]: pgmap v375: 97 pgs: 97 active+clean; 632 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:21:37.574 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:21:37 smithi138 ceph-mon[33110]: pgmap v375: 97 pgs: 97 active+clean; 632 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:21:37.687 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:37.686+0000 7fb023326700 1 -- 172.21.15.67:0/2154988884 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb01c109590 msgr2=0x7fb01c109970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:21:37.688 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:37.686+0000 7fb023326700 1 --2- 172.21.15.67:0/2154988884 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb01c109590 0x7fb01c109970 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7fb0180099c0 tx=0x7fb018009cd0 comp rx=0 tx=0).stop 2024-05-03T21:21:37.688 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:37.686+0000 7fb023326700 1 -- 172.21.15.67:0/2154988884 shutdown_connections 2024-05-03T21:21:37.688 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:37.686+0000 7fb023326700 1 --2- 172.21.15.67:0/2154988884 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb01c109590 0x7fb01c109970 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:21:37.688 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:37.686+0000 7fb023326700 1 --2- 172.21.15.67:0/2154988884 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fb01c06c630 0x7fb01c110360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:21:37.688 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:37.686+0000 7fb023326700 1 --2- 172.21.15.67:0/2154988884 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fb01c06bc70 0x7fb01c06c0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:21:37.688 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:37.686+0000 7fb023326700 1 -- 172.21.15.67:0/2154988884 >> 172.21.15.67:0/2154988884 conn(0x7fb01c079440 msgr2=0x7fb01c079850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T21:21:37.688 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:37.686+0000 7fb023326700 1 -- 172.21.15.67:0/2154988884 shutdown_connections 2024-05-03T21:21:37.688 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:37.686+0000 7fb023326700 1 -- 172.21.15.67:0/2154988884 wait complete. 2024-05-03T21:21:37.690 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:37.687+0000 7fb023326700 1 Processor -- start 2024-05-03T21:21:37.690 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:37.687+0000 7fb023326700 1 -- start start 2024-05-03T21:21:37.690 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:37.687+0000 7fb023326700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fb01c06bc70 0x7fb01c19fa60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:21:37.690 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:37.687+0000 7fb023326700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb01c06c630 0x7fb01c19ffa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:21:37.690 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:37.687+0000 7fb023326700 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fb01c109590 0x7fb01c199b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:21:37.690 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:37.687+0000 7fb023326700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fb01c113990 con 0x7fb01c06c630 2024-05-03T21:21:37.690 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:37.687+0000 7fb023326700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7fb01c113810 con 0x7fb01c06bc70 2024-05-03T21:21:37.691 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:37.687+0000 7fb023326700 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7fb01c113b10 con 0x7fb01c109590 2024-05-03T21:21:37.691 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:37.687+0000 7fb0208c1700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb01c06c630 0x7fb01c19ffa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:21:37.691 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:37.687+0000 7fb0208c1700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb01c06c630 0x7fb01c19ffa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:40738/0 (socket says 172.21.15.67:40738) 2024-05-03T21:21:37.691 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:37.687+0000 7fb0208c1700 1 -- 172.21.15.67:0/1524448263 learned_addr learned my addr 172.21.15.67:0/1524448263 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-03T21:21:37.691 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:37.687+0000 7fb0210c2700 1 --2- 172.21.15.67:0/1524448263 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fb01c06bc70 0x7fb01c19fa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:21:37.691 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:37.687+0000 7fb0218c3700 1 --2- 172.21.15.67:0/1524448263 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fb01c109590 0x7fb01c199b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:21:37.691 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:37.687+0000 7fb0208c1700 1 -- 172.21.15.67:0/1524448263 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fb01c109590 msgr2=0x7fb01c199b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:21:37.691 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:37.687+0000 7fb0208c1700 1 --2- 172.21.15.67:0/1524448263 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fb01c109590 0x7fb01c199b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:21:37.691 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:37.687+0000 7fb0208c1700 1 -- 172.21.15.67:0/1524448263 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fb01c06bc70 msgr2=0x7fb01c19fa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:21:37.691 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:37.687+0000 7fb0208c1700 1 --2- 172.21.15.67:0/1524448263 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fb01c06bc70 0x7fb01c19fa60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:21:37.691 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:37.687+0000 7fb0208c1700 1 -- 172.21.15.67:0/1524448263 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb01c19a410 con 0x7fb01c06c630 2024-05-03T21:21:37.691 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:37.687+0000 7fb0218c3700 1 --2- 172.21.15.67:0/1524448263 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fb01c109590 0x7fb01c199b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-03T21:21:37.691 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:37.688+0000 7fb0208c1700 1 --2- 172.21.15.67:0/1524448263 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb01c06c630 0x7fb01c19ffa0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7fb00c00d950 tx=0x7fb00c00dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T21:21:37.691 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:37.688+0000 7fb0127fc700 1 -- 172.21.15.67:0/1524448263 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb00c016070 con 0x7fb01c06c630 2024-05-03T21:21:37.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:37.688+0000 7fb0127fc700 1 -- 172.21.15.67:0/1524448263 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(28 keys) ==== 1170+0+0 (secure 0 0 0) 0x7fb00c009e40 con 0x7fb01c06c630 2024-05-03T21:21:37.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:37.688+0000 7fb023326700 1 -- 172.21.15.67:0/1524448263 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb01c19a700 con 0x7fb01c06c630 2024-05-03T21:21:37.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:37.688+0000 7fb0127fc700 1 -- 172.21.15.67:0/1524448263 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb00c00f620 con 0x7fb01c06c630 2024-05-03T21:21:37.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:37.688+0000 7fb023326700 1 -- 172.21.15.67:0/1524448263 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb01c1a69e0 con 0x7fb01c06c630 2024-05-03T21:21:37.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:37.690+0000 7fb0127fc700 1 -- 172.21.15.67:0/1524448263 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 17) ==== 149703+0+0 (secure 0 0 0) 0x7fb00c019070 con 0x7fb01c06c630 2024-05-03T21:21:37.693 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:37.691+0000 7fb0127fc700 1 --2- 172.21.15.67:0/1524448263 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7fb0080b2990 0x7fb0080b4e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:21:37.693 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:37.691+0000 7fb0210c2700 1 --2- 172.21.15.67:0/1524448263 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7fb0080b2990 0x7fb0080b4e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:21:37.693 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:37.692+0000 7fb0127fc700 1 -- 172.21.15.67:0/1524448263 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 8211+0+0 (secure 0 0 0) 0x7fb00c0e22e0 con 0x7fb01c06c630 2024-05-03T21:21:37.698 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:37.692+0000 7fb0210c2700 1 --2- 172.21.15.67:0/1524448263 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7fb0080b2990 0x7fb0080b4e50 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fb018000c00 tx=0x7fb018005fb0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T21:21:37.699 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:37.692+0000 7fb02dbb9b80 1 Processor -- start 2024-05-03T21:21:37.699 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:37.692+0000 7fb02dbb9b80 1 -- start start 2024-05-03T21:21:37.699 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:37.692+0000 7fb02dbb9b80 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5597fb49ac10 0x5597fb49aff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:21:37.699 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:37.692+0000 7fb02dbb9b80 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5597fb49b530 0x5597fb49ecf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:21:37.699 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:37.692+0000 7fb02dbb9b80 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5597fb49f230 0x5597fb49f690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:21:37.699 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:37.692+0000 7fb02dbb9b80 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x5597fb3ed810 con 0x5597fb49b530 2024-05-03T21:21:37.699 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:37.692+0000 7fb02dbb9b80 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x5597fb38c770 con 0x5597fb49f230 2024-05-03T21:21:37.699 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:37.692+0000 7fb02dbb9b80 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x5597fb3e97d0 con 0x5597fb49ac10 2024-05-03T21:21:37.699 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:37.692+0000 7fb0208c1700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5597fb49b530 0x5597fb49ecf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:21:37.699 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:37.692+0000 7fb0208c1700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5597fb49b530 0x5597fb49ecf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:40754/0 (socket says 172.21.15.67:40754) 2024-05-03T21:21:37.699 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:37.692+0000 7fb0208c1700 1 -- 172.21.15.67:0/3083803868 learned_addr learned my addr 172.21.15.67:0/3083803868 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-03T21:21:37.700 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:37.693+0000 7fb0218c3700 1 --2- 172.21.15.67:0/3083803868 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5597fb49f230 0x5597fb49f690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:21:37.700 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:37.693+0000 7fb0210c2700 1 --2- 172.21.15.67:0/3083803868 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5597fb49ac10 0x5597fb49aff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:21:37.700 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:37.693+0000 7fb0208c1700 1 -- 172.21.15.67:0/3083803868 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5597fb49ac10 msgr2=0x5597fb49aff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-03T21:21:37.700 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:37.693+0000 7fb0208c1700 1 --2- 172.21.15.67:0/3083803868 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5597fb49ac10 0x5597fb49aff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:21:37.700 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:37.693+0000 7fb0208c1700 1 -- 172.21.15.67:0/3083803868 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5597fb49f230 msgr2=0x5597fb49f690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-03T21:21:37.700 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:37.693+0000 7fb0208c1700 1 --2- 172.21.15.67:0/3083803868 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5597fb49f230 0x5597fb49f690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:21:37.700 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:37.693+0000 7fb0208c1700 1 -- 172.21.15.67:0/3083803868 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5597fb33a880 con 0x5597fb49b530 2024-05-03T21:21:37.700 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:37.693+0000 7fb0210c2700 1 --2- 172.21.15.67:0/3083803868 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5597fb49ac10 0x5597fb49aff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-03T21:21:37.700 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:37.693+0000 7fb0208c1700 1 --2- 172.21.15.67:0/3083803868 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5597fb49b530 0x5597fb49ecf0 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7fb00c010b80 tx=0x7fb00c0a91d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c32e1aa482f939bf server_cookie=0 in_seq=0 out_seq=0 2024-05-03T21:21:37.700 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:37.693+0000 7fb0077fe700 1 -- 172.21.15.67:0/3083803868 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb00c020070 con 0x5597fb49b530 2024-05-03T21:21:37.700 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:37.693+0000 7fb0077fe700 1 -- 172.21.15.67:0/3083803868 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(28 keys) ==== 1170+0+0 (secure 0 0 0) 0x7fb00c0a9750 con 0x5597fb49b530 2024-05-03T21:21:37.700 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:37.693+0000 7fb0077fe700 1 -- 172.21.15.67:0/3083803868 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb00c091310 con 0x5597fb49b530 2024-05-03T21:21:37.700 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:37.694+0000 7fb02dbb9b80 1 -- 172.21.15.67:0/3083803868 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5597fb49b530 msgr2=0x5597fb49ecf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:21:37.700 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:37.694+0000 7fb02dbb9b80 1 --2- 172.21.15.67:0/3083803868 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5597fb49b530 0x5597fb49ecf0 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7fb00c010b80 tx=0x7fb00c0a91d0 comp rx=0 tx=0).stop 2024-05-03T21:21:37.701 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:37.694+0000 7fb02dbb9b80 1 -- 172.21.15.67:0/3083803868 shutdown_connections 2024-05-03T21:21:37.701 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:37.694+0000 7fb02dbb9b80 1 --2- 172.21.15.67:0/3083803868 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5597fb49f230 0x5597fb49f690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:21:37.701 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:37.694+0000 7fb02dbb9b80 1 --2- 172.21.15.67:0/3083803868 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5597fb49ac10 0x5597fb49aff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:21:37.701 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:37.694+0000 7fb02dbb9b80 1 --2- 172.21.15.67:0/3083803868 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5597fb49b530 0x5597fb49ecf0 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:21:37.701 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:37.694+0000 7fb02dbb9b80 1 -- 172.21.15.67:0/3083803868 >> 172.21.15.67:0/3083803868 conn(0x5597fb3c70b0 msgr2=0x5597fb495280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T21:21:37.701 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:37.694+0000 7fb02dbb9b80 1 -- 172.21.15.67:0/3083803868 shutdown_connections 2024-05-03T21:21:37.701 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:37.694+0000 7fb02dbb9b80 1 -- 172.21.15.67:0/3083803868 wait complete. 2024-05-03T21:21:37.701 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:37.694+0000 7fb02dbb9b80 1 -- start start 2024-05-03T21:21:37.701 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:37.694+0000 7fb02dbb9b80 1 Processor -- start 2024-05-03T21:21:37.701 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:37.695+0000 7fb02dbb9b80 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5597fb49ac10 0x5597fb4a66c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:21:37.701 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:37.695+0000 7fb02dbb9b80 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5597fb49b530 0x5597fb4a3a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:21:37.701 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:37.695+0000 7fb02dbb9b80 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5597fb49f230 0x5597fb4a3fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:21:37.701 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:37.695+0000 7fb02dbb9b80 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x5597fb4a7490 con 0x5597fb49b530 2024-05-03T21:21:37.702 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:37.695+0000 7fb02dbb9b80 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x5597fb353be0 con 0x5597fb49ac10 2024-05-03T21:21:37.702 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:37.695+0000 7fb02dbb9b80 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x5597fb4a7610 con 0x5597fb49f230 2024-05-03T21:21:37.702 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:37.695+0000 7fb02dbb9b80 10 client.0 _ll_register_callbacks cb 0x5597fb3858d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-03T21:21:37.702 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:37.695+0000 7fb0208c1700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5597fb49b530 0x5597fb4a3a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:21:37.702 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:37.695+0000 7fb0208c1700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5597fb49b530 0x5597fb4a3a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:40762/0 (socket says 172.21.15.67:40762) 2024-05-03T21:21:37.702 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:37.695+0000 7fb0208c1700 1 -- 172.21.15.67:0/4245988489 learned_addr learned my addr 172.21.15.67:0/4245988489 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-03T21:21:37.702 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:37.695+0000 7fb0210c2700 1 --2- 172.21.15.67:0/4245988489 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5597fb49ac10 0x5597fb4a66c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:21:37.702 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:37.695+0000 7fb0208c1700 1 -- 172.21.15.67:0/4245988489 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5597fb49f230 msgr2=0x5597fb4a3fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-03T21:21:37.702 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:37.695+0000 7fb0218c3700 1 --2- 172.21.15.67:0/4245988489 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5597fb49f230 0x5597fb4a3fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:21:37.702 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:37.695+0000 7fb0208c1700 1 --2- 172.21.15.67:0/4245988489 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5597fb49f230 0x5597fb4a3fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:21:37.702 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:37.695+0000 7fb0208c1700 1 -- 172.21.15.67:0/4245988489 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5597fb49ac10 msgr2=0x5597fb4a66c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-03T21:21:37.702 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:37.695+0000 7fb0208c1700 1 --2- 172.21.15.67:0/4245988489 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5597fb49ac10 0x5597fb4a66c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:21:37.702 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:37.695+0000 7fb0208c1700 1 -- 172.21.15.67:0/4245988489 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5597fb389640 con 0x5597fb49b530 2024-05-03T21:21:37.703 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:37.695+0000 7fb0210c2700 1 --2- 172.21.15.67:0/4245988489 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5597fb49ac10 0x5597fb4a66c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-03T21:21:37.703 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:37.695+0000 7fb0208c1700 1 --2- 172.21.15.67:0/4245988489 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5597fb49b530 0x5597fb4a3a60 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7fb00c0a9450 tx=0x7fb00c005bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=936eb50144a279b5 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T21:21:37.703 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:37.695+0000 7fb004ff9700 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-05-03T21:21:37.703 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:37.695+0000 7fb004ff9700 1 -- 172.21.15.67:0/4245988489 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb00c020070 con 0x5597fb49b530 2024-05-03T21:21:37.703 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:37.696+0000 7fb004ff9700 1 -- 172.21.15.67:0/4245988489 <== mon.0 v2:172.21.15.67:3300/0 2 ==== osd_map(84..84 src has 1..84) ==== 8211+0+0 (secure 0 0 0) 0x7fb00c091110 con 0x5597fb49b530 2024-05-03T21:21:37.703 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:37.696+0000 7fb02dbb9b80 1 -- 172.21.15.67:0/4245988489 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb00c0a9750 con 0x5597fb49b530 2024-05-03T21:21:37.703 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:37.696+0000 7fb004ff9700 1 -- 172.21.15.67:0/4245988489 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7fb00c091310 con 0x5597fb49b530 2024-05-03T21:21:37.703 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:37.696+0000 7fb02dbb9b80 10 client.15114 fetch_fsmap learned FSMap version 25 2024-05-03T21:21:37.703 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:37.696+0000 7fb02dbb9b80 1 -- 172.21.15.67:0/4245988489 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x5597fb4a7490 con 0x5597fb49b530 2024-05-03T21:21:37.703 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:37.696+0000 7fb004ff9700 1 -- 172.21.15.67:0/4245988489 <== mon.0 v2:172.21.15.67:3300/0 4 ==== config(28 keys) ==== 1170+0+0 (secure 0 0 0) 0x7fb00c0a9750 con 0x5597fb49b530 2024-05-03T21:21:37.703 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:37.696+0000 7fb004ff9700 1 -- 172.21.15.67:0/4245988489 <== mon.0 v2:172.21.15.67:3300/0 5 ==== fsmap(e 25) ==== 8338+0+0 (secure 0 0 0) 0x7fb00c0beba0 con 0x5597fb49b530 2024-05-03T21:21:37.703 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:37.696+0000 7fb02dbb9b80 10 client.15114 fetch_fsmap finished waiting for FSMap version 25 2024-05-03T21:21:37.703 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:37.696+0000 7fb02dbb9b80 10 client.15114 resolve_mds: resolved name 'b' to daemon mds.b 2024-05-03T21:21:37.703 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:37.696+0000 7fb02dbb9b80 20 client.15114 populate_metadata read hostname 'smithi067' 2024-05-03T21:21:37.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:37.696+0000 7fb02dbb9b80 1 --2- 172.21.15.67:0/4245988489 >> [v2:172.21.15.82:6838/3644499578,v1:172.21.15.82:6839/3644499578] conn(0x5597fb4b3d20 0x5597fb4b5190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:21:37.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:37.696+0000 7fb02dbb9b80 4 client.15114 mds_command: new command op to 24473 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-03T21:21:37.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:37.696+0000 7fb02dbb9b80 1 -- 172.21.15.67:0/4245988489 --> [v2:172.21.15.82:6838/3644499578,v1:172.21.15.82:6839/3644499578] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5597fb3fb1f0 con 0x5597fb4b3d20 2024-05-03T21:21:37.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:37.696+0000 7fb0210c2700 1 --2- 172.21.15.67:0/4245988489 >> [v2:172.21.15.82:6838/3644499578,v1:172.21.15.82:6839/3644499578] conn(0x5597fb4b3d20 0x5597fb4b5190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:21:37.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:37.697+0000 7fb004ff9700 10 client.15114 ms_handle_connect on v2:172.21.15.82:6838/3644499578 2024-05-03T21:21:37.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:37.697+0000 7fb0210c2700 1 --2- 172.21.15.67:0/4245988489 >> [v2:172.21.15.82:6838/3644499578,v1:172.21.15.82:6839/3644499578] conn(0x5597fb4b3d20 0x5597fb4b5190 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1ff9d376156da99 server_cookie=690336f3800e30d9 in_seq=0 out_seq=0 2024-05-03T21:21:37.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:37.698+0000 7fb004ff9700 1 -- 172.21.15.67:0/4245988489 <== mds.0 v2:172.21.15.82:6838/3644499578 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x5597fb3fb1f0 con 0x5597fb4b3d20 2024-05-03T21:21:37.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:37.698+0000 7fb004ff9700 10 client.15114 handle_command_reply: tid=0 2024-05-03T21:21:37.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:37.698+0000 7fb02dbb9b80 1 client.15114 shutdown 2024-05-03T21:21:37.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:37.698+0000 7fb02dbb9b80 1 -- 172.21.15.67:0/4245988489 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5597fb49b530 msgr2=0x5597fb4a3a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:21:37.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:37.698+0000 7fb02dbb9b80 1 --2- 172.21.15.67:0/4245988489 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5597fb49b530 0x5597fb4a3a60 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7fb00c0a9450 tx=0x7fb00c005bb0 comp rx=0 tx=0).stop 2024-05-03T21:21:37.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:37.698+0000 7fb02dbb9b80 1 -- 172.21.15.67:0/4245988489 shutdown_connections 2024-05-03T21:21:37.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:37.698+0000 7fb02dbb9b80 1 --2- 172.21.15.67:0/4245988489 >> [v2:172.21.15.82:6838/3644499578,v1:172.21.15.82:6839/3644499578] conn(0x5597fb4b3d20 0x5597fb4b5190 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:21:37.706 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:37.698+0000 7fb02dbb9b80 1 --2- 172.21.15.67:0/4245988489 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5597fb49f230 0x5597fb4a3fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:21:37.706 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:37.698+0000 7fb02dbb9b80 1 --2- 172.21.15.67:0/4245988489 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5597fb49ac10 0x5597fb4a66c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:21:37.706 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:37.698+0000 7fb02dbb9b80 1 --2- 172.21.15.67:0/4245988489 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5597fb49b530 0x5597fb4a3a60 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:21:37.706 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:37.698+0000 7fb02dbb9b80 1 -- 172.21.15.67:0/4245988489 >> 172.21.15.67:0/4245988489 conn(0x5597fb3c70b0 msgr2=0x5597fb495240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T21:21:37.706 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:37.698+0000 7fb004ff9700 0 client.15114 ms_handle_reset on v2:172.21.15.82:6838/3644499578 2024-05-03T21:21:37.706 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:37.698+0000 7fb02dbb9b80 1 -- 172.21.15.67:0/4245988489 shutdown_connections 2024-05-03T21:21:37.706 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:37.698+0000 7fb02dbb9b80 1 -- 172.21.15.67:0/4245988489 wait complete. 2024-05-03T21:21:37.706 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:37.698+0000 7fb02dbb9b80 20 client.15114 trim_cache size 0 max 16384 2024-05-03T21:21:37.726 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:37.719+0000 7fb02dbb9b80 1 Processor -- start 2024-05-03T21:21:37.727 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:37.719+0000 7fb02dbb9b80 1 -- start start 2024-05-03T21:21:37.727 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:37.720+0000 7fb02dbb9b80 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5597fb49ac10 0x5597fb491b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:21:37.727 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:37.720+0000 7fb02dbb9b80 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5597fb49b530 0x5597fb4be9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:21:37.727 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:37.720+0000 7fb02dbb9b80 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5597fb49f230 0x5597fb49e6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:21:37.727 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:37.720+0000 7fb02dbb9b80 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x5597fb4b34d0 con 0x5597fb49b530 2024-05-03T21:21:37.727 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:37.720+0000 7fb02dbb9b80 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x5597fb4b3ba0 con 0x5597fb49ac10 2024-05-03T21:21:37.727 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:37.720+0000 7fb02dbb9b80 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x5597fb4b4270 con 0x5597fb49f230 2024-05-03T21:21:37.727 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:37.720+0000 7fb0208c1700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5597fb49b530 0x5597fb4be9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:21:37.727 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:37.720+0000 7fb0208c1700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5597fb49b530 0x5597fb4be9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:40768/0 (socket says 172.21.15.67:40768) 2024-05-03T21:21:37.727 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:37.720+0000 7fb0208c1700 1 -- 172.21.15.67:0/429461154 learned_addr learned my addr 172.21.15.67:0/429461154 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-03T21:21:37.727 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:37.720+0000 7fb0210c2700 1 --2- 172.21.15.67:0/429461154 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5597fb49ac10 0x5597fb491b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:21:37.727 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:37.720+0000 7fb0208c1700 1 -- 172.21.15.67:0/429461154 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5597fb49f230 msgr2=0x5597fb49e6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-03T21:21:37.728 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:37.720+0000 7fb0208c1700 1 --2- 172.21.15.67:0/429461154 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5597fb49f230 0x5597fb49e6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:21:37.728 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:37.720+0000 7fb0208c1700 1 -- 172.21.15.67:0/429461154 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5597fb49ac10 msgr2=0x5597fb491b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-03T21:21:37.728 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:37.720+0000 7fb0208c1700 1 --2- 172.21.15.67:0/429461154 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5597fb49ac10 0x5597fb491b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:21:37.728 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:37.720+0000 7fb0208c1700 1 -- 172.21.15.67:0/429461154 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5597fb4901f0 con 0x5597fb49b530 2024-05-03T21:21:37.728 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:37.720+0000 7fb0210c2700 1 --2- 172.21.15.67:0/429461154 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5597fb49ac10 0x5597fb491b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-03T21:21:37.728 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:37.720+0000 7fb0208c1700 1 --2- 172.21.15.67:0/429461154 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5597fb49b530 0x5597fb4be9e0 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7fb00c000c00 tx=0x7fb00c0a8dd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1f09caaffb8d085d server_cookie=0 in_seq=0 out_seq=0 2024-05-03T21:21:37.728 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:37.720+0000 7fb006ffd700 1 -- 172.21.15.67:0/429461154 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb00c020070 con 0x5597fb49b530 2024-05-03T21:21:37.728 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:37.720+0000 7fb006ffd700 1 -- 172.21.15.67:0/429461154 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(28 keys) ==== 1170+0+0 (secure 0 0 0) 0x7fb00c0ca050 con 0x5597fb49b530 2024-05-03T21:21:37.728 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:37.720+0000 7fb006ffd700 1 -- 172.21.15.67:0/429461154 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb00c083e60 con 0x5597fb49b530 2024-05-03T21:21:37.728 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:37.721+0000 7fb02dbb9b80 1 -- 172.21.15.67:0/429461154 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5597fb49b530 msgr2=0x5597fb4be9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:21:37.728 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:37.721+0000 7fb02dbb9b80 1 --2- 172.21.15.67:0/429461154 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5597fb49b530 0x5597fb4be9e0 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7fb00c000c00 tx=0x7fb00c0a8dd0 comp rx=0 tx=0).stop 2024-05-03T21:21:37.728 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:37.721+0000 7fb02dbb9b80 1 -- 172.21.15.67:0/429461154 shutdown_connections 2024-05-03T21:21:37.728 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:37.721+0000 7fb02dbb9b80 1 --2- 172.21.15.67:0/429461154 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5597fb49f230 0x5597fb49e6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:21:37.729 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:37.721+0000 7fb02dbb9b80 1 --2- 172.21.15.67:0/429461154 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5597fb49ac10 0x5597fb491b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:21:37.729 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:37.721+0000 7fb02dbb9b80 1 --2- 172.21.15.67:0/429461154 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5597fb49b530 0x5597fb4be9e0 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:21:37.729 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:37.721+0000 7fb02dbb9b80 1 -- 172.21.15.67:0/429461154 >> 172.21.15.67:0/429461154 conn(0x5597fb3c70b0 msgr2=0x5597fb49dfc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T21:21:37.729 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:37.721+0000 7fb02dbb9b80 1 -- 172.21.15.67:0/429461154 shutdown_connections 2024-05-03T21:21:37.729 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:37.721+0000 7fb02dbb9b80 1 -- 172.21.15.67:0/429461154 wait complete. 2024-05-03T21:21:37.729 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:37.722+0000 7fb02dbb9b80 1 -- start start 2024-05-03T21:21:37.729 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:37.722+0000 7fb02dbb9b80 1 Processor -- start 2024-05-03T21:21:37.729 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:37.722+0000 7fb02dbb9b80 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5597fb49ac10 0x5597fb51c850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:21:37.729 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:37.722+0000 7fb02dbb9b80 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5597fb49b530 0x5597fb51cd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:21:37.729 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:37.722+0000 7fb02dbb9b80 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5597fb49f230 0x5597fb517150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:21:37.729 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:37.722+0000 7fb02dbb9b80 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x5597fb4b5010 con 0x5597fb49f230 2024-05-03T21:21:37.729 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:37.722+0000 7fb02dbb9b80 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x5597fb4b4940 con 0x5597fb49ac10 2024-05-03T21:21:37.729 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:37.722+0000 7fb02dbb9b80 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x5597fb4b56e0 con 0x5597fb49b530 2024-05-03T21:21:37.730 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:37.722+0000 7fb02dbb9b80 10 client.0 _ll_register_callbacks cb 0x5597fb3858d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-03T21:21:37.730 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:37.722+0000 7fb0218c3700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5597fb49f230 0x5597fb517150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:21:37.730 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:37.722+0000 7fb0218c3700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5597fb49f230 0x5597fb517150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:40778/0 (socket says 172.21.15.67:40778) 2024-05-03T21:21:37.730 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:37.722+0000 7fb0218c3700 1 -- 172.21.15.67:0/176882995 learned_addr learned my addr 172.21.15.67:0/176882995 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-03T21:21:37.730 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:37.722+0000 7fb0208c1700 1 --2- 172.21.15.67:0/176882995 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5597fb49b530 0x5597fb51cd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:21:37.730 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:37.722+0000 7fb0210c2700 1 --2- 172.21.15.67:0/176882995 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5597fb49ac10 0x5597fb51c850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:21:37.730 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:37.723+0000 7fb0218c3700 1 -- 172.21.15.67:0/176882995 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5597fb49b530 msgr2=0x5597fb51cd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-03T21:21:37.730 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:37.723+0000 7fb0218c3700 1 --2- 172.21.15.67:0/176882995 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5597fb49b530 0x5597fb51cd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:21:37.730 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:37.723+0000 7fb0218c3700 1 -- 172.21.15.67:0/176882995 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5597fb49ac10 msgr2=0x5597fb51c850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-03T21:21:37.730 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:37.723+0000 7fb0218c3700 1 --2- 172.21.15.67:0/176882995 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5597fb49ac10 0x5597fb51c850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:21:37.730 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:37.723+0000 7fb0218c3700 1 -- 172.21.15.67:0/176882995 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5597fb5178a0 con 0x5597fb49f230 2024-05-03T21:21:37.730 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:37.723+0000 7fb0210c2700 1 --2- 172.21.15.67:0/176882995 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5597fb49ac10 0x5597fb51c850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-03T21:21:37.731 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:37.723+0000 7fb0218c3700 1 --2- 172.21.15.67:0/176882995 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5597fb49f230 0x5597fb517150 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7fb0140109c0 tx=0x7fb0140109f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7023ab3eaaf236d6 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T21:21:37.731 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:37.723+0000 7fb004ff9700 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-05-03T21:21:37.731 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:37.723+0000 7fb004ff9700 1 -- 172.21.15.67:0/176882995 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0140116f0 con 0x5597fb49f230 2024-05-03T21:21:37.731 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:37.723+0000 7fb004ff9700 1 -- 172.21.15.67:0/176882995 <== mon.0 v2:172.21.15.67:3300/0 2 ==== osd_map(84..84 src has 1..84) ==== 8211+0+0 (secure 0 0 0) 0x7fb014011890 con 0x5597fb49f230 2024-05-03T21:21:37.731 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:37.723+0000 7fb02dbb9b80 1 -- 172.21.15.67:0/176882995 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb00c0ca050 con 0x5597fb49f230 2024-05-03T21:21:37.731 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:37.723+0000 7fb004ff9700 1 -- 172.21.15.67:0/176882995 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7fb014011a90 con 0x5597fb49f230 2024-05-03T21:21:37.731 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:37.723+0000 7fb02dbb9b80 10 client.15120 fetch_fsmap learned FSMap version 25 2024-05-03T21:21:37.731 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:37.723+0000 7fb02dbb9b80 1 -- 172.21.15.67:0/176882995 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x5597fb524070 con 0x5597fb49f230 2024-05-03T21:21:37.731 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:37.723+0000 7fb004ff9700 1 -- 172.21.15.67:0/176882995 <== mon.0 v2:172.21.15.67:3300/0 4 ==== config(28 keys) ==== 1170+0+0 (secure 0 0 0) 0x7fb00c0ca050 con 0x5597fb49f230 2024-05-03T21:21:37.731 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:37.724+0000 7fb004ff9700 1 -- 172.21.15.67:0/176882995 <== mon.0 v2:172.21.15.67:3300/0 5 ==== fsmap(e 25) ==== 8338+0+0 (secure 0 0 0) 0x7fb0140243f0 con 0x5597fb49f230 2024-05-03T21:21:37.731 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:37.724+0000 7fb02dbb9b80 10 client.15120 fetch_fsmap finished waiting for FSMap version 25 2024-05-03T21:21:37.731 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:37.724+0000 7fb02dbb9b80 10 client.15120 resolve_mds: resolved name 'b' to daemon mds.b 2024-05-03T21:21:37.732 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:37.724+0000 7fb02dbb9b80 20 client.15120 populate_metadata read hostname 'smithi067' 2024-05-03T21:21:37.732 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:37.724+0000 7fb02dbb9b80 1 --2- 172.21.15.67:0/176882995 >> [v2:172.21.15.82:6838/3644499578,v1:172.21.15.82:6839/3644499578] conn(0x5597fb528c70 0x5597fb527980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:21:37.732 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:37.724+0000 7fb02dbb9b80 4 client.15120 mds_command: new command op to 24473 tid=0 multi_id=0 [{"prefix": "status"}] 2024-05-03T21:21:37.732 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:37.724+0000 7fb02dbb9b80 1 -- 172.21.15.67:0/176882995 --> [v2:172.21.15.82:6838/3644499578,v1:172.21.15.82:6839/3644499578] -- command(tid 0: {"prefix": "status"}) -- 0x5597fb3f5290 con 0x5597fb528c70 2024-05-03T21:21:37.732 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:37.724+0000 7fb0210c2700 1 --2- 172.21.15.67:0/176882995 >> [v2:172.21.15.82:6838/3644499578,v1:172.21.15.82:6839/3644499578] conn(0x5597fb528c70 0x5597fb527980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:21:37.732 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:37.724+0000 7fb0210c2700 1 --2- 172.21.15.67:0/176882995 >> [v2:172.21.15.82:6838/3644499578,v1:172.21.15.82:6839/3644499578] conn(0x5597fb528c70 0x5597fb527980 crc :-1 s=READY pgs=21 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e37ca21388fa8f9e server_cookie=f4f441dfcc547826 in_seq=0 out_seq=0 2024-05-03T21:21:37.732 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:37.724+0000 7fb004ff9700 10 client.15120 ms_handle_connect on v2:172.21.15.82:6838/3644499578 2024-05-03T21:21:37.732 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:37.725+0000 7fb004ff9700 1 -- 172.21.15.67:0/176882995 <== mds.0 v2:172.21.15.82:6838/3644499578 1 ==== command_reply(tid 0: 0 ) ==== 8+0+332 (crc 0 0 0) 0x5597fb3f5290 con 0x5597fb528c70 2024-05-03T21:21:37.732 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:37.725+0000 7fb004ff9700 10 client.15120 handle_command_reply: tid=0 2024-05-03T21:21:37.732 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:37.725+0000 7fb02dbb9b80 1 client.15120 shutdown 2024-05-03T21:21:37.732 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:37.725+0000 7fb02dbb9b80 1 -- 172.21.15.67:0/176882995 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5597fb49f230 msgr2=0x5597fb517150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:21:37.732 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:37.725+0000 7fb02dbb9b80 1 --2- 172.21.15.67:0/176882995 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5597fb49f230 0x5597fb517150 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7fb0140109c0 tx=0x7fb0140109f0 comp rx=0 tx=0).stop 2024-05-03T21:21:37.732 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:37.725+0000 7fb02dbb9b80 1 -- 172.21.15.67:0/176882995 shutdown_connections 2024-05-03T21:21:37.732 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:37.725+0000 7fb02dbb9b80 1 --2- 172.21.15.67:0/176882995 >> [v2:172.21.15.82:6838/3644499578,v1:172.21.15.82:6839/3644499578] conn(0x5597fb528c70 0x5597fb527980 crc :-1 s=READY pgs=21 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:21:37.733 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:37.725+0000 7fb02dbb9b80 1 --2- 172.21.15.67:0/176882995 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5597fb49f230 0x5597fb517150 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:21:37.733 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:37.725+0000 7fb02dbb9b80 1 --2- 172.21.15.67:0/176882995 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5597fb49ac10 0x5597fb51c850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:21:37.733 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:37.725+0000 7fb02dbb9b80 1 --2- 172.21.15.67:0/176882995 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5597fb49b530 0x5597fb51cd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:21:37.733 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:37.725+0000 7fb02dbb9b80 1 -- 172.21.15.67:0/176882995 >> 172.21.15.67:0/176882995 conn(0x5597fb3c70b0 msgr2=0x5597fb49e850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T21:21:37.733 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:37.725+0000 7fb004ff9700 0 client.15120 ms_handle_reset on v2:172.21.15.82:6838/3644499578 2024-05-03T21:21:37.733 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:37.725+0000 7fb02dbb9b80 1 -- 172.21.15.67:0/176882995 shutdown_connections 2024-05-03T21:21:37.733 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:37.725+0000 7fb02dbb9b80 1 -- 172.21.15.67:0/176882995 wait complete. 2024-05-03T21:21:37.733 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:37.725+0000 7fb02dbb9b80 20 client.15120 trim_cache size 0 max 16384 2024-05-03T21:21:37.733 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-05-03T21:21:37.733 INFO:teuthology.orchestra.run.smithi067.stdout: "cluster_fsid": "713bc57a-0991-11ef-bc97-c7b262605968", 2024-05-03T21:21:37.734 INFO:teuthology.orchestra.run.smithi067.stdout: "whoami": 0, 2024-05-03T21:21:37.734 INFO:teuthology.orchestra.run.smithi067.stdout: "id": 24473, 2024-05-03T21:21:37.734 INFO:teuthology.orchestra.run.smithi067.stdout: "want_state": "up:active", 2024-05-03T21:21:37.734 INFO:teuthology.orchestra.run.smithi067.stdout: "state": "up:active", 2024-05-03T21:21:37.734 INFO:teuthology.orchestra.run.smithi067.stdout: "fs_name": "cephfs", 2024-05-03T21:21:37.734 INFO:teuthology.orchestra.run.smithi067.stdout: "rank_uptime": 172.90886278100001, 2024-05-03T21:21:37.734 INFO:teuthology.orchestra.run.smithi067.stdout: "mdsmap_epoch": 25, 2024-05-03T21:21:37.734 INFO:teuthology.orchestra.run.smithi067.stdout: "osdmap_epoch": 84, 2024-05-03T21:21:37.734 INFO:teuthology.orchestra.run.smithi067.stdout: "osdmap_epoch_barrier": 82, 2024-05-03T21:21:37.734 INFO:teuthology.orchestra.run.smithi067.stdout: "uptime": 187.10935931899999 2024-05-03T21:21:37.734 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-05-03T21:21:37.735 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:37.726+0000 7fb023326700 1 -- 172.21.15.67:0/1524448263 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7fb0080b2990 msgr2=0x7fb0080b4e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:21:37.735 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:37.726+0000 7fb023326700 1 --2- 172.21.15.67:0/1524448263 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7fb0080b2990 0x7fb0080b4e50 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fb018000c00 tx=0x7fb018005fb0 comp rx=0 tx=0).stop 2024-05-03T21:21:37.735 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:37.726+0000 7fb023326700 1 -- 172.21.15.67:0/1524448263 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb01c06c630 msgr2=0x7fb01c19ffa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:21:37.735 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:37.726+0000 7fb023326700 1 --2- 172.21.15.67:0/1524448263 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb01c06c630 0x7fb01c19ffa0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7fb00c00d950 tx=0x7fb00c00dc60 comp rx=0 tx=0).stop 2024-05-03T21:21:37.735 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:37.726+0000 7fb023326700 1 -- 172.21.15.67:0/1524448263 shutdown_connections 2024-05-03T21:21:37.736 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:37.726+0000 7fb023326700 1 --2- 172.21.15.67:0/1524448263 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7fb0080b2990 0x7fb0080b4e50 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:21:37.736 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:37.726+0000 7fb023326700 1 --2- 172.21.15.67:0/1524448263 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fb01c109590 0x7fb01c199b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:21:37.736 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:37.726+0000 7fb023326700 1 --2- 172.21.15.67:0/1524448263 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fb01c06bc70 0x7fb01c19fa60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:21:37.736 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:37.726+0000 7fb023326700 1 --2- 172.21.15.67:0/1524448263 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb01c06c630 0x7fb01c19ffa0 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:21:37.736 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:37.726+0000 7fb023326700 1 -- 172.21.15.67:0/1524448263 >> 172.21.15.67:0/1524448263 conn(0x7fb01c079440 msgr2=0x7fb01c100d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T21:21:37.736 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:37.726+0000 7fb023326700 1 -- 172.21.15.67:0/1524448263 shutdown_connections 2024-05-03T21:21:37.736 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:37.726+0000 7fb023326700 1 -- 172.21.15.67:0/1524448263 wait complete. 2024-05-03T21:21:37.745 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:21:37 smithi082 ceph-mon[32933]: pgmap v375: 97 pgs: 97 active+clean; 632 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:21:38.511 INFO:teuthology.run_tasks:Running task sequential... 2024-05-03T21:21:38.521 INFO:teuthology.task.sequential:In sequential, running task sequential... 2024-05-03T21:21:38.521 INFO:teuthology.task.sequential:In sequential, running task print... 2024-05-03T21:21:38.521 INFO:teuthology.task.print:Enabling mgr modules 2024-05-03T21:21:38.521 INFO:teuthology.run_tasks:Running task kclient... 2024-05-03T21:21:38.550 INFO:tasks.kclient:Mounting kernel clients... 2024-05-03T21:21:38.550 INFO:tasks.kclient:config is {'client.0': {}} 2024-05-03T21:21:38.550 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-03T21:21:38.550 DEBUG:teuthology.orchestra.run.smithi067:> ip netns list 2024-05-03T21:21:38.567 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-03T21:21:38.568 DEBUG:teuthology.orchestra.run.smithi067:> sudo ip link delete ceph-brx 2024-05-03T21:21:38.636 INFO:teuthology.orchestra.run.smithi067.stderr:Cannot find device "ceph-brx" 2024-05-03T21:21:38.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-03T21:21:38.638 INFO:tasks.kclient:client.0 config is {'mntopts': ['ms_mode=crc', 'wsync'], 'syntax': 'v1', 'mount_subvol_num': 1} 2024-05-03T21:21:38.638 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-05-03T21:21:38.638 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-05-03T21:21:38.638 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9a047fe7599a549042d06a3fb5b6813a203eabea shell --fsid 713bc57a-0991-11ef-bc97-c7b262605968 -- ceph fs ls 2024-05-03T21:21:38.930 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/713bc57a-0991-11ef-bc97-c7b262605968/mon.a/config 2024-05-03T21:21:39.409 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:21:39 smithi067 ceph-mon[29730]: pgmap v376: 97 pgs: 97 active+clean; 632 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:21:39.574 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:21:39 smithi138 ceph-mon[33110]: pgmap v376: 97 pgs: 97 active+clean; 632 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:21:39.745 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:21:39 smithi082 ceph-mon[32933]: pgmap v376: 97 pgs: 97 active+clean; 632 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:21:40.722 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:40.719+0000 7f50ca55d700 1 -- 172.21.15.67:0/1561871304 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f50c406c630 msgr2=0x7f50c4110360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:21:40.722 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:40.719+0000 7f50ca55d700 1 --2- 172.21.15.67:0/1561871304 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f50c406c630 0x7f50c4110360 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f50bc00b0d0 tx=0x7f50bc00b3e0 comp rx=0 tx=0).stop 2024-05-03T21:21:40.722 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:40.719+0000 7f50ca55d700 1 -- 172.21.15.67:0/1561871304 shutdown_connections 2024-05-03T21:21:40.723 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:40.719+0000 7f50ca55d700 1 --2- 172.21.15.67:0/1561871304 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f50c406c630 0x7f50c4110360 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:21:40.723 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:40.719+0000 7f50ca55d700 1 --2- 172.21.15.67:0/1561871304 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f50c4109590 0x7f50c4109970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:21:40.723 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:40.719+0000 7f50ca55d700 1 --2- 172.21.15.67:0/1561871304 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f50c406bc70 0x7f50c406c0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:21:40.723 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:40.719+0000 7f50ca55d700 1 -- 172.21.15.67:0/1561871304 >> 172.21.15.67:0/1561871304 conn(0x7f50c4079440 msgr2=0x7f50c4079850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T21:21:40.723 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:40.719+0000 7f50ca55d700 1 -- 172.21.15.67:0/1561871304 shutdown_connections 2024-05-03T21:21:40.723 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:40.720+0000 7f50ca55d700 1 -- 172.21.15.67:0/1561871304 wait complete. 2024-05-03T21:21:40.723 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:40.720+0000 7f50ca55d700 1 Processor -- start 2024-05-03T21:21:40.723 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:40.720+0000 7f50ca55d700 1 -- start start 2024-05-03T21:21:40.725 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:40.720+0000 7f50ca55d700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f50c406bc70 0x7f50c419f850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:21:40.725 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:40.720+0000 7f50ca55d700 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f50c406c630 0x7f50c419fd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:21:40.725 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:40.721+0000 7f50ca55d700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f50c4109590 0x7f50c4199a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:21:40.725 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:40.721+0000 7f50ca55d700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f50c4113990 con 0x7f50c4109590 2024-05-03T21:21:40.725 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:40.721+0000 7f50ca55d700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f50c4113810 con 0x7f50c406bc70 2024-05-03T21:21:40.725 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:40.721+0000 7f50ca55d700 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f50c4113b10 con 0x7f50c406c630 2024-05-03T21:21:40.725 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:40.721+0000 7f50c8afa700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f50c4109590 0x7f50c4199a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:21:40.725 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:40.721+0000 7f50c8afa700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f50c4109590 0x7f50c4199a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:40804/0 (socket says 172.21.15.67:40804) 2024-05-03T21:21:40.725 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:40.721+0000 7f50c8afa700 1 -- 172.21.15.67:0/3938299827 learned_addr learned my addr 172.21.15.67:0/3938299827 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-03T21:21:40.726 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:40.721+0000 7f50c3fff700 1 --2- 172.21.15.67:0/3938299827 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f50c406bc70 0x7f50c419f850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:21:40.726 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:40.721+0000 7f50c37fe700 1 --2- 172.21.15.67:0/3938299827 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f50c406c630 0x7f50c419fd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:21:40.726 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:40.721+0000 7f50c8afa700 1 -- 172.21.15.67:0/3938299827 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f50c406c630 msgr2=0x7f50c419fd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:21:40.726 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:40.721+0000 7f50c8afa700 1 --2- 172.21.15.67:0/3938299827 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f50c406c630 0x7f50c419fd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:21:40.726 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:40.721+0000 7f50c8afa700 1 -- 172.21.15.67:0/3938299827 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f50c406bc70 msgr2=0x7f50c419f850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:21:40.726 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:40.721+0000 7f50c8afa700 1 --2- 172.21.15.67:0/3938299827 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f50c406bc70 0x7f50c419f850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:21:40.726 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:40.721+0000 7f50c8afa700 1 -- 172.21.15.67:0/3938299827 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f50c419a170 con 0x7f50c4109590 2024-05-03T21:21:40.726 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:40.721+0000 7f50c37fe700 1 --2- 172.21.15.67:0/3938299827 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f50c406c630 0x7f50c419fd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-03T21:21:40.726 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:40.721+0000 7f50c8afa700 1 --2- 172.21.15.67:0/3938299827 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f50c4109590 0x7f50c4199a10 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f50bc007a20 tx=0x7f50bc007a50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T21:21:40.726 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:40.721+0000 7f50c17fa700 1 -- 172.21.15.67:0/3938299827 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f50bc01f070 con 0x7f50c4109590 2024-05-03T21:21:40.726 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:40.721+0000 7f50c17fa700 1 -- 172.21.15.67:0/3938299827 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(28 keys) ==== 1170+0+0 (secure 0 0 0) 0x7f50bc003c00 con 0x7f50c4109590 2024-05-03T21:21:40.726 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:40.721+0000 7f50ca55d700 1 -- 172.21.15.67:0/3938299827 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f50c419a400 con 0x7f50c4109590 2024-05-03T21:21:40.726 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:40.721+0000 7f50c17fa700 1 -- 172.21.15.67:0/3938299827 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f50bc01b8a0 con 0x7f50c4109590 2024-05-03T21:21:40.726 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:40.722+0000 7f50ca55d700 1 -- 172.21.15.67:0/3938299827 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f50c41a6720 con 0x7f50c4109590 2024-05-03T21:21:40.727 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:40.722+0000 7f50ca55d700 1 -- 172.21.15.67:0/3938299827 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f50c406a190 con 0x7f50c4109590 2024-05-03T21:21:40.727 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:40.724+0000 7f50c17fa700 1 -- 172.21.15.67:0/3938299827 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 17) ==== 149703+0+0 (secure 0 0 0) 0x7f50bc022070 con 0x7f50c4109590 2024-05-03T21:21:40.727 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:40.724+0000 7f50c17fa700 1 --2- 172.21.15.67:0/3938299827 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f50ac0b2910 0x7f50ac0b4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:21:40.727 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:40.724+0000 7f50c3fff700 1 --2- 172.21.15.67:0/3938299827 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f50ac0b2910 0x7f50ac0b4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:21:40.728 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:40.724+0000 7f50c17fa700 1 -- 172.21.15.67:0/3938299827 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 8211+0+0 (secure 0 0 0) 0x7f50bc00f020 con 0x7f50c4109590 2024-05-03T21:21:40.728 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:40.725+0000 7f50c3fff700 1 --2- 172.21.15.67:0/3938299827 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f50ac0b2910 0x7f50ac0b4dd0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f50c419ab50 tx=0x7f50b4009190 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T21:21:40.730 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:40.728+0000 7f50c17fa700 1 -- 172.21.15.67:0/3938299827 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191839 (secure 0 0 0) 0x7f50bc029020 con 0x7f50c4109590 2024-05-03T21:21:40.991 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:40.988+0000 7f50ca55d700 1 -- 172.21.15.67:0/3938299827 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f50c41a68f0 con 0x7f50c4109590 2024-05-03T21:21:40.991 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:40.989+0000 7f50c17fa700 1 -- 172.21.15.67:0/3938299827 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v25) ==== 53+0+73 (secure 0 0 0) 0x7f50bc0ab660 con 0x7f50c4109590 2024-05-03T21:21:40.991 INFO:teuthology.orchestra.run.smithi067.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-05-03T21:21:40.993 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:40.991+0000 7f50ca55d700 1 -- 172.21.15.67:0/3938299827 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f50ac0b2910 msgr2=0x7f50ac0b4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:21:40.993 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:40.991+0000 7f50ca55d700 1 --2- 172.21.15.67:0/3938299827 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f50ac0b2910 0x7f50ac0b4dd0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f50c419ab50 tx=0x7f50b4009190 comp rx=0 tx=0).stop 2024-05-03T21:21:40.993 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:40.991+0000 7f50ca55d700 1 -- 172.21.15.67:0/3938299827 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f50c4109590 msgr2=0x7f50c4199a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:21:40.993 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:40.991+0000 7f50ca55d700 1 --2- 172.21.15.67:0/3938299827 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f50c4109590 0x7f50c4199a10 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f50bc007a20 tx=0x7f50bc007a50 comp rx=0 tx=0).stop 2024-05-03T21:21:40.993 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:40.991+0000 7f50ca55d700 1 -- 172.21.15.67:0/3938299827 shutdown_connections 2024-05-03T21:21:40.993 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:40.991+0000 7f50ca55d700 1 --2- 172.21.15.67:0/3938299827 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f50ac0b2910 0x7f50ac0b4dd0 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:21:40.994 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:40.991+0000 7f50ca55d700 1 --2- 172.21.15.67:0/3938299827 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f50c4109590 0x7f50c4199a10 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:21:40.994 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:40.991+0000 7f50ca55d700 1 --2- 172.21.15.67:0/3938299827 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f50c406bc70 0x7f50c419f850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:21:40.994 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:40.991+0000 7f50ca55d700 1 --2- 172.21.15.67:0/3938299827 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f50c406c630 0x7f50c419fd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:21:40.994 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:40.991+0000 7f50ca55d700 1 -- 172.21.15.67:0/3938299827 >> 172.21.15.67:0/3938299827 conn(0x7f50c4079440 msgr2=0x7f50c4100b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T21:21:40.994 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:40.991+0000 7f50ca55d700 1 -- 172.21.15.67:0/3938299827 shutdown_connections 2024-05-03T21:21:40.994 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:40.991+0000 7f50ca55d700 1 -- 172.21.15.67:0/3938299827 wait complete. 2024-05-03T21:21:41.574 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:21:41 smithi138 ceph-mon[33110]: pgmap v377: 97 pgs: 97 active+clean; 632 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T21:21:41.575 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:21:41 smithi138 ceph-mon[33110]: from='client.? 172.21.15.67:0/3938299827' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-05-03T21:21:41.672 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:21:41 smithi067 ceph-mon[29730]: pgmap v377: 97 pgs: 97 active+clean; 632 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T21:21:41.672 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:21:41 smithi067 ceph-mon[29730]: from='client.? 172.21.15.67:0/3938299827' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-05-03T21:21:41.706 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-05-03T21:21:41.706 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-05-03T21:21:41.706 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi067.front.sepia.ceph.com 2024-05-03T21:21:41.706 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-05-03T21:21:41.706 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-05-03T21:21:41.706 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-05-03T21:21:41.706 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-05-03T21:21:41.706 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-05-03T21:21:41.706 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.0' 2024-05-03T21:21:41.706 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-03T21:21:41.707 DEBUG:teuthology.orchestra.run.smithi067:> ip addr 2024-05-03T21:21:41.724 INFO:teuthology.orchestra.run.smithi067.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-05-03T21:21:41.724 INFO:teuthology.orchestra.run.smithi067.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-05-03T21:21:41.725 INFO:teuthology.orchestra.run.smithi067.stdout: inet 127.0.0.1/8 scope host lo 2024-05-03T21:21:41.725 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-05-03T21:21:41.725 INFO:teuthology.orchestra.run.smithi067.stdout: inet6 ::1/128 scope host 2024-05-03T21:21:41.725 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-05-03T21:21:41.725 INFO:teuthology.orchestra.run.smithi067.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-05-03T21:21:41.725 INFO:teuthology.orchestra.run.smithi067.stdout: link/ether 0c:c4:7a:a8:b4:7e brd ff:ff:ff:ff:ff:ff 2024-05-03T21:21:41.725 INFO:teuthology.orchestra.run.smithi067.stdout: altname enp7s0f0 2024-05-03T21:21:41.725 INFO:teuthology.orchestra.run.smithi067.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-05-03T21:21:41.725 INFO:teuthology.orchestra.run.smithi067.stdout: link/ether 0c:c4:7a:a8:b4:7f brd ff:ff:ff:ff:ff:ff 2024-05-03T21:21:41.725 INFO:teuthology.orchestra.run.smithi067.stdout: altname enp7s0f1 2024-05-03T21:21:41.725 INFO:teuthology.orchestra.run.smithi067.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-05-03T21:21:41.726 INFO:teuthology.orchestra.run.smithi067.stdout: link/ether 0c:c4:7a:b6:fc:0a brd ff:ff:ff:ff:ff:ff 2024-05-03T21:21:41.726 INFO:teuthology.orchestra.run.smithi067.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-05-03T21:21:41.726 INFO:teuthology.orchestra.run.smithi067.stdout: link/ether 0c:c4:7a:b6:fc:0b brd ff:ff:ff:ff:ff:ff 2024-05-03T21:21:41.726 INFO:teuthology.orchestra.run.smithi067.stdout: inet 172.21.15.67/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-05-03T21:21:41.726 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft 42055sec preferred_lft 42055sec 2024-05-03T21:21:41.726 INFO:teuthology.orchestra.run.smithi067.stdout: inet6 fe80::b1d0:1ed6:6e4a:bb3d/64 scope link noprefixroute 2024-05-03T21:21:41.726 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-05-03T21:21:41.726 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-05-03T21:21:41.727 DEBUG:teuthology.orchestra.run.smithi067:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-05-03T21:21:41.727 DEBUG:teuthology.orchestra.run.smithi067:> set -e 2024-05-03T21:21:41.727 DEBUG:teuthology.orchestra.run.smithi067:> sudo ip link add name ceph-brx type bridge 2024-05-03T21:21:41.727 DEBUG:teuthology.orchestra.run.smithi067:> sudo ip addr flush dev ceph-brx 2024-05-03T21:21:41.727 DEBUG:teuthology.orchestra.run.smithi067:> sudo ip link set ceph-brx up 2024-05-03T21:21:41.727 DEBUG:teuthology.orchestra.run.smithi067:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-05-03T21:21:41.727 DEBUG:teuthology.orchestra.run.smithi067:> ') 2024-05-03T21:21:41.745 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:21:41 smithi082 ceph-mon[32933]: pgmap v377: 97 pgs: 97 active+clean; 632 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T21:21:41.745 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:21:41 smithi082 ceph-mon[32933]: from='client.? 172.21.15.67:0/3938299827' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-05-03T21:21:41.818 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:41 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-05-03T21:21:41.915 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:41 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-05-03T21:21:41.921 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-03T21:21:41.921 DEBUG:teuthology.orchestra.run.smithi067:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-05-03T21:21:41.946 INFO:teuthology.orchestra.run.smithi067.stdout:1 2024-05-03T21:21:41.948 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-03T21:21:41.948 DEBUG:teuthology.orchestra.run.smithi067:> route 2024-05-03T21:21:42.020 INFO:teuthology.orchestra.run.smithi067.stdout:Kernel IP routing table 2024-05-03T21:21:42.020 INFO:teuthology.orchestra.run.smithi067.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-05-03T21:21:42.020 INFO:teuthology.orchestra.run.smithi067.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-05-03T21:21:42.020 INFO:teuthology.orchestra.run.smithi067.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-05-03T21:21:42.020 INFO:teuthology.orchestra.run.smithi067.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-05-03T21:21:42.021 DEBUG:teuthology.orchestra.run.smithi067:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-05-03T21:21:42.021 DEBUG:teuthology.orchestra.run.smithi067:> set -e 2024-05-03T21:21:42.021 DEBUG:teuthology.orchestra.run.smithi067:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-05-03T21:21:42.021 DEBUG:teuthology.orchestra.run.smithi067:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-05-03T21:21:42.021 DEBUG:teuthology.orchestra.run.smithi067:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-05-03T21:21:42.021 DEBUG:teuthology.orchestra.run.smithi067:> ') 2024-05-03T21:21:42.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:42 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-05-03T21:21:42.301 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:42 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-05-03T21:21:42.307 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-03T21:21:42.307 DEBUG:teuthology.orchestra.run.smithi067:> ip netns list 2024-05-03T21:21:42.323 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-03T21:21:42.323 DEBUG:teuthology.orchestra.run.smithi067:> ip netns list-id 2024-05-03T21:21:42.378 DEBUG:teuthology.orchestra.run.smithi067:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-05-03T21:21:42.379 DEBUG:teuthology.orchestra.run.smithi067:> set -e 2024-05-03T21:21:42.379 DEBUG:teuthology.orchestra.run.smithi067:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.0 2024-05-03T21:21:42.379 DEBUG:teuthology.orchestra.run.smithi067:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.0 0 2024-05-03T21:21:42.379 DEBUG:teuthology.orchestra.run.smithi067:> ') 2024-05-03T21:21:42.468 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:42 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-05-03T21:21:42.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:42 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-05-03T21:21:42.499 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.0' with 192.168.0.1/16 2024-05-03T21:21:42.499 DEBUG:teuthology.orchestra.run.smithi067:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-05-03T21:21:42.499 DEBUG:teuthology.orchestra.run.smithi067:> set -e 2024-05-03T21:21:42.499 DEBUG:teuthology.orchestra.run.smithi067:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.0 type veth peer name brx.0 2024-05-03T21:21:42.500 DEBUG:teuthology.orchestra.run.smithi067:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-05-03T21:21:42.500 DEBUG:teuthology.orchestra.run.smithi067:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set veth0 up 2024-05-03T21:21:42.500 DEBUG:teuthology.orchestra.run.smithi067:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set lo up 2024-05-03T21:21:42.500 DEBUG:teuthology.orchestra.run.smithi067:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip route add default via 192.168.255.254 2024-05-03T21:21:42.500 DEBUG:teuthology.orchestra.run.smithi067:> ') 2024-05-03T21:21:42.549 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:42 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-05-03T21:21:42.620 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:42 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-05-03T21:21:42.626 DEBUG:teuthology.orchestra.run.smithi067:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-05-03T21:21:42.626 DEBUG:teuthology.orchestra.run.smithi067:> set -e 2024-05-03T21:21:42.626 DEBUG:teuthology.orchestra.run.smithi067:> sudo ip link set brx.0 up 2024-05-03T21:21:42.626 DEBUG:teuthology.orchestra.run.smithi067:> sudo ip link set dev brx.0 master ceph-brx 2024-05-03T21:21:42.626 DEBUG:teuthology.orchestra.run.smithi067:> ') 2024-05-03T21:21:42.675 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:42 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-05-03T21:21:42.697 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:42 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-05-03T21:21:42.703 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-05-03T21:21:42.703 DEBUG:teuthology.orchestra.run.smithi067:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-05-03T21:21:42.718 INFO:teuthology.orchestra.run.smithi067.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-05-03T21:21:42.718 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-05-03T21:21:42.718 DEBUG:teuthology.orchestra.run.smithi067:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-05-03T21:21:42.774 WARNING:tasks.cephfs.mount:You cannot specify both: cephfs_mntpt and mount_subvol_num 2024-05-03T21:21:42.774 INFO:tasks.cephfs.mount:Mounting subvol 1 for now 2024-05-03T21:21:42.774 INFO:tasks.cephfs.kernel_mount:mounting using device: :/volumes/qa/sv_1/bfa342a3-081d-4b09-8d8d-b10dd6f0067e 2024-05-03T21:21:42.774 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-03T21:21:42.775 DEBUG:teuthology.orchestra.run.smithi067:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/volumes/qa/sv_1/bfa342a3-081d-4b09-8d8d-b10dd6f0067e /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs,ms_mode=crc,wsync 2024-05-03T21:21:42.872 INFO:teuthology.orchestra.run.smithi067.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs,ms_mode=crc,wsync 2024-05-03T21:21:42.872 INFO:teuthology.orchestra.run.smithi067.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,ms_mode=crc,wsync". 2024-05-03T21:21:42.872 INFO:teuthology.orchestra.run.smithi067.stdout:invalid new device string format 2024-05-03T21:21:42.989 INFO:teuthology.orchestra.run.smithi067.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs,ms_mode=crc,wsync 2024-05-03T21:21:42.989 INFO:teuthology.orchestra.run.smithi067.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,ms_mode=crc,wsync". 2024-05-03T21:21:42.989 INFO:teuthology.orchestra.run.smithi067.stdout:invalid new device string format 2024-05-03T21:21:42.989 INFO:teuthology.orchestra.run.smithi067.stdout:mount.ceph: resolved to: "172.21.15.67:3300,172.21.15.82:3300,172.21.15.138:3300" 2024-05-03T21:21:42.990 INFO:teuthology.orchestra.run.smithi067.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.67:3300,172.21.15.82:3300,172.21.15.138:3300:/volumes/qa/sv_1/bfa342a3-081d-4b09-8d8d-b10dd6f0067e 2024-05-03T21:21:42.990 INFO:teuthology.orchestra.run.smithi067.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,ms_mode=crc,wsync,key=0,fsid=713bc57a-0991-11ef-bc97-c7b262605968" will pass to kernel 2024-05-03T21:21:42.991 INFO:teuthology.orchestra.run.smithi067.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2024-05-03T21:21:42.991 INFO:teuthology.orchestra.run.smithi067.stdout: You just mounted a file system that supports labels which does not 2024-05-03T21:21:42.991 INFO:teuthology.orchestra.run.smithi067.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-05-03T21:21:42.991 INFO:teuthology.orchestra.run.smithi067.stdout: applications will generate AVC messages and not be allowed access to 2024-05-03T21:21:42.991 INFO:teuthology.orchestra.run.smithi067.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-05-03T21:21:42.992 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-05-03T21:21:42.993 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-03T21:21:42.993 DEBUG:teuthology.orchestra.run.smithi067:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-05-03T21:21:43.018 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-05-03T21:21:43.110 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:43 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-05-03T21:21:43.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:43 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-05-03T21:21:43.116 INFO:teuthology.orchestra.run.smithi067.stdout:client24760 2024-05-03T21:21:43.116 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-05-03T21:21:43.116 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/proc/self/mounts of=/dev/stdout 2024-05-03T21:21:43.134 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-05-03T21:21:43.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:43 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-05-03T21:21:43.226 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:43 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-05-03T21:21:43.232 INFO:teuthology.orchestra.run.smithi067.stdout:client24760 2024-05-03T21:21:43.233 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-05-03T21:21:43.285 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:43 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-05-03T21:21:43.287 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:43 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-05-03T21:21:43.292 INFO:teuthology.orchestra.run.smithi067.stdout:client24760 2024-05-03T21:21:43.292 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=300 -- bash -c 'sudo dd if=/sys/kernel/debug/ceph/713bc57a-0991-11ef-bc97-c7b262605968.client24760/status') 2024-05-03T21:21:43.342 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:43 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-05-03T21:21:43.351 INFO:teuthology.orchestra.run.smithi067.stdout:instance: client.24760 (3)172.21.15.67:0/2734669116 2024-05-03T21:21:43.351 INFO:teuthology.orchestra.run.smithi067.stdout:blocklisted: false 2024-05-03T21:21:43.352 INFO:teuthology.orchestra.run.smithi067.stderr:0+1 records in 2024-05-03T21:21:43.352 INFO:teuthology.orchestra.run.smithi067.stderr:0+1 records out 2024-05-03T21:21:43.352 INFO:teuthology.orchestra.run.smithi067.stderr:71 bytes copied, 2.3451e-05 s, 3.0 MB/s 2024-05-03T21:21:43.352 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:43 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-05-03T21:21:43.359 INFO:teuthology.run_tasks:Running task exec... 2024-05-03T21:21:43.369 INFO:teuthology.task.exec:Executing custom commands... 2024-05-03T21:21:43.370 INFO:teuthology.task.exec:Running commands on role mon.a host ubuntu@smithi067.front.sepia.ceph.com 2024-05-03T21:21:43.370 DEBUG:teuthology.orchestra.run.smithi067:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs set cephfs balance_automate false' 2024-05-03T21:21:43.476 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:43.471+0000 7f0e8fe2a640 1 -- 172.21.15.67:0/2239223081 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0e881087a0 msgr2=0x7f0e8810abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:21:43.476 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:43.471+0000 7f0e8fe2a640 1 --2- 172.21.15.67:0/2239223081 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0e881087a0 0x7f0e8810abf0 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f0e80009ca0 tx=0x7f0e8002a8c0 comp rx=0 tx=0).stop 2024-05-03T21:21:43.476 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:43.471+0000 7f0e8fe2a640 1 -- 172.21.15.67:0/2239223081 shutdown_connections 2024-05-03T21:21:43.476 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:43.471+0000 7f0e8fe2a640 1 --2- 172.21.15.67:0/2239223081 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0e881087a0 0x7f0e8810abf0 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:21:43.476 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:43.471+0000 7f0e8fe2a640 1 --2- 172.21.15.67:0/2239223081 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f0e88105e70 0x7f0e88108260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:21:43.476 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:43.471+0000 7f0e8fe2a640 1 --2- 172.21.15.67:0/2239223081 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f0e88103540 0x7f0e88105930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:21:43.476 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:43.471+0000 7f0e8fe2a640 1 -- 172.21.15.67:0/2239223081 >> 172.21.15.67:0/2239223081 conn(0x7f0e880fcf30 msgr2=0x7f0e880ff370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T21:21:43.477 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:43.471+0000 7f0e8fe2a640 1 -- 172.21.15.67:0/2239223081 shutdown_connections 2024-05-03T21:21:43.477 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:43.471+0000 7f0e8fe2a640 1 -- 172.21.15.67:0/2239223081 wait complete. 2024-05-03T21:21:43.477 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:43.471+0000 7f0e8fe2a640 1 Processor -- start 2024-05-03T21:21:43.477 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:43.471+0000 7f0e8fe2a640 1 -- start start 2024-05-03T21:21:43.477 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:43.471+0000 7f0e8fe2a640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f0e88103540 0x7f0e88198c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:21:43.477 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:43.471+0000 7f0e8fe2a640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f0e88105e70 0x7f0e881991d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:21:43.477 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:43.471+0000 7f0e8fe2a640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0e881087a0 0x7f0e881a0230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:21:43.477 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:43.471+0000 7f0e8fe2a640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f0e8810e810 con 0x7f0e881087a0 2024-05-03T21:21:43.477 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:43.471+0000 7f0e8fe2a640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f0e8810e690 con 0x7f0e88103540 2024-05-03T21:21:43.477 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:43.471+0000 7f0e8fe2a640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f0e8810e990 con 0x7f0e88105e70 2024-05-03T21:21:43.477 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:43.472+0000 7f0e8e3c7640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0e881087a0 0x7f0e881a0230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:21:43.477 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:43.472+0000 7f0e8e3c7640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0e881087a0 0x7f0e881a0230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:40822/0 (socket says 172.21.15.67:40822) 2024-05-03T21:21:43.478 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:43.472+0000 7f0e8e3c7640 1 -- 172.21.15.67:0/636348223 learned_addr learned my addr 172.21.15.67:0/636348223 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-03T21:21:43.478 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:43.472+0000 7f0e8dbc6640 1 --2- 172.21.15.67:0/636348223 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f0e88103540 0x7f0e88198c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:21:43.478 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:43.472+0000 7f0e8d3c5640 1 --2- 172.21.15.67:0/636348223 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f0e88105e70 0x7f0e881991d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:21:43.478 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:43.472+0000 7f0e8e3c7640 1 -- 172.21.15.67:0/636348223 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f0e88105e70 msgr2=0x7f0e881991d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:21:43.478 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:43.472+0000 7f0e8e3c7640 1 --2- 172.21.15.67:0/636348223 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f0e88105e70 0x7f0e881991d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:21:43.478 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:43.472+0000 7f0e8e3c7640 1 -- 172.21.15.67:0/636348223 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f0e88103540 msgr2=0x7f0e88198c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:21:43.478 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:43.472+0000 7f0e8e3c7640 1 --2- 172.21.15.67:0/636348223 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f0e88103540 0x7f0e88198c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:21:43.478 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:43.472+0000 7f0e8e3c7640 1 -- 172.21.15.67:0/636348223 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0e881a08d0 con 0x7f0e881087a0 2024-05-03T21:21:43.478 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:43.472+0000 7f0e8e3c7640 1 --2- 172.21.15.67:0/636348223 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0e881087a0 0x7f0e881a0230 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f0e800098d0 tx=0x7f0e8002f830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T21:21:43.478 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:43.472+0000 7f0e7effd640 1 -- 172.21.15.67:0/636348223 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e8002fc50 con 0x7f0e881087a0 2024-05-03T21:21:43.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:43.472+0000 7f0e8fe2a640 1 -- 172.21.15.67:0/636348223 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0e881a0b60 con 0x7f0e881087a0 2024-05-03T21:21:43.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:43.472+0000 7f0e7effd640 1 -- 172.21.15.67:0/636348223 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(28 keys) ==== 1170+0+0 (secure 0 0 0) 0x7f0e8002f310 con 0x7f0e881087a0 2024-05-03T21:21:43.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:43.472+0000 7f0e7effd640 1 -- 172.21.15.67:0/636348223 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e80004790 con 0x7f0e881087a0 2024-05-03T21:21:43.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:43.472+0000 7f0e8fe2a640 1 -- 172.21.15.67:0/636348223 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0e881a1100 con 0x7f0e881087a0 2024-05-03T21:21:43.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:43.473+0000 7f0e8fe2a640 1 -- 172.21.15.67:0/636348223 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0e880fea20 con 0x7f0e881087a0 2024-05-03T21:21:43.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:43.475+0000 7f0e7effd640 1 -- 172.21.15.67:0/636348223 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 17) ==== 149703+0+0 (secure 0 0 0) 0x7f0e80004b40 con 0x7f0e881087a0 2024-05-03T21:21:43.480 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:43.476+0000 7f0e7effd640 1 --2- 172.21.15.67:0/636348223 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f0e740b2920 0x7f0e740b4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:21:43.480 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:43.476+0000 7f0e8dbc6640 1 --2- 172.21.15.67:0/636348223 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f0e740b2920 0x7f0e740b4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:21:43.480 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:43.476+0000 7f0e7effd640 1 -- 172.21.15.67:0/636348223 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 8211+0+0 (secure 0 0 0) 0x7f0e80103650 con 0x7f0e881087a0 2024-05-03T21:21:43.480 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:43.476+0000 7f0e8dbc6640 1 --2- 172.21.15.67:0/636348223 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f0e740b2920 0x7f0e740b4de0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f0e88199b60 tx=0x7f0e8400b390 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T21:21:43.482 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:43.480+0000 7f0e7effd640 1 -- 172.21.15.67:0/636348223 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191839 (secure 0 0 0) 0x7f0e800cb2d0 con 0x7f0e881087a0 2024-05-03T21:21:43.574 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:21:43 smithi138 ceph-mon[33110]: pgmap v378: 97 pgs: 97 active+clean; 632 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:21:43.714 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:43.711+0000 7f0e8fe2a640 1 -- 172.21.15.67:0/636348223 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"} v 0) -- 0x7f0e8802db50 con 0x7f0e881087a0 2024-05-03T21:21:43.722 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:21:43 smithi067 ceph-mon[29730]: pgmap v378: 97 pgs: 97 active+clean; 632 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:21:43.745 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:21:43 smithi082 ceph-mon[32933]: pgmap v378: 97 pgs: 97 active+clean; 632 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:21:44.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:44.282+0000 7f0e7effd640 1 -- 172.21.15.67:0/636348223 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]=0 v26) ==== 118+0+0 (secure 0 0 0) 0x7f0e800cabd0 con 0x7f0e881087a0 2024-05-03T21:21:44.287 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:44.285+0000 7f0e8fe2a640 1 -- 172.21.15.67:0/636348223 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f0e740b2920 msgr2=0x7f0e740b4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:21:44.287 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:44.285+0000 7f0e8fe2a640 1 --2- 172.21.15.67:0/636348223 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f0e740b2920 0x7f0e740b4de0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f0e88199b60 tx=0x7f0e8400b390 comp rx=0 tx=0).stop 2024-05-03T21:21:44.287 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:44.285+0000 7f0e8fe2a640 1 -- 172.21.15.67:0/636348223 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0e881087a0 msgr2=0x7f0e881a0230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:21:44.287 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:44.285+0000 7f0e8fe2a640 1 --2- 172.21.15.67:0/636348223 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0e881087a0 0x7f0e881a0230 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f0e800098d0 tx=0x7f0e8002f830 comp rx=0 tx=0).stop 2024-05-03T21:21:44.287 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:44.285+0000 7f0e8fe2a640 1 -- 172.21.15.67:0/636348223 shutdown_connections 2024-05-03T21:21:44.287 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:44.285+0000 7f0e8fe2a640 1 --2- 172.21.15.67:0/636348223 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f0e740b2920 0x7f0e740b4de0 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:21:44.287 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:44.285+0000 7f0e8fe2a640 1 --2- 172.21.15.67:0/636348223 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0e881087a0 0x7f0e881a0230 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:21:44.288 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:44.285+0000 7f0e8fe2a640 1 --2- 172.21.15.67:0/636348223 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f0e88105e70 0x7f0e881991d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:21:44.288 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:44.285+0000 7f0e8fe2a640 1 --2- 172.21.15.67:0/636348223 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f0e88103540 0x7f0e88198c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:21:44.288 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:44.285+0000 7f0e8fe2a640 1 -- 172.21.15.67:0/636348223 >> 172.21.15.67:0/636348223 conn(0x7f0e880fcf30 msgr2=0x7f0e88103e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T21:21:44.288 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:44.285+0000 7f0e8fe2a640 1 -- 172.21.15.67:0/636348223 shutdown_connections 2024-05-03T21:21:44.288 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:44.285+0000 7f0e8fe2a640 1 -- 172.21.15.67:0/636348223 wait complete. 2024-05-03T21:21:44.299 DEBUG:teuthology.orchestra.run.smithi067:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs subvolumegroup pin cephfs qa random 0.10' 2024-05-03T21:21:44.408 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:44.404+0000 7f90db420640 1 -- 172.21.15.67:0/1165888135 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f90d41045f0 msgr2=0x7f90d410ae80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:21:44.409 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:44.404+0000 7f90db420640 1 --2- 172.21.15.67:0/1165888135 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f90d41045f0 0x7f90d410ae80 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f90d0009900 tx=0x7f90d002a410 comp rx=0 tx=0).stop 2024-05-03T21:21:44.409 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:44.404+0000 7f90db420640 1 -- 172.21.15.67:0/1165888135 shutdown_connections 2024-05-03T21:21:44.409 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:44.404+0000 7f90db420640 1 --2- 172.21.15.67:0/1165888135 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f90d41045f0 0x7f90d410ae80 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:21:44.409 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:44.404+0000 7f90db420640 1 --2- 172.21.15.67:0/1165888135 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f90d4103c30 0x7f90d41040b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:21:44.409 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:44.404+0000 7f90db420640 1 --2- 172.21.15.67:0/1165888135 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f90d4102a40 0x7f90d4102e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:21:44.409 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:44.404+0000 7f90db420640 1 -- 172.21.15.67:0/1165888135 >> 172.21.15.67:0/1165888135 conn(0x7f90d40fdfd0 msgr2=0x7f90d41003f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T21:21:44.409 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:44.404+0000 7f90db420640 1 -- 172.21.15.67:0/1165888135 shutdown_connections 2024-05-03T21:21:44.409 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:44.404+0000 7f90db420640 1 -- 172.21.15.67:0/1165888135 wait complete. 2024-05-03T21:21:44.409 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:44.405+0000 7f90db420640 1 Processor -- start 2024-05-03T21:21:44.409 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:44.405+0000 7f90db420640 1 -- start start 2024-05-03T21:21:44.409 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:44.405+0000 7f90db420640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f90d4102a40 0x7f90d4076760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:21:44.409 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:44.405+0000 7f90db420640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f90d4103c30 0x7f90d4074da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:21:44.410 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:44.405+0000 7f90db420640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f90d41045f0 0x7f90d40752e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:21:44.410 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:44.405+0000 7f90db420640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f90d410e830 con 0x7f90d41045f0 2024-05-03T21:21:44.410 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:44.405+0000 7f90db420640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f90d410e6b0 con 0x7f90d4103c30 2024-05-03T21:21:44.410 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:44.405+0000 7f90db420640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f90d410e9b0 con 0x7f90d4102a40 2024-05-03T21:21:44.410 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:44.405+0000 7f90d99bd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f90d41045f0 0x7f90d40752e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:21:44.410 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:44.405+0000 7f90d99bd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f90d41045f0 0x7f90d40752e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:40852/0 (socket says 172.21.15.67:40852) 2024-05-03T21:21:44.410 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:44.405+0000 7f90d99bd640 1 -- 172.21.15.67:0/432660756 learned_addr learned my addr 172.21.15.67:0/432660756 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-03T21:21:44.410 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:44.405+0000 7f90d91bc640 1 --2- 172.21.15.67:0/432660756 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f90d4102a40 0x7f90d4076760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:21:44.410 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:44.405+0000 7f90d89bb640 1 --2- 172.21.15.67:0/432660756 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f90d4103c30 0x7f90d4074da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:21:44.410 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:44.405+0000 7f90d99bd640 1 -- 172.21.15.67:0/432660756 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f90d4102a40 msgr2=0x7f90d4076760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:21:44.410 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:44.405+0000 7f90d99bd640 1 --2- 172.21.15.67:0/432660756 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f90d4102a40 0x7f90d4076760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:21:44.410 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:44.405+0000 7f90d99bd640 1 -- 172.21.15.67:0/432660756 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f90d4103c30 msgr2=0x7f90d4074da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:21:44.411 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:44.405+0000 7f90d99bd640 1 --2- 172.21.15.67:0/432660756 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f90d4103c30 0x7f90d4074da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:21:44.411 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:44.405+0000 7f90d99bd640 1 -- 172.21.15.67:0/432660756 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f90d4075b70 con 0x7f90d41045f0 2024-05-03T21:21:44.411 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:44.406+0000 7f90d99bd640 1 --2- 172.21.15.67:0/432660756 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f90d41045f0 0x7f90d40752e0 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f90d0005ab0 tx=0x7f90d0032cf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T21:21:44.411 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:44.406+0000 7f90ca7fc640 1 -- 172.21.15.67:0/432660756 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90d003f030 con 0x7f90d41045f0 2024-05-03T21:21:44.411 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:44.406+0000 7f90ca7fc640 1 -- 172.21.15.67:0/432660756 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(28 keys) ==== 1170+0+0 (secure 0 0 0) 0x7f90d0003d00 con 0x7f90d41045f0 2024-05-03T21:21:44.411 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:44.406+0000 7f90ca7fc640 1 -- 172.21.15.67:0/432660756 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90d0035070 con 0x7f90d41045f0 2024-05-03T21:21:44.411 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:44.406+0000 7f90db420640 1 -- 172.21.15.67:0/432660756 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f90d41a2d60 con 0x7f90d41045f0 2024-05-03T21:21:44.411 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:44.406+0000 7f90db420640 1 -- 172.21.15.67:0/432660756 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f90d41a31f0 con 0x7f90d41045f0 2024-05-03T21:21:44.412 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:44.407+0000 7f90db420640 1 -- 172.21.15.67:0/432660756 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f90d4109c90 con 0x7f90d41045f0 2024-05-03T21:21:44.412 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:44.408+0000 7f90ca7fc640 1 -- 172.21.15.67:0/432660756 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 17) ==== 149703+0+0 (secure 0 0 0) 0x7f90d0042030 con 0x7f90d41045f0 2024-05-03T21:21:44.412 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:44.409+0000 7f90ca7fc640 1 --2- 172.21.15.67:0/432660756 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f90c00b2920 0x7f90c00b4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:21:44.413 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:44.409+0000 7f90d91bc640 1 --2- 172.21.15.67:0/432660756 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f90c00b2920 0x7f90c00b4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:21:44.413 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:44.409+0000 7f90ca7fc640 1 -- 172.21.15.67:0/432660756 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 8211+0+0 (secure 0 0 0) 0x7f90d0102ca0 con 0x7f90d41045f0 2024-05-03T21:21:44.413 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:44.409+0000 7f90d91bc640 1 --2- 172.21.15.67:0/432660756 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f90c00b2920 0x7f90c00b4de0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f90d4103a90 tx=0x7f90cc009720 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T21:21:44.415 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:44.413+0000 7f90ca7fc640 1 -- 172.21.15.67:0/432660756 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191839 (secure 0 0 0) 0x7f90d00ca870 con 0x7f90d41045f0 2024-05-03T21:21:44.574 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:21:44 smithi138 ceph-mon[33110]: from='client.? 172.21.15.67:0/636348223' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]: dispatch 2024-05-03T21:21:44.634 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:21:44.632+0000 7f90db420640 1 -- 172.21.15.67:0/432660756 --> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] -- mgr_command(tid 0: {"prefix": "fs subvolumegroup pin", "vol_name": "cephfs", "group_name": "qa", "pin_type": "random", "pin_setting": "0.10", "target": ["mon-mgr", ""]}) -- 0x7f90d4062e90 con 0x7f90c00b2920 2024-05-03T21:21:44.721 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:21:44 smithi067 ceph-mon[29730]: from='client.? 172.21.15.67:0/636348223' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]: dispatch 2024-05-03T21:21:44.746 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:21:44 smithi082 ceph-mon[32933]: from='client.? 172.21.15.67:0/636348223' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]: dispatch 2024-05-03T21:21:45.575 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:21:45 smithi138 ceph-mon[33110]: pgmap v379: 97 pgs: 97 active+clean; 632 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:21:45.575 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:21:45 smithi138 ceph-mon[33110]: from='client.? 172.21.15.67:0/636348223' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]': finished 2024-05-03T21:21:45.575 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:21:45 smithi138 ceph-mon[33110]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 3 up:standby-replay 6 up:standby 2024-05-03T21:21:45.647 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:21:45 smithi082 ceph-mon[32933]: pgmap v379: 97 pgs: 97 active+clean; 632 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:21:45.647 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:21:45 smithi082 ceph-mon[32933]: from='client.? 172.21.15.67:0/636348223' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]': finished 2024-05-03T21:21:45.647 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:21:45 smithi082 ceph-mon[32933]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 3 up:standby-replay 6 up:standby 2024-05-03T21:21:45.721 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:21:45 smithi067 ceph-mon[29730]: pgmap v379: 97 pgs: 97 active+clean; 632 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:21:45.721 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:21:45 smithi067 ceph-mon[29730]: from='client.? 172.21.15.67:0/636348223' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]': finished 2024-05-03T21:21:45.721 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:21:45 smithi067 ceph-mon[29730]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 3 up:standby-replay 6 up:standby 2024-05-03T21:21:46.574 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:21:46 smithi138 ceph-mon[33110]: from='client.15138 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup pin", "vol_name": "cephfs", "group_name": "qa", "pin_type": "random", "pin_setting": "0.10", "target": ["mon-mgr", ""]}]: dispatch 2024-05-03T21:21:46.721 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:21:46 smithi067 ceph-mon[29730]: from='client.15138 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup pin", "vol_name": "cephfs", "group_name": "qa", "pin_type": "random", "pin_setting": "0.10", "target": ["mon-mgr", ""]}]: dispatch 2024-05-03T21:21:46.745 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:21:46 smithi082 ceph-mon[32933]: from='client.15138 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup pin", "vol_name": "cephfs", "group_name": "qa", "pin_type": "random", "pin_setting": "0.10", "target": ["mon-mgr", ""]}]: dispatch 2024-05-03T21:21:47.574 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:21:47 smithi138 ceph-mon[33110]: pgmap v380: 97 pgs: 97 active+clean; 632 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 170 B/s wr, 6 op/s 2024-05-03T21:21:47.721 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:21:47 smithi067 ceph-mon[29730]: pgmap v380: 97 pgs: 97 active+clean; 632 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 170 B/s wr, 6 op/s 2024-05-03T21:21:47.745 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:21:47 smithi082 ceph-mon[32933]: pgmap v380: 97 pgs: 97 active+clean; 632 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 170 B/s wr, 6 op/s 2024-05-03T21:21:49.574 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:21:49 smithi138 ceph-mon[33110]: pgmap v381: 97 pgs: 97 active+clean; 632 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 170 B/s wr, 6 op/s 2024-05-03T21:21:49.721 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:21:49 smithi067 ceph-mon[29730]: pgmap v381: 97 pgs: 97 active+clean; 632 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 170 B/s wr, 6 op/s 2024-05-03T21:21:49.745 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:21:49 smithi082 ceph-mon[32933]: pgmap v381: 97 pgs: 97 active+clean; 632 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 170 B/s wr, 6 op/s 2024-05-03T21:21:51.574 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:21:51 smithi138 ceph-mon[33110]: pgmap v382: 97 pgs: 97 active+clean; 632 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 170 B/s wr, 5 op/s 2024-05-03T21:21:51.721 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:21:51 smithi067 ceph-mon[29730]: pgmap v382: 97 pgs: 97 active+clean; 632 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 170 B/s wr, 5 op/s 2024-05-03T21:21:51.745 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:21:51 smithi082 ceph-mon[32933]: pgmap v382: 97 pgs: 97 active+clean; 632 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 170 B/s wr, 5 op/s 2024-05-03T21:21:52.574 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:21:52 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T21:21:52.721 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:21:52 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T21:21:52.745 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:21:52 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T21:21:53.575 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:21:53 smithi138 ceph-mon[33110]: pgmap v383: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 682 B/s wr, 6 op/s 2024-05-03T21:21:53.575 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:21:53 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:21:53.575 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:21:53 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:21:53.721 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:21:53 smithi067 ceph-mon[29730]: pgmap v383: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 682 B/s wr, 6 op/s 2024-05-03T21:21:53.721 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:21:53 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:21:53.721 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:21:53 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:21:53.745 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:21:53 smithi082 ceph-mon[32933]: pgmap v383: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 682 B/s wr, 6 op/s 2024-05-03T21:21:53.745 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:21:53 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:21:53.745 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:21:53 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:21:54.574 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:21:54 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T21:21:54.575 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:21:54 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T21:21:54.575 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:21:54 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:21:54.721 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:21:54 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T21:21:54.721 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:21:54 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T21:21:54.721 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:21:54 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:21:54.745 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:21:54 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T21:21:54.745 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:21:54 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T21:21:54.745 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:21:54 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:21:55.574 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:21:55 smithi138 ceph-mon[33110]: pgmap v384: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 682 B/s wr, 6 op/s 2024-05-03T21:21:55.622 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:21:55 smithi067 ceph-mon[29730]: pgmap v384: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 682 B/s wr, 6 op/s 2024-05-03T21:21:55.745 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:21:55 smithi082 ceph-mon[32933]: pgmap v384: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 682 B/s wr, 6 op/s 2024-05-03T21:21:57.574 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:21:57 smithi138 ceph-mon[33110]: pgmap v385: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 682 B/s wr, 6 op/s 2024-05-03T21:21:57.721 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:21:57 smithi067 ceph-mon[29730]: pgmap v385: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 682 B/s wr, 6 op/s 2024-05-03T21:21:57.746 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:21:57 smithi082 ceph-mon[32933]: pgmap v385: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 682 B/s wr, 6 op/s 2024-05-03T21:21:59.574 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:21:59 smithi138 ceph-mon[33110]: pgmap v386: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 511 B/s wr, 6 op/s 2024-05-03T21:21:59.721 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:21:59 smithi067 ceph-mon[29730]: pgmap v386: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 511 B/s wr, 6 op/s 2024-05-03T21:21:59.745 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:21:59 smithi082 ceph-mon[32933]: pgmap v386: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 511 B/s wr, 6 op/s 2024-05-03T21:22:01.574 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:22:01 smithi138 ceph-mon[33110]: pgmap v387: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 511 B/s wr, 5 op/s 2024-05-03T21:22:01.721 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:22:01 smithi067 ceph-mon[29730]: pgmap v387: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 511 B/s wr, 5 op/s 2024-05-03T21:22:01.745 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:22:01 smithi082 ceph-mon[32933]: pgmap v387: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 511 B/s wr, 5 op/s 2024-05-03T21:22:03.574 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:22:03 smithi138 ceph-mon[33110]: pgmap v388: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 511 B/s wr, 6 op/s 2024-05-03T21:22:03.721 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:22:03 smithi067 ceph-mon[29730]: pgmap v388: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 511 B/s wr, 6 op/s 2024-05-03T21:22:03.746 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:22:03 smithi082 ceph-mon[32933]: pgmap v388: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 511 B/s wr, 6 op/s 2024-05-03T21:22:05.722 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:22:05 smithi067 ceph-mon[29730]: pgmap v389: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:22:05.746 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:22:05 smithi082 ceph-mon[32933]: pgmap v389: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:22:05.825 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:22:05 smithi138 ceph-mon[33110]: pgmap v389: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:22:07.721 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:22:07 smithi067 ceph-mon[29730]: pgmap v390: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:22:07.745 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:22:07 smithi082 ceph-mon[32933]: pgmap v390: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:22:07.824 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:22:07 smithi138 ceph-mon[33110]: pgmap v390: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:22:09.721 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:22:09 smithi067 ceph-mon[29730]: pgmap v391: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:22:09.745 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:22:09 smithi082 ceph-mon[32933]: pgmap v391: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:22:09.824 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:22:09 smithi138 ceph-mon[33110]: pgmap v391: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:22:11.721 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:22:11 smithi067 ceph-mon[29730]: pgmap v392: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T21:22:11.745 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:22:11 smithi082 ceph-mon[32933]: pgmap v392: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T21:22:11.824 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:22:11 smithi138 ceph-mon[33110]: pgmap v392: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T21:22:13.627 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:22:13 smithi138 ceph-mon[33110]: pgmap v393: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:22:13.721 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:22:13 smithi067 ceph-mon[29730]: pgmap v393: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:22:13.745 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:22:13 smithi082 ceph-mon[32933]: pgmap v393: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:22:15.721 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:22:15 smithi067 ceph-mon[29730]: pgmap v394: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:22:15.745 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:22:15 smithi082 ceph-mon[32933]: pgmap v394: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:22:15.824 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:22:15 smithi138 ceph-mon[33110]: pgmap v394: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:22:17.721 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:22:17 smithi067 ceph-mon[29730]: pgmap v395: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:22:17.745 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:22:17 smithi082 ceph-mon[32933]: pgmap v395: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:22:17.824 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:22:17 smithi138 ceph-mon[33110]: pgmap v395: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:22:19.721 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:22:19 smithi067 ceph-mon[29730]: pgmap v396: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:22:19.745 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:22:19 smithi082 ceph-mon[32933]: pgmap v396: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:22:19.824 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:22:19 smithi138 ceph-mon[33110]: pgmap v396: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:22:21.721 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:22:21 smithi067 ceph-mon[29730]: pgmap v397: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T21:22:21.745 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:22:21 smithi082 ceph-mon[32933]: pgmap v397: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T21:22:21.824 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:22:21 smithi138 ceph-mon[33110]: pgmap v397: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T21:22:23.721 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:22:23 smithi067 ceph-mon[29730]: pgmap v398: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:22:23.745 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:22:23 smithi082 ceph-mon[32933]: pgmap v398: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:22:23.824 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:22:23 smithi138 ceph-mon[33110]: pgmap v398: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:22:25.721 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:22:25 smithi067 ceph-mon[29730]: pgmap v399: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:22:25.745 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:22:25 smithi082 ceph-mon[32933]: pgmap v399: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:22:25.824 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:22:25 smithi138 ceph-mon[33110]: pgmap v399: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:22:27.721 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:22:27 smithi067 ceph-mon[29730]: pgmap v400: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:22:27.745 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:22:27 smithi082 ceph-mon[32933]: pgmap v400: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:22:27.824 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:22:27 smithi138 ceph-mon[33110]: pgmap v400: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:22:29.721 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:22:29 smithi067 ceph-mon[29730]: pgmap v401: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:22:29.745 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:22:29 smithi082 ceph-mon[32933]: pgmap v401: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:22:29.824 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:22:29 smithi138 ceph-mon[33110]: pgmap v401: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:22:31.721 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:22:31 smithi067 ceph-mon[29730]: pgmap v402: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T21:22:31.745 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:22:31 smithi082 ceph-mon[32933]: pgmap v402: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T21:22:31.824 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:22:31 smithi138 ceph-mon[33110]: pgmap v402: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T21:22:33.721 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:22:33 smithi067 ceph-mon[29730]: pgmap v403: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:22:33.745 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:22:33 smithi082 ceph-mon[32933]: pgmap v403: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:22:33.824 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:22:33 smithi138 ceph-mon[33110]: pgmap v403: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:22:35.721 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:22:35 smithi067 ceph-mon[29730]: pgmap v404: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:22:35.745 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:22:35 smithi082 ceph-mon[32933]: pgmap v404: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:22:35.824 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:22:35 smithi138 ceph-mon[33110]: pgmap v404: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:22:37.721 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:22:37 smithi067 ceph-mon[29730]: pgmap v405: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:22:37.745 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:22:37 smithi082 ceph-mon[32933]: pgmap v405: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:22:37.824 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:22:37 smithi138 ceph-mon[33110]: pgmap v405: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:22:39.721 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:22:39 smithi067 ceph-mon[29730]: pgmap v406: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:22:39.745 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:22:39 smithi082 ceph-mon[32933]: pgmap v406: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:22:39.824 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:22:39 smithi138 ceph-mon[33110]: pgmap v406: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:22:41.721 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:22:41 smithi067 ceph-mon[29730]: pgmap v407: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T21:22:41.745 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:22:41 smithi082 ceph-mon[32933]: pgmap v407: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T21:22:41.824 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:22:41 smithi138 ceph-mon[33110]: pgmap v407: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T21:22:43.721 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:22:43 smithi067 ceph-mon[29730]: pgmap v408: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:22:43.745 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:22:43 smithi082 ceph-mon[32933]: pgmap v408: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:22:43.824 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:22:43 smithi138 ceph-mon[33110]: pgmap v408: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:22:45.721 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:22:45 smithi067 ceph-mon[29730]: pgmap v409: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:22:45.745 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:22:45 smithi082 ceph-mon[32933]: pgmap v409: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:22:45.824 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:22:45 smithi138 ceph-mon[33110]: pgmap v409: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:22:47.721 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:22:47 smithi067 ceph-mon[29730]: pgmap v410: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:22:47.745 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:22:47 smithi082 ceph-mon[32933]: pgmap v410: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:22:47.824 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:22:47 smithi138 ceph-mon[33110]: pgmap v410: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:22:49.721 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:22:49 smithi067 ceph-mon[29730]: pgmap v411: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:22:49.745 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:22:49 smithi082 ceph-mon[32933]: pgmap v411: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:22:49.824 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:22:49 smithi138 ceph-mon[33110]: pgmap v411: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:22:51.721 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:22:51 smithi067 ceph-mon[29730]: pgmap v412: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T21:22:51.745 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:22:51 smithi082 ceph-mon[32933]: pgmap v412: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T21:22:51.824 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:22:51 smithi138 ceph-mon[33110]: pgmap v412: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T21:22:53.721 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:22:53 smithi067 ceph-mon[29730]: pgmap v413: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:22:53.745 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:22:53 smithi082 ceph-mon[32933]: pgmap v413: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:22:53.824 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:22:53 smithi138 ceph-mon[33110]: pgmap v413: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:22:54.721 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:22:54 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T21:22:54.721 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:22:54 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:22:54.721 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:22:54 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:22:54.721 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:22:54 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:22:54.721 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:22:54 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:22:54.745 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:22:54 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T21:22:54.745 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:22:54 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:22:54.745 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:22:54 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:22:54.746 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:22:54 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:22:54.746 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:22:54 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:22:54.824 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:22:54 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T21:22:54.824 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:22:54 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:22:54.824 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:22:54 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:22:54.825 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:22:54 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:22:54.825 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:22:54 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:22:55.721 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:22:55 smithi067 ceph-mon[29730]: pgmap v414: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:22:55.721 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:22:55 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T21:22:55.721 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:22:55 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T21:22:55.721 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:22:55 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:22:55.745 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:22:55 smithi082 ceph-mon[32933]: pgmap v414: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:22:55.745 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:22:55 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T21:22:55.745 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:22:55 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T21:22:55.745 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:22:55 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:22:55.824 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:22:55 smithi138 ceph-mon[33110]: pgmap v414: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:22:55.824 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:22:55 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T21:22:55.824 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:22:55 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T21:22:55.825 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:22:55 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:22:57.721 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:22:57 smithi067 ceph-mon[29730]: pgmap v415: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:22:57.745 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:22:57 smithi082 ceph-mon[32933]: pgmap v415: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:22:57.824 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:22:57 smithi138 ceph-mon[33110]: pgmap v415: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:22:58.721 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:22:58 smithi067 ceph-mon[29730]: pgmap v416: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:22:58.745 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:22:58 smithi082 ceph-mon[32933]: pgmap v416: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:22:58.824 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:22:58 smithi138 ceph-mon[33110]: pgmap v416: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:23:01.221 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:23:00 smithi067 ceph-mon[29730]: pgmap v417: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T21:23:01.245 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:23:00 smithi082 ceph-mon[32933]: pgmap v417: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T21:23:01.324 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:23:00 smithi138 ceph-mon[33110]: pgmap v417: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T21:23:03.221 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:23:02 smithi067 ceph-mon[29730]: pgmap v418: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:23:03.245 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:23:02 smithi082 ceph-mon[32933]: pgmap v418: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:23:03.324 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:23:02 smithi138 ceph-mon[33110]: pgmap v418: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:23:05.221 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:23:04 smithi067 ceph-mon[29730]: pgmap v419: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:23:05.245 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:23:04 smithi082 ceph-mon[32933]: pgmap v419: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:23:05.324 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:23:04 smithi138 ceph-mon[33110]: pgmap v419: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:23:07.221 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:23:06 smithi067 ceph-mon[29730]: pgmap v420: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:23:07.245 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:23:06 smithi082 ceph-mon[32933]: pgmap v420: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:23:07.324 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:23:06 smithi138 ceph-mon[33110]: pgmap v420: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:23:09.221 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:23:08 smithi067 ceph-mon[29730]: pgmap v421: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:23:09.245 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:23:08 smithi082 ceph-mon[32933]: pgmap v421: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:23:09.324 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:23:08 smithi138 ceph-mon[33110]: pgmap v421: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:23:11.221 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:23:10 smithi067 ceph-mon[29730]: pgmap v422: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T21:23:11.245 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:23:10 smithi082 ceph-mon[32933]: pgmap v422: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T21:23:11.324 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:23:10 smithi138 ceph-mon[33110]: pgmap v422: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T21:23:13.221 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:23:12 smithi067 ceph-mon[29730]: pgmap v423: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:23:13.245 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:23:12 smithi082 ceph-mon[32933]: pgmap v423: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:23:13.324 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:23:12 smithi138 ceph-mon[33110]: pgmap v423: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:23:15.221 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:23:14 smithi067 ceph-mon[29730]: pgmap v424: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:23:15.245 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:23:14 smithi082 ceph-mon[32933]: pgmap v424: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:23:15.324 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:23:14 smithi138 ceph-mon[33110]: pgmap v424: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:23:17.221 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:23:16 smithi067 ceph-mon[29730]: pgmap v425: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:23:17.245 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:23:16 smithi082 ceph-mon[32933]: pgmap v425: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:23:17.324 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:23:16 smithi138 ceph-mon[33110]: pgmap v425: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:23:19.221 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:23:18 smithi067 ceph-mon[29730]: pgmap v426: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:23:19.245 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:23:18 smithi082 ceph-mon[32933]: pgmap v426: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:23:19.324 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:23:18 smithi138 ceph-mon[33110]: pgmap v426: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:23:21.221 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:23:20 smithi067 ceph-mon[29730]: pgmap v427: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T21:23:21.245 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:23:20 smithi082 ceph-mon[32933]: pgmap v427: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T21:23:21.324 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:23:20 smithi138 ceph-mon[33110]: pgmap v427: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T21:23:23.221 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:23:22 smithi067 ceph-mon[29730]: pgmap v428: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:23:23.245 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:23:22 smithi082 ceph-mon[32933]: pgmap v428: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:23:23.324 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:23:22 smithi138 ceph-mon[33110]: pgmap v428: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:23:25.221 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:23:24 smithi067 ceph-mon[29730]: pgmap v429: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:23:25.245 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:23:24 smithi082 ceph-mon[32933]: pgmap v429: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:23:25.324 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:23:24 smithi138 ceph-mon[33110]: pgmap v429: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:23:27.221 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:23:26 smithi067 ceph-mon[29730]: pgmap v430: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:23:27.245 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:23:26 smithi082 ceph-mon[32933]: pgmap v430: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:23:27.324 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:23:26 smithi138 ceph-mon[33110]: pgmap v430: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:23:29.245 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:23:28 smithi082 ceph-mon[32933]: pgmap v431: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:23:29.324 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:23:28 smithi138 ceph-mon[33110]: pgmap v431: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:23:29.471 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:23:28 smithi067 ceph-mon[29730]: pgmap v431: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:23:31.245 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:23:30 smithi082 ceph-mon[32933]: pgmap v432: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T21:23:31.325 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:23:30 smithi138 ceph-mon[33110]: pgmap v432: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T21:23:31.471 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:23:30 smithi067 ceph-mon[29730]: pgmap v432: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T21:23:33.245 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:23:32 smithi082 ceph-mon[32933]: pgmap v433: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:23:33.324 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:23:32 smithi138 ceph-mon[33110]: pgmap v433: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:23:33.471 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:23:32 smithi067 ceph-mon[29730]: pgmap v433: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:23:35.245 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:23:34 smithi082 ceph-mon[32933]: pgmap v434: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:23:35.324 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:23:34 smithi138 ceph-mon[33110]: pgmap v434: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:23:35.471 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:23:34 smithi067 ceph-mon[29730]: pgmap v434: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:23:37.245 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:23:36 smithi082 ceph-mon[32933]: pgmap v435: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:23:37.324 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:23:36 smithi138 ceph-mon[33110]: pgmap v435: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:23:37.471 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:23:36 smithi067 ceph-mon[29730]: pgmap v435: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:23:39.246 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:23:38 smithi082 ceph-mon[32933]: pgmap v436: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:23:39.324 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:23:38 smithi138 ceph-mon[33110]: pgmap v436: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:23:39.471 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:23:38 smithi067 ceph-mon[29730]: pgmap v436: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:23:41.245 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:23:40 smithi082 ceph-mon[32933]: pgmap v437: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T21:23:41.324 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:23:40 smithi138 ceph-mon[33110]: pgmap v437: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T21:23:41.471 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:23:40 smithi067 ceph-mon[29730]: pgmap v437: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T21:23:43.245 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:23:42 smithi082 ceph-mon[32933]: pgmap v438: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:23:43.324 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:23:42 smithi138 ceph-mon[33110]: pgmap v438: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:23:43.471 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:23:42 smithi067 ceph-mon[29730]: pgmap v438: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:23:45.245 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:23:44 smithi082 ceph-mon[32933]: pgmap v439: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:23:45.324 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:23:44 smithi138 ceph-mon[33110]: pgmap v439: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:23:45.471 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:23:44 smithi067 ceph-mon[29730]: pgmap v439: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:23:47.324 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:23:46 smithi138 ceph-mon[33110]: pgmap v440: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:23:47.471 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:23:46 smithi067 ceph-mon[29730]: pgmap v440: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:23:47.495 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:23:46 smithi082 ceph-mon[32933]: pgmap v440: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:23:49.324 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:23:49 smithi138 ceph-mon[33110]: pgmap v441: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:23:49.471 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:23:48 smithi067 ceph-mon[29730]: pgmap v441: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:23:49.495 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:23:49 smithi082 ceph-mon[32933]: pgmap v441: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:23:51.324 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:23:51 smithi138 ceph-mon[33110]: pgmap v442: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T21:23:51.471 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:23:51 smithi067 ceph-mon[29730]: pgmap v442: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T21:23:51.495 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:23:51 smithi082 ceph-mon[32933]: pgmap v442: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T21:23:53.324 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:23:53 smithi138 ceph-mon[33110]: pgmap v443: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:23:53.471 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:23:53 smithi067 ceph-mon[29730]: pgmap v443: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:23:53.495 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:23:53 smithi082 ceph-mon[32933]: pgmap v443: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:23:55.273 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:23:55 smithi067 ceph-mon[29730]: pgmap v444: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:23:55.274 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:23:55 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T21:23:55.324 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:23:55 smithi138 ceph-mon[33110]: pgmap v444: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:23:55.324 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:23:55 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T21:23:55.495 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:23:55 smithi082 ceph-mon[32933]: pgmap v444: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:23:55.495 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:23:55 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T21:23:56.324 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:23:56 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T21:23:56.324 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:23:56 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T21:23:56.325 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:23:56 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:23:56.471 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:23:56 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T21:23:56.472 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:23:56 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T21:23:56.472 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:23:56 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:23:56.495 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:23:56 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T21:23:56.496 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:23:56 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T21:23:56.496 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:23:56 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:23:57.324 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:23:57 smithi138 ceph-mon[33110]: pgmap v445: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:23:57.471 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:23:57 smithi067 ceph-mon[29730]: pgmap v445: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:23:57.495 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:23:57 smithi082 ceph-mon[32933]: pgmap v445: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:23:59.324 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:23:59 smithi138 ceph-mon[33110]: pgmap v446: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:23:59.471 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:23:59 smithi067 ceph-mon[29730]: pgmap v446: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:23:59.495 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:23:59 smithi082 ceph-mon[32933]: pgmap v446: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:24:01.324 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:24:01 smithi138 ceph-mon[33110]: pgmap v447: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T21:24:01.471 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:24:01 smithi067 ceph-mon[29730]: pgmap v447: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T21:24:01.495 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:24:01 smithi082 ceph-mon[32933]: pgmap v447: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T21:24:03.325 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:24:03 smithi138 ceph-mon[33110]: pgmap v448: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:24:03.471 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:24:03 smithi067 ceph-mon[29730]: pgmap v448: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:24:03.495 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:24:03 smithi082 ceph-mon[32933]: pgmap v448: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:24:05.324 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:24:05 smithi138 ceph-mon[33110]: pgmap v449: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:24:05.471 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:24:05 smithi067 ceph-mon[29730]: pgmap v449: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:24:05.495 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:24:05 smithi082 ceph-mon[32933]: pgmap v449: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:24:07.324 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:24:07 smithi138 ceph-mon[33110]: pgmap v450: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:24:07.471 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:24:07 smithi067 ceph-mon[29730]: pgmap v450: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:24:07.495 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:24:07 smithi082 ceph-mon[32933]: pgmap v450: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:24:09.324 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:24:09 smithi138 ceph-mon[33110]: pgmap v451: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:24:09.471 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:24:09 smithi067 ceph-mon[29730]: pgmap v451: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:24:09.495 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:24:09 smithi082 ceph-mon[32933]: pgmap v451: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:24:11.325 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:24:11 smithi138 ceph-mon[33110]: pgmap v452: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T21:24:11.471 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:24:11 smithi067 ceph-mon[29730]: pgmap v452: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T21:24:11.495 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:24:11 smithi082 ceph-mon[32933]: pgmap v452: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T21:24:13.324 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:24:13 smithi138 ceph-mon[33110]: pgmap v453: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:24:13.471 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:24:13 smithi067 ceph-mon[29730]: pgmap v453: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:24:13.495 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:24:13 smithi082 ceph-mon[32933]: pgmap v453: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:24:15.298 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:24:15 smithi082 ceph-mon[32933]: pgmap v454: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:24:15.324 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:24:15 smithi138 ceph-mon[33110]: pgmap v454: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:24:15.471 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:24:15 smithi067 ceph-mon[29730]: pgmap v454: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:24:17.325 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:24:17 smithi138 ceph-mon[33110]: pgmap v455: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 5 op/s 2024-05-03T21:24:17.471 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:24:17 smithi067 ceph-mon[29730]: pgmap v455: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 5 op/s 2024-05-03T21:24:17.495 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:24:17 smithi082 ceph-mon[32933]: pgmap v455: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 5 op/s 2024-05-03T21:24:19.324 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:24:19 smithi138 ceph-mon[33110]: pgmap v456: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-05-03T21:24:19.471 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:24:19 smithi067 ceph-mon[29730]: pgmap v456: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-05-03T21:24:19.495 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:24:19 smithi082 ceph-mon[32933]: pgmap v456: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-05-03T21:24:21.324 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:24:21 smithi138 ceph-mon[33110]: pgmap v457: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-05-03T21:24:21.471 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:24:21 smithi067 ceph-mon[29730]: pgmap v457: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-05-03T21:24:21.495 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:24:21 smithi082 ceph-mon[32933]: pgmap v457: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-05-03T21:24:23.324 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:24:23 smithi138 ceph-mon[33110]: pgmap v458: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 5 op/s 2024-05-03T21:24:23.471 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:24:23 smithi067 ceph-mon[29730]: pgmap v458: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 5 op/s 2024-05-03T21:24:23.495 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:24:23 smithi082 ceph-mon[32933]: pgmap v458: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 5 op/s 2024-05-03T21:24:25.324 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:24:25 smithi138 ceph-mon[33110]: pgmap v459: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-05-03T21:24:25.471 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:24:25 smithi067 ceph-mon[29730]: pgmap v459: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-05-03T21:24:25.495 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:24:25 smithi082 ceph-mon[32933]: pgmap v459: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-05-03T21:24:27.324 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:24:27 smithi138 ceph-mon[33110]: pgmap v460: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 5 op/s 2024-05-03T21:24:27.471 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:24:27 smithi067 ceph-mon[29730]: pgmap v460: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 5 op/s 2024-05-03T21:24:27.495 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:24:27 smithi082 ceph-mon[32933]: pgmap v460: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 5 op/s 2024-05-03T21:24:29.324 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:24:29 smithi138 ceph-mon[33110]: pgmap v461: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:24:29.471 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:24:29 smithi067 ceph-mon[29730]: pgmap v461: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:24:29.497 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:24:29 smithi082 ceph-mon[32933]: pgmap v461: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:24:31.325 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:24:31 smithi138 ceph-mon[33110]: pgmap v462: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T21:24:31.471 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:24:31 smithi067 ceph-mon[29730]: pgmap v462: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T21:24:31.495 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:24:31 smithi082 ceph-mon[32933]: pgmap v462: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T21:24:33.324 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:24:33 smithi138 ceph-mon[33110]: pgmap v463: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:24:33.471 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:24:33 smithi067 ceph-mon[29730]: pgmap v463: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:24:33.495 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:24:33 smithi082 ceph-mon[32933]: pgmap v463: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:24:35.324 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:24:35 smithi138 ceph-mon[33110]: pgmap v464: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:24:35.471 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:24:35 smithi067 ceph-mon[29730]: pgmap v464: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:24:35.495 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:24:35 smithi082 ceph-mon[32933]: pgmap v464: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:24:37.324 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:24:37 smithi138 ceph-mon[33110]: pgmap v465: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:24:37.471 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:24:37 smithi067 ceph-mon[29730]: pgmap v465: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:24:37.495 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:24:37 smithi082 ceph-mon[32933]: pgmap v465: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:24:39.471 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:24:39 smithi067 ceph-mon[29730]: pgmap v466: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:24:39.495 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:24:39 smithi082 ceph-mon[32933]: pgmap v466: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:24:39.574 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:24:39 smithi138 ceph-mon[33110]: pgmap v466: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:24:41.471 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:24:41 smithi067 ceph-mon[29730]: pgmap v467: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T21:24:41.495 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:24:41 smithi082 ceph-mon[32933]: pgmap v467: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T21:24:41.574 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:24:41 smithi138 ceph-mon[33110]: pgmap v467: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T21:24:43.377 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:24:43 smithi138 ceph-mon[33110]: pgmap v468: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:24:43.471 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:24:43 smithi067 ceph-mon[29730]: pgmap v468: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:24:43.495 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:24:43 smithi082 ceph-mon[32933]: pgmap v468: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:24:44.432 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:24:44.430+0000 7f90c97fa640 1 -- 172.21.15.67:0/432660756 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f90bc0034c0 con 0x7f90d41045f0 2024-05-03T21:24:44.432 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:24:44.430+0000 7f90ca7fc640 1 -- 172.21.15.67:0/432660756 <== mon.0 v2:172.21.15.67:3300/0 7 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f90bc0034c0 con 0x7f90d41045f0 2024-05-03T21:24:45.471 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:24:45 smithi067 ceph-mon[29730]: pgmap v469: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:24:45.495 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:24:45 smithi082 ceph-mon[32933]: pgmap v469: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:24:45.574 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:24:45 smithi138 ceph-mon[33110]: pgmap v469: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:24:47.471 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:24:47 smithi067 ceph-mon[29730]: pgmap v470: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:24:47.495 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:24:47 smithi082 ceph-mon[32933]: pgmap v470: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:24:47.574 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:24:47 smithi138 ceph-mon[33110]: pgmap v470: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:24:49.471 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:24:49 smithi067 ceph-mon[29730]: pgmap v471: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:24:49.495 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:24:49 smithi082 ceph-mon[32933]: pgmap v471: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:24:49.574 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:24:49 smithi138 ceph-mon[33110]: pgmap v471: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:24:51.472 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:24:51 smithi067 ceph-mon[29730]: pgmap v472: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T21:24:51.495 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:24:51 smithi082 ceph-mon[32933]: pgmap v472: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T21:24:51.574 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:24:51 smithi138 ceph-mon[33110]: pgmap v472: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T21:24:53.471 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:24:53 smithi067 ceph-mon[29730]: pgmap v473: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:24:53.495 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:24:53 smithi082 ceph-mon[32933]: pgmap v473: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:24:53.574 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:24:53 smithi138 ceph-mon[33110]: pgmap v473: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:24:55.471 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:24:55 smithi067 ceph-mon[29730]: pgmap v474: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:24:55.495 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:24:55 smithi082 ceph-mon[32933]: pgmap v474: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:24:55.574 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:24:55 smithi138 ceph-mon[33110]: pgmap v474: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:24:56.471 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:24:56 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T21:24:56.495 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:24:56 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T21:24:56.574 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:24:56 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T21:24:57.471 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:24:57 smithi067 ceph-mon[29730]: pgmap v475: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:24:57.471 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:24:57 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T21:24:57.471 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:24:57 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T21:24:57.472 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:24:57 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:24:57.495 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:24:57 smithi082 ceph-mon[32933]: pgmap v475: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:24:57.496 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:24:57 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T21:24:57.496 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:24:57 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T21:24:57.496 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:24:57 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:24:57.574 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:24:57 smithi138 ceph-mon[33110]: pgmap v475: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:24:57.574 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:24:57 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T21:24:57.575 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:24:57 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T21:24:57.575 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:24:57 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:24:59.471 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:24:59 smithi067 ceph-mon[29730]: pgmap v476: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:24:59.495 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:24:59 smithi082 ceph-mon[32933]: pgmap v476: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:24:59.574 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:24:59 smithi138 ceph-mon[33110]: pgmap v476: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:25:01.471 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:25:01 smithi067 ceph-mon[29730]: pgmap v477: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T21:25:01.495 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:25:01 smithi082 ceph-mon[32933]: pgmap v477: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T21:25:01.574 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:25:01 smithi138 ceph-mon[33110]: pgmap v477: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T21:25:03.471 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:25:03 smithi067 ceph-mon[29730]: pgmap v478: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:25:03.495 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:25:03 smithi082 ceph-mon[32933]: pgmap v478: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:25:03.574 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:25:03 smithi138 ceph-mon[33110]: pgmap v478: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:25:05.471 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:25:05 smithi067 ceph-mon[29730]: pgmap v479: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:25:05.495 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:25:05 smithi082 ceph-mon[32933]: pgmap v479: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:25:05.574 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:25:05 smithi138 ceph-mon[33110]: pgmap v479: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:25:07.471 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:25:07 smithi067 ceph-mon[29730]: pgmap v480: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:25:07.495 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:25:07 smithi082 ceph-mon[32933]: pgmap v480: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:25:07.574 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:25:07 smithi138 ceph-mon[33110]: pgmap v480: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:25:09.471 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:25:09 smithi067 ceph-mon[29730]: pgmap v481: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:25:09.495 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:25:09 smithi082 ceph-mon[32933]: pgmap v481: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:25:09.574 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:25:09 smithi138 ceph-mon[33110]: pgmap v481: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:25:11.471 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:25:11 smithi067 ceph-mon[29730]: pgmap v482: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T21:25:11.495 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:25:11 smithi082 ceph-mon[32933]: pgmap v482: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T21:25:11.574 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:25:11 smithi138 ceph-mon[33110]: pgmap v482: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T21:25:13.471 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:25:13 smithi067 ceph-mon[29730]: pgmap v483: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:25:13.495 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:25:13 smithi082 ceph-mon[32933]: pgmap v483: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:25:13.574 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:25:13 smithi138 ceph-mon[33110]: pgmap v483: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:25:15.471 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:25:15 smithi067 ceph-mon[29730]: pgmap v484: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:25:15.495 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:25:15 smithi082 ceph-mon[32933]: pgmap v484: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:25:15.574 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:25:15 smithi138 ceph-mon[33110]: pgmap v484: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:25:17.471 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:25:17 smithi067 ceph-mon[29730]: pgmap v485: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:25:17.495 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:25:17 smithi082 ceph-mon[32933]: pgmap v485: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:25:17.574 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:25:17 smithi138 ceph-mon[33110]: pgmap v485: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:25:19.471 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:25:19 smithi067 ceph-mon[29730]: pgmap v486: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:25:19.495 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:25:19 smithi082 ceph-mon[32933]: pgmap v486: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:25:19.574 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:25:19 smithi138 ceph-mon[33110]: pgmap v486: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:25:21.471 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:25:21 smithi067 ceph-mon[29730]: pgmap v487: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T21:25:21.495 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:25:21 smithi082 ceph-mon[32933]: pgmap v487: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T21:25:21.574 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:25:21 smithi138 ceph-mon[33110]: pgmap v487: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T21:25:23.471 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:25:23 smithi067 ceph-mon[29730]: pgmap v488: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:25:23.495 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:25:23 smithi082 ceph-mon[32933]: pgmap v488: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:25:23.574 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:25:23 smithi138 ceph-mon[33110]: pgmap v488: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:25:25.471 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:25:25 smithi067 ceph-mon[29730]: pgmap v489: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:25:25.495 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:25:25 smithi082 ceph-mon[32933]: pgmap v489: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:25:25.574 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:25:25 smithi138 ceph-mon[33110]: pgmap v489: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:25:27.471 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:25:27 smithi067 ceph-mon[29730]: pgmap v490: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:25:27.495 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:25:27 smithi082 ceph-mon[32933]: pgmap v490: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:25:27.574 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:25:27 smithi138 ceph-mon[33110]: pgmap v490: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:25:29.471 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:25:29 smithi067 ceph-mon[29730]: pgmap v491: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:25:29.495 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:25:29 smithi082 ceph-mon[32933]: pgmap v491: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:25:29.574 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:25:29 smithi138 ceph-mon[33110]: pgmap v491: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:25:31.471 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:25:31 smithi067 ceph-mon[29730]: pgmap v492: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T21:25:31.495 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:25:31 smithi082 ceph-mon[32933]: pgmap v492: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T21:25:31.574 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:25:31 smithi138 ceph-mon[33110]: pgmap v492: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T21:25:33.471 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:25:33 smithi067 ceph-mon[29730]: pgmap v493: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:25:33.495 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:25:33 smithi082 ceph-mon[32933]: pgmap v493: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:25:33.574 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:25:33 smithi138 ceph-mon[33110]: pgmap v493: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:25:35.471 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:25:35 smithi067 ceph-mon[29730]: pgmap v494: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:25:35.495 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:25:35 smithi082 ceph-mon[32933]: pgmap v494: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:25:35.574 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:25:35 smithi138 ceph-mon[33110]: pgmap v494: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:25:37.471 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:25:37 smithi067 ceph-mon[29730]: pgmap v495: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:25:37.495 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:25:37 smithi082 ceph-mon[32933]: pgmap v495: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:25:37.574 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:25:37 smithi138 ceph-mon[33110]: pgmap v495: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:25:39.471 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:25:39 smithi067 ceph-mon[29730]: pgmap v496: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:25:39.495 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:25:39 smithi082 ceph-mon[32933]: pgmap v496: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:25:39.574 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:25:39 smithi138 ceph-mon[33110]: pgmap v496: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:25:41.471 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:25:41 smithi067 ceph-mon[29730]: pgmap v497: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T21:25:41.495 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:25:41 smithi082 ceph-mon[32933]: pgmap v497: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T21:25:41.574 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:25:41 smithi138 ceph-mon[33110]: pgmap v497: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T21:25:43.471 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:25:43 smithi067 ceph-mon[29730]: pgmap v498: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:25:43.495 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:25:43 smithi082 ceph-mon[32933]: pgmap v498: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:25:43.575 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:25:43 smithi138 ceph-mon[33110]: pgmap v498: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:25:45.471 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:25:45 smithi067 ceph-mon[29730]: pgmap v499: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:25:45.495 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:25:45 smithi082 ceph-mon[32933]: pgmap v499: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:25:45.574 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:25:45 smithi138 ceph-mon[33110]: pgmap v499: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:25:47.471 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:25:47 smithi067 ceph-mon[29730]: pgmap v500: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:25:47.495 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:25:47 smithi082 ceph-mon[32933]: pgmap v500: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:25:47.574 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:25:47 smithi138 ceph-mon[33110]: pgmap v500: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:25:49.471 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:25:49 smithi067 ceph-mon[29730]: pgmap v501: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:25:49.495 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:25:49 smithi082 ceph-mon[32933]: pgmap v501: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:25:49.574 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:25:49 smithi138 ceph-mon[33110]: pgmap v501: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:25:51.471 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:25:51 smithi067 ceph-mon[29730]: pgmap v502: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T21:25:51.495 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:25:51 smithi082 ceph-mon[32933]: pgmap v502: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T21:25:51.574 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:25:51 smithi138 ceph-mon[33110]: pgmap v502: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T21:25:53.471 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:25:53 smithi067 ceph-mon[29730]: pgmap v503: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:25:53.495 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:25:53 smithi082 ceph-mon[32933]: pgmap v503: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:25:53.574 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:25:53 smithi138 ceph-mon[33110]: pgmap v503: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:25:55.471 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:25:55 smithi067 ceph-mon[29730]: pgmap v504: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:25:55.495 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:25:55 smithi082 ceph-mon[32933]: pgmap v504: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:25:55.574 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:25:55 smithi138 ceph-mon[33110]: pgmap v504: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:25:57.471 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:25:57 smithi067 ceph-mon[29730]: pgmap v505: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:25:57.471 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:25:57 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T21:25:57.495 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:25:57 smithi082 ceph-mon[32933]: pgmap v505: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:25:57.495 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:25:57 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T21:25:57.574 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:25:57 smithi138 ceph-mon[33110]: pgmap v505: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:25:57.574 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:25:57 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T21:25:58.471 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:25:58 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T21:25:58.471 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:25:58 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T21:25:58.471 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:25:58 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:25:58.495 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:25:58 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T21:25:58.495 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:25:58 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T21:25:58.495 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:25:58 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:25:58.574 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:25:58 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T21:25:58.575 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:25:58 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T21:25:58.575 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:25:58 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:25:59.471 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:25:59 smithi067 ceph-mon[29730]: pgmap v506: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:25:59.495 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:25:59 smithi082 ceph-mon[32933]: pgmap v506: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:25:59.574 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:25:59 smithi138 ceph-mon[33110]: pgmap v506: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:26:01.471 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:26:01 smithi067 ceph-mon[29730]: pgmap v507: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T21:26:01.495 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:26:01 smithi082 ceph-mon[32933]: pgmap v507: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T21:26:01.574 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:26:01 smithi138 ceph-mon[33110]: pgmap v507: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T21:26:03.495 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:26:03 smithi082 ceph-mon[32933]: pgmap v508: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:26:03.574 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:26:03 smithi138 ceph-mon[33110]: pgmap v508: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:26:03.721 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:26:03 smithi067 ceph-mon[29730]: pgmap v508: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:26:05.495 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:26:05 smithi082 ceph-mon[32933]: pgmap v509: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:26:05.574 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:26:05 smithi138 ceph-mon[33110]: pgmap v509: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:26:05.721 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:26:05 smithi067 ceph-mon[29730]: pgmap v509: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:26:07.496 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:26:07 smithi082 ceph-mon[32933]: pgmap v510: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:26:07.574 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:26:07 smithi138 ceph-mon[33110]: pgmap v510: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:26:07.721 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:26:07 smithi067 ceph-mon[29730]: pgmap v510: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:26:09.496 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:26:09 smithi082 ceph-mon[32933]: pgmap v511: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:26:09.574 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:26:09 smithi138 ceph-mon[33110]: pgmap v511: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:26:09.721 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:26:09 smithi067 ceph-mon[29730]: pgmap v511: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:26:11.495 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:26:11 smithi082 ceph-mon[32933]: pgmap v512: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T21:26:11.574 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:26:11 smithi138 ceph-mon[33110]: pgmap v512: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T21:26:11.721 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:26:11 smithi067 ceph-mon[29730]: pgmap v512: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T21:26:13.495 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:26:13 smithi082 ceph-mon[32933]: pgmap v513: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:26:13.574 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:26:13 smithi138 ceph-mon[33110]: pgmap v513: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:26:13.721 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:26:13 smithi067 ceph-mon[29730]: pgmap v513: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:26:15.495 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:26:15 smithi082 ceph-mon[32933]: pgmap v514: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:26:15.574 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:26:15 smithi138 ceph-mon[33110]: pgmap v514: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:26:15.721 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:26:15 smithi067 ceph-mon[29730]: pgmap v514: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:26:17.495 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:26:17 smithi082 ceph-mon[32933]: pgmap v515: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:26:17.574 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:26:17 smithi138 ceph-mon[33110]: pgmap v515: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:26:17.721 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:26:17 smithi067 ceph-mon[29730]: pgmap v515: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:26:19.574 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:26:19 smithi138 ceph-mon[33110]: pgmap v516: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:26:19.721 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:26:19 smithi067 ceph-mon[29730]: pgmap v516: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:26:19.745 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:26:19 smithi082 ceph-mon[32933]: pgmap v516: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:26:21.574 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:26:21 smithi138 ceph-mon[33110]: pgmap v517: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T21:26:21.721 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:26:21 smithi067 ceph-mon[29730]: pgmap v517: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T21:26:21.745 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:26:21 smithi082 ceph-mon[32933]: pgmap v517: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T21:26:23.574 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:26:23 smithi138 ceph-mon[33110]: pgmap v518: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:26:23.721 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:26:23 smithi067 ceph-mon[29730]: pgmap v518: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:26:23.745 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:26:23 smithi082 ceph-mon[32933]: pgmap v518: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:26:25.524 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:26:25 smithi067 ceph-mon[29730]: pgmap v519: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:26:25.574 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:26:25 smithi138 ceph-mon[33110]: pgmap v519: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:26:25.745 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:26:25 smithi082 ceph-mon[32933]: pgmap v519: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:26:27.574 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:26:27 smithi138 ceph-mon[33110]: pgmap v520: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:26:27.721 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:26:27 smithi067 ceph-mon[29730]: pgmap v520: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:26:27.745 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:26:27 smithi082 ceph-mon[32933]: pgmap v520: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:26:29.574 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:26:29 smithi138 ceph-mon[33110]: pgmap v521: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:26:29.721 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:26:29 smithi067 ceph-mon[29730]: pgmap v521: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:26:29.746 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:26:29 smithi082 ceph-mon[32933]: pgmap v521: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:26:31.574 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:26:31 smithi138 ceph-mon[33110]: pgmap v522: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T21:26:31.721 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:26:31 smithi067 ceph-mon[29730]: pgmap v522: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T21:26:31.745 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:26:31 smithi082 ceph-mon[32933]: pgmap v522: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T21:26:33.574 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:26:33 smithi138 ceph-mon[33110]: pgmap v523: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:26:33.721 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:26:33 smithi067 ceph-mon[29730]: pgmap v523: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:26:33.745 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:26:33 smithi082 ceph-mon[32933]: pgmap v523: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:26:35.574 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:26:35 smithi138 ceph-mon[33110]: pgmap v524: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:26:35.721 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:26:35 smithi067 ceph-mon[29730]: pgmap v524: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:26:35.745 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:26:35 smithi082 ceph-mon[32933]: pgmap v524: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:26:37.574 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:26:37 smithi138 ceph-mon[33110]: pgmap v525: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:26:37.721 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:26:37 smithi067 ceph-mon[29730]: pgmap v525: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:26:37.745 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:26:37 smithi082 ceph-mon[32933]: pgmap v525: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:26:39.574 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:26:39 smithi138 ceph-mon[33110]: pgmap v526: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:26:39.721 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:26:39 smithi067 ceph-mon[29730]: pgmap v526: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:26:39.745 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:26:39 smithi082 ceph-mon[32933]: pgmap v526: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:26:41.574 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:26:41 smithi138 ceph-mon[33110]: pgmap v527: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T21:26:41.721 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:26:41 smithi067 ceph-mon[29730]: pgmap v527: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T21:26:41.745 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:26:41 smithi082 ceph-mon[32933]: pgmap v527: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T21:26:43.574 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:26:43 smithi138 ceph-mon[33110]: pgmap v528: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:26:43.721 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:26:43 smithi067 ceph-mon[29730]: pgmap v528: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:26:43.745 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:26:43 smithi082 ceph-mon[32933]: pgmap v528: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:26:45.548 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:26:45 smithi082 ceph-mon[32933]: pgmap v529: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:26:45.574 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:26:45 smithi138 ceph-mon[33110]: pgmap v529: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:26:45.721 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:26:45 smithi067 ceph-mon[29730]: pgmap v529: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:26:47.574 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:26:47 smithi138 ceph-mon[33110]: pgmap v530: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:26:47.721 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:26:47 smithi067 ceph-mon[29730]: pgmap v530: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:26:47.746 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:26:47 smithi082 ceph-mon[32933]: pgmap v530: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:26:49.574 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:26:49 smithi138 ceph-mon[33110]: pgmap v531: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:26:49.721 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:26:49 smithi067 ceph-mon[29730]: pgmap v531: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:26:49.745 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:26:49 smithi082 ceph-mon[32933]: pgmap v531: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:26:51.574 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:26:51 smithi138 ceph-mon[33110]: pgmap v532: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T21:26:51.721 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:26:51 smithi067 ceph-mon[29730]: pgmap v532: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T21:26:51.745 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:26:51 smithi082 ceph-mon[32933]: pgmap v532: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T21:26:53.574 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:26:53 smithi138 ceph-mon[33110]: pgmap v533: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:26:53.721 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:26:53 smithi067 ceph-mon[29730]: pgmap v533: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:26:53.745 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:26:53 smithi082 ceph-mon[32933]: pgmap v533: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:26:55.574 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:26:55 smithi138 ceph-mon[33110]: pgmap v534: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:26:55.721 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:26:55 smithi067 ceph-mon[29730]: pgmap v534: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:26:55.745 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:26:55 smithi082 ceph-mon[32933]: pgmap v534: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:26:57.574 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:26:57 smithi138 ceph-mon[33110]: pgmap v535: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:26:57.721 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:26:57 smithi067 ceph-mon[29730]: pgmap v535: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:26:57.745 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:26:57 smithi082 ceph-mon[32933]: pgmap v535: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:26:58.574 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:26:58 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T21:26:58.575 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:26:58 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-05-03T21:26:58.575 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:26:58 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-05-03T21:26:58.575 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:26:58 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-05-03T21:26:58.575 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:26:58 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-05-03T21:26:58.575 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:26:58 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T21:26:58.575 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:26:58 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T21:26:58.575 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:26:58 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:26:58.721 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:26:58 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T21:26:58.721 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:26:58 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-05-03T21:26:58.721 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:26:58 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-05-03T21:26:58.721 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:26:58 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-05-03T21:26:58.721 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:26:58 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-05-03T21:26:58.722 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:26:58 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T21:26:58.722 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:26:58 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T21:26:58.722 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:26:58 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:26:58.745 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:26:58 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T21:26:58.746 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:26:58 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-05-03T21:26:58.746 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:26:58 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-05-03T21:26:58.746 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:26:58 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-05-03T21:26:58.746 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:26:58 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-05-03T21:26:58.746 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:26:58 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T21:26:58.746 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:26:58 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T21:26:58.746 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:26:58 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:26:59.574 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:26:59 smithi138 ceph-mon[33110]: pgmap v536: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:26:59.575 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:26:59 smithi138 ceph-mon[33110]: Adjusting osd_memory_target on smithi067 to 176.2M 2024-05-03T21:26:59.575 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:26:59 smithi138 ceph-mon[33110]: Unable to set osd_memory_target on smithi067 to 184792473: error parsing value: Value '184792473' is below minimum 939524096 2024-05-03T21:26:59.721 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:26:59 smithi067 ceph-mon[29730]: pgmap v536: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:26:59.721 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:26:59 smithi067 ceph-mon[29730]: Adjusting osd_memory_target on smithi067 to 176.2M 2024-05-03T21:26:59.721 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:26:59 smithi067 ceph-mon[29730]: Unable to set osd_memory_target on smithi067 to 184792473: error parsing value: Value '184792473' is below minimum 939524096 2024-05-03T21:26:59.745 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:26:59 smithi082 ceph-mon[32933]: pgmap v536: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:26:59.745 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:26:59 smithi082 ceph-mon[32933]: Adjusting osd_memory_target on smithi067 to 176.2M 2024-05-03T21:26:59.746 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:26:59 smithi082 ceph-mon[32933]: Unable to set osd_memory_target on smithi067 to 184792473: error parsing value: Value '184792473' is below minimum 939524096 2024-05-03T21:27:01.574 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:27:01 smithi138 ceph-mon[33110]: pgmap v537: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T21:27:01.721 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:27:01 smithi067 ceph-mon[29730]: pgmap v537: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T21:27:01.745 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:27:01 smithi082 ceph-mon[32933]: pgmap v537: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T21:27:03.574 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:27:03 smithi138 ceph-mon[33110]: pgmap v538: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:27:03.721 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:27:03 smithi067 ceph-mon[29730]: pgmap v538: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:27:03.745 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:27:03 smithi082 ceph-mon[32933]: pgmap v538: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:27:05.574 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:27:05 smithi138 ceph-mon[33110]: pgmap v539: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:27:05.721 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:27:05 smithi067 ceph-mon[29730]: pgmap v539: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:27:05.745 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:27:05 smithi082 ceph-mon[32933]: pgmap v539: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:27:07.577 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:27:07 smithi138 ceph-mon[33110]: pgmap v540: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:27:07.721 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:27:07 smithi067 ceph-mon[29730]: pgmap v540: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:27:07.745 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:27:07 smithi082 ceph-mon[32933]: pgmap v540: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:27:09.721 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:27:09 smithi067 ceph-mon[29730]: pgmap v541: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:27:09.745 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:27:09 smithi082 ceph-mon[32933]: pgmap v541: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:27:09.824 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:27:09 smithi138 ceph-mon[33110]: pgmap v541: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:27:11.721 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:27:11 smithi067 ceph-mon[29730]: pgmap v542: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T21:27:11.745 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:27:11 smithi082 ceph-mon[32933]: pgmap v542: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T21:27:11.824 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:27:11 smithi138 ceph-mon[33110]: pgmap v542: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T21:27:13.627 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:27:13 smithi138 ceph-mon[33110]: pgmap v543: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:27:13.721 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:27:13 smithi067 ceph-mon[29730]: pgmap v543: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:27:13.745 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:27:13 smithi082 ceph-mon[32933]: pgmap v543: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:27:15.721 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:27:15 smithi067 ceph-mon[29730]: pgmap v544: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:27:15.745 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:27:15 smithi082 ceph-mon[32933]: pgmap v544: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:27:15.824 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:27:15 smithi138 ceph-mon[33110]: pgmap v544: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:27:17.721 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:27:17 smithi067 ceph-mon[29730]: pgmap v545: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:27:17.745 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:27:17 smithi082 ceph-mon[32933]: pgmap v545: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:27:17.824 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:27:17 smithi138 ceph-mon[33110]: pgmap v545: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:27:19.721 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:27:19 smithi067 ceph-mon[29730]: pgmap v546: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:27:19.745 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:27:19 smithi082 ceph-mon[32933]: pgmap v546: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:27:19.824 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:27:19 smithi138 ceph-mon[33110]: pgmap v546: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:27:21.721 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:27:21 smithi067 ceph-mon[29730]: pgmap v547: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T21:27:21.745 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:27:21 smithi082 ceph-mon[32933]: pgmap v547: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T21:27:21.824 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:27:21 smithi138 ceph-mon[33110]: pgmap v547: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T21:27:23.721 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:27:23 smithi067 ceph-mon[29730]: pgmap v548: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:27:23.745 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:27:23 smithi082 ceph-mon[32933]: pgmap v548: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:27:23.824 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:27:23 smithi138 ceph-mon[33110]: pgmap v548: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:27:25.721 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:27:25 smithi067 ceph-mon[29730]: pgmap v549: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:27:25.745 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:27:25 smithi082 ceph-mon[32933]: pgmap v549: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:27:25.824 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:27:25 smithi138 ceph-mon[33110]: pgmap v549: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:27:27.721 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:27:27 smithi067 ceph-mon[29730]: pgmap v550: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:27:27.745 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:27:27 smithi082 ceph-mon[32933]: pgmap v550: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:27:27.824 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:27:27 smithi138 ceph-mon[33110]: pgmap v550: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:27:29.721 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:27:29 smithi067 ceph-mon[29730]: pgmap v551: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:27:29.745 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:27:29 smithi082 ceph-mon[32933]: pgmap v551: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:27:29.824 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:27:29 smithi138 ceph-mon[33110]: pgmap v551: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:27:31.721 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:27:31 smithi067 ceph-mon[29730]: pgmap v552: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T21:27:31.745 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:27:31 smithi082 ceph-mon[32933]: pgmap v552: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T21:27:31.824 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:27:31 smithi138 ceph-mon[33110]: pgmap v552: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T21:27:33.721 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:27:33 smithi067 ceph-mon[29730]: pgmap v553: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:27:33.745 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:27:33 smithi082 ceph-mon[32933]: pgmap v553: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:27:33.824 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:27:33 smithi138 ceph-mon[33110]: pgmap v553: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:27:35.721 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:27:35 smithi067 ceph-mon[29730]: pgmap v554: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:27:35.745 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:27:35 smithi082 ceph-mon[32933]: pgmap v554: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:27:35.824 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:27:35 smithi138 ceph-mon[33110]: pgmap v554: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:27:37.721 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:27:37 smithi067 ceph-mon[29730]: pgmap v555: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:27:37.745 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:27:37 smithi082 ceph-mon[32933]: pgmap v555: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:27:37.824 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:27:37 smithi138 ceph-mon[33110]: pgmap v555: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:27:39.721 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:27:39 smithi067 ceph-mon[29730]: pgmap v556: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:27:39.745 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:27:39 smithi082 ceph-mon[32933]: pgmap v556: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:27:39.824 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:27:39 smithi138 ceph-mon[33110]: pgmap v556: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:27:41.721 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:27:41 smithi067 ceph-mon[29730]: pgmap v557: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T21:27:41.745 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:27:41 smithi082 ceph-mon[32933]: pgmap v557: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T21:27:41.824 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:27:41 smithi138 ceph-mon[33110]: pgmap v557: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T21:27:43.721 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:27:43 smithi067 ceph-mon[29730]: pgmap v558: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:27:43.745 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:27:43 smithi082 ceph-mon[32933]: pgmap v558: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:27:43.824 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:27:43 smithi138 ceph-mon[33110]: pgmap v558: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:27:44.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:27:44.456+0000 7f90c97fa640 1 -- 172.21.15.67:0/432660756 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f90bc005850 con 0x7f90d41045f0 2024-05-03T21:27:44.458 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:27:44.457+0000 7f90ca7fc640 1 -- 172.21.15.67:0/432660756 <== mon.0 v2:172.21.15.67:3300/0 8 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f90bc005850 con 0x7f90d41045f0 2024-05-03T21:27:45.721 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:27:45 smithi067 ceph-mon[29730]: pgmap v559: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:27:45.745 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:27:45 smithi082 ceph-mon[32933]: pgmap v559: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:27:45.824 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:27:45 smithi138 ceph-mon[33110]: pgmap v559: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:27:47.721 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:27:47 smithi067 ceph-mon[29730]: pgmap v560: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:27:47.745 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:27:47 smithi082 ceph-mon[32933]: pgmap v560: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:27:47.824 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:27:47 smithi138 ceph-mon[33110]: pgmap v560: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:27:49.721 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:27:49 smithi067 ceph-mon[29730]: pgmap v561: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:27:49.745 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:27:49 smithi082 ceph-mon[32933]: pgmap v561: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:27:49.824 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:27:49 smithi138 ceph-mon[33110]: pgmap v561: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:27:51.721 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:27:51 smithi067 ceph-mon[29730]: pgmap v562: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T21:27:51.745 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:27:51 smithi082 ceph-mon[32933]: pgmap v562: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T21:27:51.824 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:27:51 smithi138 ceph-mon[33110]: pgmap v562: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T21:27:53.721 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:27:53 smithi067 ceph-mon[29730]: pgmap v563: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:27:53.745 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:27:53 smithi082 ceph-mon[32933]: pgmap v563: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:27:53.824 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:27:53 smithi138 ceph-mon[33110]: pgmap v563: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:27:55.721 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:27:55 smithi067 ceph-mon[29730]: pgmap v564: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:27:55.745 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:27:55 smithi082 ceph-mon[32933]: pgmap v564: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:27:55.824 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:27:55 smithi138 ceph-mon[33110]: pgmap v564: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:27:57.721 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:27:57 smithi067 ceph-mon[29730]: pgmap v565: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:27:57.745 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:27:57 smithi082 ceph-mon[32933]: pgmap v565: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:27:57.824 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:27:57 smithi138 ceph-mon[33110]: pgmap v565: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:27:58.721 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:27:58 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T21:27:58.745 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:27:58 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T21:27:58.824 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:27:58 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T21:27:59.721 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:27:59 smithi067 ceph-mon[29730]: pgmap v566: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:27:59.721 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:27:59 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-05-03T21:27:59.721 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:27:59 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-05-03T21:27:59.721 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:27:59 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-05-03T21:27:59.722 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:27:59 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-05-03T21:27:59.722 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:27:59 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-05-03T21:27:59.722 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:27:59 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-05-03T21:27:59.722 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:27:59 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-05-03T21:27:59.722 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:27:59 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-05-03T21:27:59.722 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:27:59 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T21:27:59.722 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:27:59 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T21:27:59.722 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:27:59 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:27:59.745 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:27:59 smithi082 ceph-mon[32933]: pgmap v566: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:27:59.745 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:27:59 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-05-03T21:27:59.745 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:27:59 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-05-03T21:27:59.745 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:27:59 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-05-03T21:27:59.746 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:27:59 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-05-03T21:27:59.746 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:27:59 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-05-03T21:27:59.746 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:27:59 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-05-03T21:27:59.746 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:27:59 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-05-03T21:27:59.746 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:27:59 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-05-03T21:27:59.746 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:27:59 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T21:27:59.746 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:27:59 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T21:27:59.746 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:27:59 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:27:59.824 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:27:59 smithi138 ceph-mon[33110]: pgmap v566: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:27:59.825 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:27:59 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-05-03T21:27:59.825 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:27:59 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-05-03T21:27:59.825 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:27:59 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-05-03T21:27:59.825 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:27:59 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-05-03T21:27:59.825 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:27:59 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-05-03T21:27:59.825 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:27:59 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-05-03T21:27:59.825 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:27:59 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-05-03T21:27:59.825 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:27:59 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-05-03T21:27:59.825 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:27:59 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T21:27:59.825 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:27:59 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T21:27:59.825 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:27:59 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:28:00.721 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:28:00 smithi067 ceph-mon[29730]: Adjusting osd_memory_target on smithi138 to 176.2M 2024-05-03T21:28:00.721 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:28:00 smithi067 ceph-mon[29730]: Adjusting osd_memory_target on smithi082 to 176.2M 2024-05-03T21:28:00.721 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:28:00 smithi067 ceph-mon[29730]: Unable to set osd_memory_target on smithi138 to 184770969: error parsing value: Value '184770969' is below minimum 939524096 2024-05-03T21:28:00.721 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:28:00 smithi067 ceph-mon[29730]: Unable to set osd_memory_target on smithi082 to 184763801: error parsing value: Value '184763801' is below minimum 939524096 2024-05-03T21:28:00.745 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:28:00 smithi082 ceph-mon[32933]: Adjusting osd_memory_target on smithi138 to 176.2M 2024-05-03T21:28:00.745 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:28:00 smithi082 ceph-mon[32933]: Adjusting osd_memory_target on smithi082 to 176.2M 2024-05-03T21:28:00.745 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:28:00 smithi082 ceph-mon[32933]: Unable to set osd_memory_target on smithi138 to 184770969: error parsing value: Value '184770969' is below minimum 939524096 2024-05-03T21:28:00.745 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:28:00 smithi082 ceph-mon[32933]: Unable to set osd_memory_target on smithi082 to 184763801: error parsing value: Value '184763801' is below minimum 939524096 2024-05-03T21:28:00.824 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:28:00 smithi138 ceph-mon[33110]: Adjusting osd_memory_target on smithi138 to 176.2M 2024-05-03T21:28:00.824 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:28:00 smithi138 ceph-mon[33110]: Adjusting osd_memory_target on smithi082 to 176.2M 2024-05-03T21:28:00.825 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:28:00 smithi138 ceph-mon[33110]: Unable to set osd_memory_target on smithi138 to 184770969: error parsing value: Value '184770969' is below minimum 939524096 2024-05-03T21:28:00.825 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:28:00 smithi138 ceph-mon[33110]: Unable to set osd_memory_target on smithi082 to 184763801: error parsing value: Value '184763801' is below minimum 939524096 2024-05-03T21:28:01.721 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:28:01 smithi067 ceph-mon[29730]: pgmap v567: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T21:28:01.745 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:28:01 smithi082 ceph-mon[32933]: pgmap v567: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T21:28:01.824 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:28:01 smithi138 ceph-mon[33110]: pgmap v567: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T21:28:03.721 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:28:03 smithi067 ceph-mon[29730]: pgmap v568: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:28:03.745 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:28:03 smithi082 ceph-mon[32933]: pgmap v568: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:28:03.824 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:28:03 smithi138 ceph-mon[33110]: pgmap v568: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:28:05.721 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:28:05 smithi067 ceph-mon[29730]: pgmap v569: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:28:05.745 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:28:05 smithi082 ceph-mon[32933]: pgmap v569: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:28:05.824 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:28:05 smithi138 ceph-mon[33110]: pgmap v569: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:28:07.721 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:28:07 smithi067 ceph-mon[29730]: pgmap v570: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:28:07.745 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:28:07 smithi082 ceph-mon[32933]: pgmap v570: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:28:07.824 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:28:07 smithi138 ceph-mon[33110]: pgmap v570: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:28:09.721 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:28:09 smithi067 ceph-mon[29730]: pgmap v571: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:28:09.745 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:28:09 smithi082 ceph-mon[32933]: pgmap v571: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:28:09.824 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:28:09 smithi138 ceph-mon[33110]: pgmap v571: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:28:11.721 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:28:11 smithi067 ceph-mon[29730]: pgmap v572: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T21:28:11.745 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:28:11 smithi082 ceph-mon[32933]: pgmap v572: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T21:28:11.824 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:28:11 smithi138 ceph-mon[33110]: pgmap v572: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T21:28:13.721 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:28:13 smithi067 ceph-mon[29730]: pgmap v573: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:28:13.745 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:28:13 smithi082 ceph-mon[32933]: pgmap v573: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:28:13.824 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:28:13 smithi138 ceph-mon[33110]: pgmap v573: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:28:15.721 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:28:15 smithi067 ceph-mon[29730]: pgmap v574: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:28:15.745 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:28:15 smithi082 ceph-mon[32933]: pgmap v574: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:28:15.824 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:28:15 smithi138 ceph-mon[33110]: pgmap v574: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:28:17.721 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:28:17 smithi067 ceph-mon[29730]: pgmap v575: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:28:17.745 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:28:17 smithi082 ceph-mon[32933]: pgmap v575: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:28:17.824 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:28:17 smithi138 ceph-mon[33110]: pgmap v575: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:28:19.721 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:28:19 smithi067 ceph-mon[29730]: pgmap v576: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:28:19.745 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:28:19 smithi082 ceph-mon[32933]: pgmap v576: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:28:19.824 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:28:19 smithi138 ceph-mon[33110]: pgmap v576: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:28:21.721 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:28:21 smithi067 ceph-mon[29730]: pgmap v577: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T21:28:21.745 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:28:21 smithi082 ceph-mon[32933]: pgmap v577: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T21:28:21.824 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:28:21 smithi138 ceph-mon[33110]: pgmap v577: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T21:28:23.721 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:28:23 smithi067 ceph-mon[29730]: pgmap v578: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:28:23.745 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:28:23 smithi082 ceph-mon[32933]: pgmap v578: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:28:23.824 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:28:23 smithi138 ceph-mon[33110]: pgmap v578: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:28:25.721 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:28:25 smithi067 ceph-mon[29730]: pgmap v579: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:28:25.745 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:28:25 smithi082 ceph-mon[32933]: pgmap v579: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:28:25.824 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:28:25 smithi138 ceph-mon[33110]: pgmap v579: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:28:27.721 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:28:27 smithi067 ceph-mon[29730]: pgmap v580: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:28:27.745 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:28:27 smithi082 ceph-mon[32933]: pgmap v580: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:28:27.824 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:28:27 smithi138 ceph-mon[33110]: pgmap v580: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:28:29.721 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:28:29 smithi067 ceph-mon[29730]: pgmap v581: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:28:29.745 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:28:29 smithi082 ceph-mon[32933]: pgmap v581: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:28:29.824 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:28:29 smithi138 ceph-mon[33110]: pgmap v581: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:28:31.721 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:28:31 smithi067 ceph-mon[29730]: pgmap v582: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T21:28:31.745 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:28:31 smithi082 ceph-mon[32933]: pgmap v582: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T21:28:31.824 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:28:31 smithi138 ceph-mon[33110]: pgmap v582: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T21:28:33.721 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:28:33 smithi067 ceph-mon[29730]: pgmap v583: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:28:33.745 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:28:33 smithi082 ceph-mon[32933]: pgmap v583: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:28:33.824 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:28:33 smithi138 ceph-mon[33110]: pgmap v583: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:28:35.721 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:28:35 smithi067 ceph-mon[29730]: pgmap v584: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:28:35.745 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:28:35 smithi082 ceph-mon[32933]: pgmap v584: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:28:35.824 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:28:35 smithi138 ceph-mon[33110]: pgmap v584: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:28:36.824 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:28:36 smithi138 ceph-mon[33110]: pgmap v585: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:28:36.971 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:28:36 smithi067 ceph-mon[29730]: pgmap v585: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:28:36.995 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:28:36 smithi082 ceph-mon[32933]: pgmap v585: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:28:39.324 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:28:39 smithi138 ceph-mon[33110]: pgmap v586: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:28:39.471 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:28:39 smithi067 ceph-mon[29730]: pgmap v586: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:28:39.495 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:28:39 smithi082 ceph-mon[32933]: pgmap v586: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:28:41.324 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:28:41 smithi138 ceph-mon[33110]: pgmap v587: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T21:28:41.471 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:28:41 smithi067 ceph-mon[29730]: pgmap v587: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T21:28:41.495 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:28:41 smithi082 ceph-mon[32933]: pgmap v587: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T21:28:43.324 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:28:43 smithi138 ceph-mon[33110]: pgmap v588: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:28:43.471 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:28:43 smithi067 ceph-mon[29730]: pgmap v588: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:28:43.495 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:28:43 smithi082 ceph-mon[32933]: pgmap v588: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:28:45.298 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:28:45 smithi082 ceph-mon[32933]: pgmap v589: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:28:45.324 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:28:45 smithi138 ceph-mon[33110]: pgmap v589: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:28:45.471 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:28:45 smithi067 ceph-mon[29730]: pgmap v589: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:28:47.324 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:28:47 smithi138 ceph-mon[33110]: pgmap v590: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:28:47.471 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:28:47 smithi067 ceph-mon[29730]: pgmap v590: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:28:47.495 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:28:47 smithi082 ceph-mon[32933]: pgmap v590: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:28:49.324 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:28:49 smithi138 ceph-mon[33110]: pgmap v591: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:28:49.471 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:28:49 smithi067 ceph-mon[29730]: pgmap v591: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:28:49.495 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:28:49 smithi082 ceph-mon[32933]: pgmap v591: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:28:51.495 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:28:51 smithi082 ceph-mon[32933]: pgmap v592: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T21:28:51.574 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:28:51 smithi138 ceph-mon[33110]: pgmap v592: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T21:28:51.721 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:28:51 smithi067 ceph-mon[29730]: pgmap v592: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T21:28:53.495 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:28:53 smithi082 ceph-mon[32933]: pgmap v593: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:28:53.574 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:28:53 smithi138 ceph-mon[33110]: pgmap v593: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:28:53.721 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:28:53 smithi067 ceph-mon[29730]: pgmap v593: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:28:55.523 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:28:55 smithi067 ceph-mon[29730]: pgmap v594: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:28:55.574 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:28:55 smithi138 ceph-mon[33110]: pgmap v594: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:28:55.745 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:28:55 smithi082 ceph-mon[32933]: pgmap v594: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:28:57.574 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:28:57 smithi138 ceph-mon[33110]: pgmap v595: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:28:57.721 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:28:57 smithi067 ceph-mon[29730]: pgmap v595: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:28:57.745 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:28:57 smithi082 ceph-mon[32933]: pgmap v595: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:28:59.574 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:28:59 smithi138 ceph-mon[33110]: pgmap v596: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:28:59.574 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:28:59 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T21:28:59.721 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:28:59 smithi067 ceph-mon[29730]: pgmap v596: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:28:59.721 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:28:59 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T21:28:59.745 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:28:59 smithi082 ceph-mon[32933]: pgmap v596: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:28:59.745 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:28:59 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T21:29:00.574 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:29:00 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T21:29:00.574 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:29:00 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T21:29:00.574 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:29:00 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:29:00.721 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:29:00 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T21:29:00.721 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:29:00 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T21:29:00.721 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:29:00 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:29:00.745 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:29:00 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T21:29:00.745 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:29:00 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T21:29:00.745 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:29:00 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:29:01.574 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:29:01 smithi138 ceph-mon[33110]: pgmap v597: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T21:29:01.721 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:29:01 smithi067 ceph-mon[29730]: pgmap v597: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T21:29:01.745 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:29:01 smithi082 ceph-mon[32933]: pgmap v597: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T21:29:03.574 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:29:03 smithi138 ceph-mon[33110]: pgmap v598: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:29:03.721 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:29:03 smithi067 ceph-mon[29730]: pgmap v598: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:29:03.745 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:29:03 smithi082 ceph-mon[32933]: pgmap v598: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:29:05.574 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:29:05 smithi138 ceph-mon[33110]: pgmap v599: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:29:05.721 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:29:05 smithi067 ceph-mon[29730]: pgmap v599: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:29:05.745 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:29:05 smithi082 ceph-mon[32933]: pgmap v599: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:29:07.574 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:29:07 smithi138 ceph-mon[33110]: pgmap v600: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:29:07.721 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:29:07 smithi067 ceph-mon[29730]: pgmap v600: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:29:07.745 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:29:07 smithi082 ceph-mon[32933]: pgmap v600: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:29:09.574 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:29:09 smithi138 ceph-mon[33110]: pgmap v601: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:29:09.721 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:29:09 smithi067 ceph-mon[29730]: pgmap v601: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:29:09.745 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:29:09 smithi082 ceph-mon[32933]: pgmap v601: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:29:11.574 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:29:11 smithi138 ceph-mon[33110]: pgmap v602: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T21:29:11.721 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:29:11 smithi067 ceph-mon[29730]: pgmap v602: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T21:29:11.745 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:29:11 smithi082 ceph-mon[32933]: pgmap v602: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T21:29:13.574 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:29:13 smithi138 ceph-mon[33110]: pgmap v603: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:29:13.721 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:29:13 smithi067 ceph-mon[29730]: pgmap v603: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:29:13.745 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:29:13 smithi082 ceph-mon[32933]: pgmap v603: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:29:15.547 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:29:15 smithi082 ceph-mon[32933]: pgmap v604: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:29:15.574 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:29:15 smithi138 ceph-mon[33110]: pgmap v604: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:29:15.721 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:29:15 smithi067 ceph-mon[29730]: pgmap v604: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:29:17.574 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:29:17 smithi138 ceph-mon[33110]: pgmap v605: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:29:17.721 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:29:17 smithi067 ceph-mon[29730]: pgmap v605: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:29:17.745 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:29:17 smithi082 ceph-mon[32933]: pgmap v605: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:29:19.574 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:29:19 smithi138 ceph-mon[33110]: pgmap v606: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:29:19.721 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:29:19 smithi067 ceph-mon[29730]: pgmap v606: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:29:19.745 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:29:19 smithi082 ceph-mon[32933]: pgmap v606: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:29:21.574 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:29:21 smithi138 ceph-mon[33110]: pgmap v607: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T21:29:21.721 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:29:21 smithi067 ceph-mon[29730]: pgmap v607: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T21:29:21.745 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:29:21 smithi082 ceph-mon[32933]: pgmap v607: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T21:29:23.574 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:29:23 smithi138 ceph-mon[33110]: pgmap v608: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:29:23.721 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:29:23 smithi067 ceph-mon[29730]: pgmap v608: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:29:23.745 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:29:23 smithi082 ceph-mon[32933]: pgmap v608: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:29:25.574 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:29:25 smithi138 ceph-mon[33110]: pgmap v609: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:29:25.721 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:29:25 smithi067 ceph-mon[29730]: pgmap v609: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:29:25.745 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:29:25 smithi082 ceph-mon[32933]: pgmap v609: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:29:27.574 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:29:27 smithi138 ceph-mon[33110]: pgmap v610: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:29:27.721 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:29:27 smithi067 ceph-mon[29730]: pgmap v610: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:29:27.745 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:29:27 smithi082 ceph-mon[32933]: pgmap v610: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:29:29.574 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:29:29 smithi138 ceph-mon[33110]: pgmap v611: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:29:29.721 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:29:29 smithi067 ceph-mon[29730]: pgmap v611: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:29:29.745 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:29:29 smithi082 ceph-mon[32933]: pgmap v611: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:29:31.574 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:29:31 smithi138 ceph-mon[33110]: pgmap v612: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T21:29:31.721 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:29:31 smithi067 ceph-mon[29730]: pgmap v612: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T21:29:31.745 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:29:31 smithi082 ceph-mon[32933]: pgmap v612: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T21:29:33.574 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:29:33 smithi138 ceph-mon[33110]: pgmap v613: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:29:33.721 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:29:33 smithi067 ceph-mon[29730]: pgmap v613: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:29:33.745 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:29:33 smithi082 ceph-mon[32933]: pgmap v613: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:29:35.574 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:29:35 smithi138 ceph-mon[33110]: pgmap v614: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:29:35.721 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:29:35 smithi067 ceph-mon[29730]: pgmap v614: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:29:35.745 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:29:35 smithi082 ceph-mon[32933]: pgmap v614: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:29:37.574 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:29:37 smithi138 ceph-mon[33110]: pgmap v615: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:29:37.721 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:29:37 smithi067 ceph-mon[29730]: pgmap v615: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:29:37.745 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:29:37 smithi082 ceph-mon[32933]: pgmap v615: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:29:39.575 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:29:39 smithi138 ceph-mon[33110]: pgmap v616: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:29:39.721 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:29:39 smithi067 ceph-mon[29730]: pgmap v616: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:29:39.745 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:29:39 smithi082 ceph-mon[32933]: pgmap v616: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:29:41.574 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:29:41 smithi138 ceph-mon[33110]: pgmap v617: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T21:29:41.720 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:29:41 smithi067 ceph-mon[29730]: pgmap v617: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T21:29:41.746 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:29:41 smithi082 ceph-mon[32933]: pgmap v617: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T21:29:43.574 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:29:43 smithi138 ceph-mon[33110]: pgmap v618: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:29:43.721 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:29:43 smithi067 ceph-mon[29730]: pgmap v618: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:29:43.745 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:29:43 smithi082 ceph-mon[32933]: pgmap v618: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:29:45.574 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:29:45 smithi138 ceph-mon[33110]: pgmap v619: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:29:45.721 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:29:45 smithi067 ceph-mon[29730]: pgmap v619: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:29:45.745 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:29:45 smithi082 ceph-mon[32933]: pgmap v619: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:29:47.721 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:29:47 smithi067 ceph-mon[29730]: pgmap v620: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:29:47.745 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:29:47 smithi082 ceph-mon[32933]: pgmap v620: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:29:47.824 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:29:47 smithi138 ceph-mon[33110]: pgmap v620: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:29:49.721 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:29:49 smithi067 ceph-mon[29730]: pgmap v621: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:29:49.745 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:29:49 smithi082 ceph-mon[32933]: pgmap v621: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:29:49.824 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:29:49 smithi138 ceph-mon[33110]: pgmap v621: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:29:51.721 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:29:51 smithi067 ceph-mon[29730]: pgmap v622: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T21:29:51.745 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:29:51 smithi082 ceph-mon[32933]: pgmap v622: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T21:29:51.824 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:29:51 smithi138 ceph-mon[33110]: pgmap v622: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T21:29:53.721 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:29:53 smithi067 ceph-mon[29730]: pgmap v623: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:29:53.745 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:29:53 smithi082 ceph-mon[32933]: pgmap v623: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:29:53.824 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:29:53 smithi138 ceph-mon[33110]: pgmap v623: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:29:55.721 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:29:55 smithi067 ceph-mon[29730]: pgmap v624: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:29:55.745 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:29:55 smithi082 ceph-mon[32933]: pgmap v624: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:29:55.824 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:29:55 smithi138 ceph-mon[33110]: pgmap v624: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:29:57.720 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:29:57 smithi067 ceph-mon[29730]: pgmap v625: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:29:57.745 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:29:57 smithi082 ceph-mon[32933]: pgmap v625: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:29:57.824 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:29:57 smithi138 ceph-mon[33110]: pgmap v625: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:29:59.477 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:29:59.476+0000 7f90c97fa640 1 -- 172.21.15.67:0/432660756 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f90bc005b90 con 0x7f90d41045f0 2024-05-03T21:29:59.477 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:29:59.476+0000 7f90ca7fc640 1 -- 172.21.15.67:0/432660756 <== mon.0 v2:172.21.15.67:3300/0 9 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7f90bc005b90 con 0x7f90d41045f0 2024-05-03T21:29:59.721 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:29:59 smithi067 ceph-mon[29730]: pgmap v626: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:29:59.745 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:29:59 smithi082 ceph-mon[32933]: pgmap v626: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:29:59.824 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:29:59 smithi138 ceph-mon[33110]: pgmap v626: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:30:00.720 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:30:00 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T21:30:00.721 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:30:00 smithi067 ceph-mon[29730]: overall HEALTH_OK 2024-05-03T21:30:00.745 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:30:00 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T21:30:00.745 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:30:00 smithi082 ceph-mon[32933]: overall HEALTH_OK 2024-05-03T21:30:00.824 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:30:00 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T21:30:00.824 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:30:00 smithi138 ceph-mon[33110]: overall HEALTH_OK 2024-05-03T21:30:01.721 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:30:01 smithi067 ceph-mon[29730]: pgmap v627: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T21:30:01.721 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:30:01 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T21:30:01.721 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:30:01 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T21:30:01.721 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:30:01 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:30:01.745 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:30:01 smithi082 ceph-mon[32933]: pgmap v627: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T21:30:01.745 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:30:01 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T21:30:01.745 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:30:01 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T21:30:01.745 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:30:01 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:30:01.824 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:30:01 smithi138 ceph-mon[33110]: pgmap v627: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T21:30:01.824 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:30:01 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T21:30:01.824 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:30:01 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T21:30:01.824 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:30:01 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:30:03.721 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:30:03 smithi067 ceph-mon[29730]: pgmap v628: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:30:03.745 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:30:03 smithi082 ceph-mon[32933]: pgmap v628: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:30:03.824 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:30:03 smithi138 ceph-mon[33110]: pgmap v628: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:30:05.721 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:30:05 smithi067 ceph-mon[29730]: pgmap v629: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:30:05.745 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:30:05 smithi082 ceph-mon[32933]: pgmap v629: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:30:05.824 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:30:05 smithi138 ceph-mon[33110]: pgmap v629: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:30:07.721 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:30:07 smithi067 ceph-mon[29730]: pgmap v630: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:30:07.745 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:30:07 smithi082 ceph-mon[32933]: pgmap v630: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:30:07.824 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:30:07 smithi138 ceph-mon[33110]: pgmap v630: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:30:09.720 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:30:09 smithi067 ceph-mon[29730]: pgmap v631: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:30:09.745 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:30:09 smithi082 ceph-mon[32933]: pgmap v631: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:30:09.824 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:30:09 smithi138 ceph-mon[33110]: pgmap v631: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:30:11.721 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:30:11 smithi067 ceph-mon[29730]: pgmap v632: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T21:30:11.745 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:30:11 smithi082 ceph-mon[32933]: pgmap v632: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T21:30:11.824 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:30:11 smithi138 ceph-mon[33110]: pgmap v632: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T21:30:13.626 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:30:13 smithi138 ceph-mon[33110]: pgmap v633: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:30:13.721 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:30:13 smithi067 ceph-mon[29730]: pgmap v633: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:30:13.745 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:30:13 smithi082 ceph-mon[32933]: pgmap v633: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:30:15.721 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:30:15 smithi067 ceph-mon[29730]: pgmap v634: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:30:15.745 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:30:15 smithi082 ceph-mon[32933]: pgmap v634: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:30:15.824 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:30:15 smithi138 ceph-mon[33110]: pgmap v634: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:30:17.721 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:30:17 smithi067 ceph-mon[29730]: pgmap v635: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:30:17.745 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:30:17 smithi082 ceph-mon[32933]: pgmap v635: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:30:17.824 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:30:17 smithi138 ceph-mon[33110]: pgmap v635: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:30:19.721 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:30:19 smithi067 ceph-mon[29730]: pgmap v636: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:30:19.745 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:30:19 smithi082 ceph-mon[32933]: pgmap v636: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:30:19.824 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:30:19 smithi138 ceph-mon[33110]: pgmap v636: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:30:21.721 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:30:21 smithi067 ceph-mon[29730]: pgmap v637: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T21:30:21.745 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:30:21 smithi082 ceph-mon[32933]: pgmap v637: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T21:30:21.824 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:30:21 smithi138 ceph-mon[33110]: pgmap v637: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T21:30:23.721 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:30:23 smithi067 ceph-mon[29730]: pgmap v638: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:30:23.745 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:30:23 smithi082 ceph-mon[32933]: pgmap v638: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:30:23.824 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:30:23 smithi138 ceph-mon[33110]: pgmap v638: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:30:25.721 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:30:25 smithi067 ceph-mon[29730]: pgmap v639: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:30:25.745 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:30:25 smithi082 ceph-mon[32933]: pgmap v639: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:30:25.824 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:30:25 smithi138 ceph-mon[33110]: pgmap v639: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:30:27.721 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:30:27 smithi067 ceph-mon[29730]: pgmap v640: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:30:27.745 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:30:27 smithi082 ceph-mon[32933]: pgmap v640: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:30:27.824 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:30:27 smithi138 ceph-mon[33110]: pgmap v640: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:30:29.721 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:30:29 smithi067 ceph-mon[29730]: pgmap v641: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:30:29.745 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:30:29 smithi082 ceph-mon[32933]: pgmap v641: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:30:29.824 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:30:29 smithi138 ceph-mon[33110]: pgmap v641: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:30:31.721 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:30:31 smithi067 ceph-mon[29730]: pgmap v642: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T21:30:31.745 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:30:31 smithi082 ceph-mon[32933]: pgmap v642: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T21:30:31.824 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:30:31 smithi138 ceph-mon[33110]: pgmap v642: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T21:30:33.721 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:30:33 smithi067 ceph-mon[29730]: pgmap v643: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:30:33.745 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:30:33 smithi082 ceph-mon[32933]: pgmap v643: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:30:33.826 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:30:33 smithi138 ceph-mon[33110]: pgmap v643: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:30:35.721 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:30:35 smithi067 ceph-mon[29730]: pgmap v644: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 6 op/s 2024-05-03T21:30:35.745 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:30:35 smithi082 ceph-mon[32933]: pgmap v644: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 6 op/s 2024-05-03T21:30:35.824 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:30:35 smithi138 ceph-mon[33110]: pgmap v644: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 6 op/s 2024-05-03T21:30:37.721 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:30:37 smithi067 ceph-mon[29730]: pgmap v645: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-05-03T21:30:37.745 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:30:37 smithi082 ceph-mon[32933]: pgmap v645: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-05-03T21:30:37.824 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:30:37 smithi138 ceph-mon[33110]: pgmap v645: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-05-03T21:30:39.721 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:30:39 smithi067 ceph-mon[29730]: pgmap v646: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 6 op/s 2024-05-03T21:30:39.745 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:30:39 smithi082 ceph-mon[32933]: pgmap v646: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 6 op/s 2024-05-03T21:30:39.824 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:30:39 smithi138 ceph-mon[33110]: pgmap v646: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 6 op/s 2024-05-03T21:30:41.720 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:30:41 smithi067 ceph-mon[29730]: pgmap v647: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-05-03T21:30:41.745 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:30:41 smithi082 ceph-mon[32933]: pgmap v647: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-05-03T21:30:41.824 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:30:41 smithi138 ceph-mon[33110]: pgmap v647: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-05-03T21:30:43.721 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:30:43 smithi067 ceph-mon[29730]: pgmap v648: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-05-03T21:30:43.745 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:30:43 smithi082 ceph-mon[32933]: pgmap v648: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-05-03T21:30:43.824 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:30:43 smithi138 ceph-mon[33110]: pgmap v648: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-05-03T21:30:44.484 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:30:44.483+0000 7f90c97fa640 1 -- 172.21.15.67:0/432660756 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f90bc005ef0 con 0x7f90d41045f0 2024-05-03T21:30:44.484 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:30:44.483+0000 7f90ca7fc640 1 -- 172.21.15.67:0/432660756 <== mon.0 v2:172.21.15.67:3300/0 10 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f90bc005ef0 con 0x7f90d41045f0 2024-05-03T21:30:45.721 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:30:45 smithi067 ceph-mon[29730]: pgmap v649: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 6 op/s 2024-05-03T21:30:45.745 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:30:45 smithi082 ceph-mon[32933]: pgmap v649: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 6 op/s 2024-05-03T21:30:45.824 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:30:45 smithi138 ceph-mon[33110]: pgmap v649: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 6 op/s 2024-05-03T21:30:47.721 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:30:47 smithi067 ceph-mon[29730]: pgmap v650: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:30:47.745 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:30:47 smithi082 ceph-mon[32933]: pgmap v650: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:30:47.824 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:30:47 smithi138 ceph-mon[33110]: pgmap v650: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:30:49.721 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:30:49 smithi067 ceph-mon[29730]: pgmap v651: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:30:49.745 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:30:49 smithi082 ceph-mon[32933]: pgmap v651: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:30:49.824 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:30:49 smithi138 ceph-mon[33110]: pgmap v651: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:30:51.721 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:30:51 smithi067 ceph-mon[29730]: pgmap v652: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T21:30:51.745 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:30:51 smithi082 ceph-mon[32933]: pgmap v652: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T21:30:51.824 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:30:51 smithi138 ceph-mon[33110]: pgmap v652: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T21:30:53.721 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:30:53 smithi067 ceph-mon[29730]: pgmap v653: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:30:53.745 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:30:53 smithi082 ceph-mon[32933]: pgmap v653: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:30:53.824 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:30:53 smithi138 ceph-mon[33110]: pgmap v653: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:30:55.721 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:30:55 smithi067 ceph-mon[29730]: pgmap v654: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:30:55.745 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:30:55 smithi082 ceph-mon[32933]: pgmap v654: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:30:55.824 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:30:55 smithi138 ceph-mon[33110]: pgmap v654: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:30:57.722 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:30:57 smithi067 ceph-mon[29730]: pgmap v655: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:30:57.745 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:30:57 smithi082 ceph-mon[32933]: pgmap v655: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:30:57.824 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:30:57 smithi138 ceph-mon[33110]: pgmap v655: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:30:59.721 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:30:59 smithi067 ceph-mon[29730]: pgmap v656: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:30:59.745 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:30:59 smithi082 ceph-mon[32933]: pgmap v656: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:30:59.824 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:30:59 smithi138 ceph-mon[33110]: pgmap v656: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:31:01.721 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:31:01 smithi067 ceph-mon[29730]: pgmap v657: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T21:31:01.721 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:31:01 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T21:31:01.737 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:31:01 smithi138 ceph-mon[33110]: pgmap v657: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T21:31:01.737 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:31:01 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T21:31:01.745 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:31:01 smithi082 ceph-mon[32933]: pgmap v657: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T21:31:01.745 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:31:01 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T21:31:03.574 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:31:03 smithi138 ceph-mon[33110]: pgmap v658: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:31:03.574 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:31:03 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:31:03.574 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:31:03 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:31:03.574 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:31:03 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:31:03.575 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:31:03 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:31:03.575 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:31:03 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:31:03.575 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:31:03 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:31:03.575 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:31:03 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T21:31:03.575 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:31:03 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T21:31:03.575 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:31:03 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:31:03.721 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:31:03 smithi067 ceph-mon[29730]: pgmap v658: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:31:03.721 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:31:03 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:31:03.721 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:31:03 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:31:03.721 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:31:03 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:31:03.721 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:31:03 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:31:03.721 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:31:03 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:31:03.721 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:31:03 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:31:03.722 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:31:03 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T21:31:03.722 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:31:03 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T21:31:03.722 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:31:03 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:31:03.745 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:31:03 smithi082 ceph-mon[32933]: pgmap v658: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:31:03.745 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:31:03 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:31:03.745 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:31:03 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:31:03.745 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:31:03 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:31:03.746 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:31:03 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:31:03.746 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:31:03 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:31:03.746 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:31:03 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:31:03.746 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:31:03 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T21:31:03.746 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:31:03 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T21:31:03.746 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:31:03 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:31:05.574 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:31:05 smithi138 ceph-mon[33110]: pgmap v659: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:31:05.721 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:31:05 smithi067 ceph-mon[29730]: pgmap v659: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:31:05.745 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:31:05 smithi082 ceph-mon[32933]: pgmap v659: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:31:07.574 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:31:07 smithi138 ceph-mon[33110]: pgmap v660: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:31:07.720 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:31:07 smithi067 ceph-mon[29730]: pgmap v660: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:31:07.745 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:31:07 smithi082 ceph-mon[32933]: pgmap v660: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:31:09.574 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:31:09 smithi138 ceph-mon[33110]: pgmap v661: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:31:09.720 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:31:09 smithi067 ceph-mon[29730]: pgmap v661: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:31:09.745 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:31:09 smithi082 ceph-mon[32933]: pgmap v661: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:31:11.574 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:31:11 smithi138 ceph-mon[33110]: pgmap v662: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T21:31:11.721 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:31:11 smithi067 ceph-mon[29730]: pgmap v662: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T21:31:11.745 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:31:11 smithi082 ceph-mon[32933]: pgmap v662: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T21:31:13.574 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:31:13 smithi138 ceph-mon[33110]: pgmap v663: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:31:13.720 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:31:13 smithi067 ceph-mon[29730]: pgmap v663: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:31:13.745 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:31:13 smithi082 ceph-mon[32933]: pgmap v663: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:31:15.574 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:31:15 smithi138 ceph-mon[33110]: pgmap v664: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:31:15.720 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:31:15 smithi067 ceph-mon[29730]: pgmap v664: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:31:15.745 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:31:15 smithi082 ceph-mon[32933]: pgmap v664: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:31:17.574 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:31:17 smithi138 ceph-mon[33110]: pgmap v665: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:31:17.720 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:31:17 smithi067 ceph-mon[29730]: pgmap v665: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:31:17.745 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:31:17 smithi082 ceph-mon[32933]: pgmap v665: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:31:19.574 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:31:19 smithi138 ceph-mon[33110]: pgmap v666: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:31:19.721 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:31:19 smithi067 ceph-mon[29730]: pgmap v666: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:31:19.813 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:31:19 smithi082 ceph-mon[32933]: pgmap v666: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:31:21.574 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:31:21 smithi138 ceph-mon[33110]: pgmap v667: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T21:31:21.721 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:31:21 smithi067 ceph-mon[29730]: pgmap v667: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T21:31:21.745 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:31:21 smithi082 ceph-mon[32933]: pgmap v667: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T21:31:23.721 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:31:23 smithi067 ceph-mon[29730]: pgmap v668: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:31:23.745 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:31:23 smithi082 ceph-mon[32933]: pgmap v668: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:31:23.824 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:31:23 smithi138 ceph-mon[33110]: pgmap v668: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:31:25.720 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:31:25 smithi067 ceph-mon[29730]: pgmap v669: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:31:25.745 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:31:25 smithi082 ceph-mon[32933]: pgmap v669: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:31:25.824 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:31:25 smithi138 ceph-mon[33110]: pgmap v669: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:31:27.720 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:31:27 smithi067 ceph-mon[29730]: pgmap v670: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:31:27.745 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:31:27 smithi082 ceph-mon[32933]: pgmap v670: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:31:27.824 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:31:27 smithi138 ceph-mon[33110]: pgmap v670: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:31:29.720 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:31:29 smithi067 ceph-mon[29730]: pgmap v671: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:31:29.745 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:31:29 smithi082 ceph-mon[32933]: pgmap v671: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:31:29.824 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:31:29 smithi138 ceph-mon[33110]: pgmap v671: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:31:31.721 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:31:31 smithi067 ceph-mon[29730]: pgmap v672: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T21:31:31.745 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:31:31 smithi082 ceph-mon[32933]: pgmap v672: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T21:31:31.824 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:31:31 smithi138 ceph-mon[33110]: pgmap v672: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T21:31:33.721 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:31:33 smithi067 ceph-mon[29730]: pgmap v673: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:31:33.745 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:31:33 smithi082 ceph-mon[32933]: pgmap v673: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:31:33.824 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:31:33 smithi138 ceph-mon[33110]: pgmap v673: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:31:35.721 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:31:35 smithi067 ceph-mon[29730]: pgmap v674: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:31:35.745 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:31:35 smithi082 ceph-mon[32933]: pgmap v674: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:31:35.824 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:31:35 smithi138 ceph-mon[33110]: pgmap v674: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:31:37.721 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:31:37 smithi067 ceph-mon[29730]: pgmap v675: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:31:37.745 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:31:37 smithi082 ceph-mon[32933]: pgmap v675: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:31:37.824 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:31:37 smithi138 ceph-mon[33110]: pgmap v675: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:31:39.721 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:31:39 smithi067 ceph-mon[29730]: pgmap v676: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:31:39.745 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:31:39 smithi082 ceph-mon[32933]: pgmap v676: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:31:39.824 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:31:39 smithi138 ceph-mon[33110]: pgmap v676: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:31:41.721 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:31:41 smithi067 ceph-mon[29730]: pgmap v677: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T21:31:41.745 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:31:41 smithi082 ceph-mon[32933]: pgmap v677: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T21:31:41.824 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:31:41 smithi138 ceph-mon[33110]: pgmap v677: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T21:31:43.626 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:31:43 smithi138 ceph-mon[33110]: pgmap v678: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:31:43.720 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:31:43 smithi067 ceph-mon[29730]: pgmap v678: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:31:43.745 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:31:43 smithi082 ceph-mon[32933]: pgmap v678: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:31:45.720 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:31:45 smithi067 ceph-mon[29730]: pgmap v679: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:31:45.745 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:31:45 smithi082 ceph-mon[32933]: pgmap v679: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:31:45.824 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:31:45 smithi138 ceph-mon[33110]: pgmap v679: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:31:47.721 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:31:47 smithi067 ceph-mon[29730]: pgmap v680: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:31:47.745 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:31:47 smithi082 ceph-mon[32933]: pgmap v680: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:31:47.824 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:31:47 smithi138 ceph-mon[33110]: pgmap v680: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:31:49.721 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:31:49 smithi067 ceph-mon[29730]: pgmap v681: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:31:49.745 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:31:49 smithi082 ceph-mon[32933]: pgmap v681: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:31:49.824 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:31:49 smithi138 ceph-mon[33110]: pgmap v681: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:31:51.720 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:31:51 smithi067 ceph-mon[29730]: pgmap v682: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T21:31:51.745 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:31:51 smithi082 ceph-mon[32933]: pgmap v682: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T21:31:51.824 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:31:51 smithi138 ceph-mon[33110]: pgmap v682: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T21:31:53.721 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:31:53 smithi067 ceph-mon[29730]: pgmap v683: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:31:53.745 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:31:53 smithi082 ceph-mon[32933]: pgmap v683: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:31:53.824 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:31:53 smithi138 ceph-mon[33110]: pgmap v683: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:31:55.721 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:31:55 smithi067 ceph-mon[29730]: pgmap v684: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:31:55.745 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:31:55 smithi082 ceph-mon[32933]: pgmap v684: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:31:55.824 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:31:55 smithi138 ceph-mon[33110]: pgmap v684: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:31:57.720 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:31:57 smithi067 ceph-mon[29730]: pgmap v685: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:31:57.745 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:31:57 smithi082 ceph-mon[32933]: pgmap v685: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:31:57.824 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:31:57 smithi138 ceph-mon[33110]: pgmap v685: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:31:59.720 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:31:59 smithi067 ceph-mon[29730]: pgmap v686: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:31:59.745 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:31:59 smithi082 ceph-mon[32933]: pgmap v686: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:31:59.824 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:31:59 smithi138 ceph-mon[33110]: pgmap v686: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:32:01.720 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:32:01 smithi067 ceph-mon[29730]: pgmap v687: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T21:32:01.745 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:32:01 smithi082 ceph-mon[32933]: pgmap v687: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T21:32:01.824 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:32:01 smithi138 ceph-mon[33110]: pgmap v687: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T21:32:03.720 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:32:03 smithi067 ceph-mon[29730]: pgmap v688: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:32:03.721 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:32:03 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T21:32:03.745 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:32:03 smithi082 ceph-mon[32933]: pgmap v688: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:32:03.745 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:32:03 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T21:32:03.824 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:32:03 smithi138 ceph-mon[33110]: pgmap v688: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:32:03.824 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:32:03 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T21:32:05.221 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:32:04 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:32:05.221 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:32:04 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:32:05.221 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:32:04 smithi067 ceph-mon[29730]: pgmap v689: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:32:05.221 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:32:04 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T21:32:05.221 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:32:04 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T21:32:05.221 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:32:04 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:32:05.245 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:32:04 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:32:05.245 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:32:04 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:32:05.245 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:32:04 smithi082 ceph-mon[32933]: pgmap v689: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:32:05.245 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:32:04 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T21:32:05.245 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:32:04 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T21:32:05.245 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:32:04 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:32:05.324 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:32:04 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:32:05.324 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:32:04 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:32:05.324 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:32:04 smithi138 ceph-mon[33110]: pgmap v689: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:32:05.324 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:32:04 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T21:32:05.324 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:32:04 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T21:32:05.324 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:32:04 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:32:07.470 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:32:07 smithi067 ceph-mon[29730]: pgmap v690: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:32:07.495 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:32:07 smithi082 ceph-mon[32933]: pgmap v690: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:32:07.574 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:32:07 smithi138 ceph-mon[33110]: pgmap v690: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:32:09.470 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:32:09 smithi067 ceph-mon[29730]: pgmap v691: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:32:09.495 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:32:09 smithi082 ceph-mon[32933]: pgmap v691: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:32:09.574 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:32:09 smithi138 ceph-mon[33110]: pgmap v691: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:32:11.471 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:32:11 smithi067 ceph-mon[29730]: pgmap v692: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T21:32:11.495 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:32:11 smithi082 ceph-mon[32933]: pgmap v692: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T21:32:11.574 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:32:11 smithi138 ceph-mon[33110]: pgmap v692: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T21:32:13.471 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:32:13 smithi067 ceph-mon[29730]: pgmap v693: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:32:13.495 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:32:13 smithi082 ceph-mon[32933]: pgmap v693: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:32:13.574 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:32:13 smithi138 ceph-mon[33110]: pgmap v693: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:32:15.471 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:32:15 smithi067 ceph-mon[29730]: pgmap v694: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:32:15.495 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:32:15 smithi082 ceph-mon[32933]: pgmap v694: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:32:15.574 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:32:15 smithi138 ceph-mon[33110]: pgmap v694: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:32:17.470 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:32:17 smithi067 ceph-mon[29730]: pgmap v695: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:32:17.495 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:32:17 smithi082 ceph-mon[32933]: pgmap v695: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:32:17.574 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:32:17 smithi138 ceph-mon[33110]: pgmap v695: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:32:19.471 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:32:19 smithi067 ceph-mon[29730]: pgmap v696: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:32:19.495 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:32:19 smithi082 ceph-mon[32933]: pgmap v696: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:32:19.574 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:32:19 smithi138 ceph-mon[33110]: pgmap v696: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:32:21.470 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:32:21 smithi067 ceph-mon[29730]: pgmap v697: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T21:32:21.495 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:32:21 smithi082 ceph-mon[32933]: pgmap v697: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T21:32:21.574 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:32:21 smithi138 ceph-mon[33110]: pgmap v697: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T21:32:23.471 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:32:23 smithi067 ceph-mon[29730]: pgmap v698: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:32:23.495 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:32:23 smithi082 ceph-mon[32933]: pgmap v698: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:32:23.574 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:32:23 smithi138 ceph-mon[33110]: pgmap v698: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:32:25.470 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:32:25 smithi067 ceph-mon[29730]: pgmap v699: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:32:25.495 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:32:25 smithi082 ceph-mon[32933]: pgmap v699: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:32:25.574 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:32:25 smithi138 ceph-mon[33110]: pgmap v699: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:32:27.471 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:32:27 smithi067 ceph-mon[29730]: pgmap v700: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:32:27.495 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:32:27 smithi082 ceph-mon[32933]: pgmap v700: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:32:27.574 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:32:27 smithi138 ceph-mon[33110]: pgmap v700: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:32:29.471 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:32:29 smithi067 ceph-mon[29730]: pgmap v701: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:32:29.495 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:32:29 smithi082 ceph-mon[32933]: pgmap v701: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:32:29.574 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:32:29 smithi138 ceph-mon[33110]: pgmap v701: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:32:31.470 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:32:31 smithi067 ceph-mon[29730]: pgmap v702: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T21:32:31.495 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:32:31 smithi082 ceph-mon[32933]: pgmap v702: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T21:32:31.574 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:32:31 smithi138 ceph-mon[33110]: pgmap v702: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T21:32:33.471 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:32:33 smithi067 ceph-mon[29730]: pgmap v703: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:32:33.495 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:32:33 smithi082 ceph-mon[32933]: pgmap v703: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:32:33.574 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:32:33 smithi138 ceph-mon[33110]: pgmap v703: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:32:35.574 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:32:35 smithi138 ceph-mon[33110]: pgmap v704: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:32:35.720 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:32:35 smithi067 ceph-mon[29730]: pgmap v704: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:32:35.745 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:32:35 smithi082 ceph-mon[32933]: pgmap v704: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:32:37.574 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:32:37 smithi138 ceph-mon[33110]: pgmap v705: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:32:37.623 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:32:37 smithi067 ceph-mon[29730]: pgmap v705: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:32:37.745 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:32:37 smithi082 ceph-mon[32933]: pgmap v705: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:32:39.574 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:32:39 smithi138 ceph-mon[33110]: pgmap v706: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:32:39.721 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:32:39 smithi067 ceph-mon[29730]: pgmap v706: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:32:39.745 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:32:39 smithi082 ceph-mon[32933]: pgmap v706: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:32:41.574 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:32:41 smithi138 ceph-mon[33110]: pgmap v707: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T21:32:41.720 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:32:41 smithi067 ceph-mon[29730]: pgmap v707: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T21:32:41.745 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:32:41 smithi082 ceph-mon[32933]: pgmap v707: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T21:32:43.574 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:32:43 smithi138 ceph-mon[33110]: pgmap v708: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:32:43.720 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:32:43 smithi067 ceph-mon[29730]: pgmap v708: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:32:43.745 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:32:43 smithi082 ceph-mon[32933]: pgmap v708: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:32:45.574 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:32:45 smithi138 ceph-mon[33110]: pgmap v709: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:32:45.720 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:32:45 smithi067 ceph-mon[29730]: pgmap v709: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:32:45.745 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:32:45 smithi082 ceph-mon[32933]: pgmap v709: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:32:47.574 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:32:47 smithi138 ceph-mon[33110]: pgmap v710: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:32:47.720 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:32:47 smithi067 ceph-mon[29730]: pgmap v710: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:32:47.745 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:32:47 smithi082 ceph-mon[32933]: pgmap v710: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:32:49.574 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:32:49 smithi138 ceph-mon[33110]: pgmap v711: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:32:49.720 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:32:49 smithi067 ceph-mon[29730]: pgmap v711: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:32:49.745 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:32:49 smithi082 ceph-mon[32933]: pgmap v711: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:32:51.574 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:32:51 smithi138 ceph-mon[33110]: pgmap v712: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T21:32:51.720 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:32:51 smithi067 ceph-mon[29730]: pgmap v712: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T21:32:51.745 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:32:51 smithi082 ceph-mon[32933]: pgmap v712: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T21:32:53.574 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:32:53 smithi138 ceph-mon[33110]: pgmap v713: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:32:53.720 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:32:53 smithi067 ceph-mon[29730]: pgmap v713: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:32:53.745 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:32:53 smithi082 ceph-mon[32933]: pgmap v713: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:32:55.574 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:32:55 smithi138 ceph-mon[33110]: pgmap v714: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:32:55.720 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:32:55 smithi067 ceph-mon[29730]: pgmap v714: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:32:55.745 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:32:55 smithi082 ceph-mon[32933]: pgmap v714: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:32:57.720 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:32:57 smithi067 ceph-mon[29730]: pgmap v715: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:32:57.745 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:32:57 smithi082 ceph-mon[32933]: pgmap v715: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:32:57.824 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:32:57 smithi138 ceph-mon[33110]: pgmap v715: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:32:59.720 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:32:59 smithi067 ceph-mon[29730]: pgmap v716: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:32:59.745 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:32:59 smithi082 ceph-mon[32933]: pgmap v716: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:32:59.824 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:32:59 smithi138 ceph-mon[33110]: pgmap v716: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:33:01.720 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:33:01 smithi067 ceph-mon[29730]: pgmap v717: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T21:33:01.745 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:33:01 smithi082 ceph-mon[32933]: pgmap v717: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T21:33:01.824 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:33:01 smithi138 ceph-mon[33110]: pgmap v717: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T21:33:03.720 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:33:03 smithi067 ceph-mon[29730]: pgmap v718: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:33:03.745 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:33:03 smithi082 ceph-mon[32933]: pgmap v718: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:33:03.824 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:33:03 smithi138 ceph-mon[33110]: pgmap v718: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:33:05.721 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:33:05 smithi067 ceph-mon[29730]: pgmap v719: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:33:05.721 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:33:05 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T21:33:05.721 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:33:05 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:33:05.721 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:33:05 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:33:05.721 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:33:05 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:33:05.721 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:33:05 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:33:05.745 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:33:05 smithi082 ceph-mon[32933]: pgmap v719: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:33:05.745 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:33:05 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T21:33:05.745 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:33:05 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:33:05.745 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:33:05 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:33:05.745 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:33:05 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:33:05.745 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:33:05 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:33:05.824 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:33:05 smithi138 ceph-mon[33110]: pgmap v719: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:33:05.824 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:33:05 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T21:33:05.824 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:33:05 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:33:05.824 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:33:05 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:33:05.824 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:33:05 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:33:05.824 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:33:05 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:33:06.720 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:33:06 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T21:33:06.720 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:33:06 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T21:33:06.721 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:33:06 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:33:06.745 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:33:06 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T21:33:06.745 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:33:06 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T21:33:06.745 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:33:06 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:33:06.824 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:33:06 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T21:33:06.824 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:33:06 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T21:33:06.824 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:33:06 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:33:07.721 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:33:07 smithi067 ceph-mon[29730]: pgmap v720: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:33:07.745 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:33:07 smithi082 ceph-mon[32933]: pgmap v720: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:33:07.824 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:33:07 smithi138 ceph-mon[33110]: pgmap v720: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:33:09.720 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:33:09 smithi067 ceph-mon[29730]: pgmap v721: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:33:09.745 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:33:09 smithi082 ceph-mon[32933]: pgmap v721: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:33:09.824 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:33:09 smithi138 ceph-mon[33110]: pgmap v721: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:33:11.720 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:33:11 smithi067 ceph-mon[29730]: pgmap v722: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T21:33:11.745 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:33:11 smithi082 ceph-mon[32933]: pgmap v722: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T21:33:11.824 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:33:11 smithi138 ceph-mon[33110]: pgmap v722: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T21:33:13.626 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:33:13 smithi138 ceph-mon[33110]: pgmap v723: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:33:13.720 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:33:13 smithi067 ceph-mon[29730]: pgmap v723: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:33:13.745 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:33:13 smithi082 ceph-mon[32933]: pgmap v723: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:33:15.720 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:33:15 smithi067 ceph-mon[29730]: pgmap v724: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:33:15.745 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:33:15 smithi082 ceph-mon[32933]: pgmap v724: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:33:15.824 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:33:15 smithi138 ceph-mon[33110]: pgmap v724: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:33:17.720 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:33:17 smithi067 ceph-mon[29730]: pgmap v725: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:33:17.745 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:33:17 smithi082 ceph-mon[32933]: pgmap v725: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:33:17.824 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:33:17 smithi138 ceph-mon[33110]: pgmap v725: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:33:19.720 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:33:19 smithi067 ceph-mon[29730]: pgmap v726: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:33:19.745 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:33:19 smithi082 ceph-mon[32933]: pgmap v726: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:33:19.824 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:33:19 smithi138 ceph-mon[33110]: pgmap v726: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:33:21.720 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:33:21 smithi067 ceph-mon[29730]: pgmap v727: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T21:33:21.745 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:33:21 smithi082 ceph-mon[32933]: pgmap v727: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T21:33:21.824 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:33:21 smithi138 ceph-mon[33110]: pgmap v727: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T21:33:23.720 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:33:23 smithi067 ceph-mon[29730]: pgmap v728: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:33:23.745 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:33:23 smithi082 ceph-mon[32933]: pgmap v728: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:33:23.824 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:33:23 smithi138 ceph-mon[33110]: pgmap v728: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:33:25.721 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:33:25 smithi067 ceph-mon[29730]: pgmap v729: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:33:25.745 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:33:25 smithi082 ceph-mon[32933]: pgmap v729: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:33:25.824 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:33:25 smithi138 ceph-mon[33110]: pgmap v729: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:33:27.720 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:33:27 smithi067 ceph-mon[29730]: pgmap v730: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:33:27.745 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:33:27 smithi082 ceph-mon[32933]: pgmap v730: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:33:27.824 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:33:27 smithi138 ceph-mon[33110]: pgmap v730: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:33:29.720 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:33:29 smithi067 ceph-mon[29730]: pgmap v731: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:33:29.745 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:33:29 smithi082 ceph-mon[32933]: pgmap v731: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:33:29.824 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:33:29 smithi138 ceph-mon[33110]: pgmap v731: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:33:31.720 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:33:31 smithi067 ceph-mon[29730]: pgmap v732: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T21:33:31.745 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:33:31 smithi082 ceph-mon[32933]: pgmap v732: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T21:33:31.824 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:33:31 smithi138 ceph-mon[33110]: pgmap v732: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T21:33:33.720 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:33:33 smithi067 ceph-mon[29730]: pgmap v733: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:33:33.745 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:33:33 smithi082 ceph-mon[32933]: pgmap v733: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:33:33.824 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:33:33 smithi138 ceph-mon[33110]: pgmap v733: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:33:35.720 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:33:35 smithi067 ceph-mon[29730]: pgmap v734: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:33:35.745 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:33:35 smithi082 ceph-mon[32933]: pgmap v734: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:33:35.824 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:33:35 smithi138 ceph-mon[33110]: pgmap v734: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:33:37.720 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:33:37 smithi067 ceph-mon[29730]: pgmap v735: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:33:37.745 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:33:37 smithi082 ceph-mon[32933]: pgmap v735: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:33:37.824 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:33:37 smithi138 ceph-mon[33110]: pgmap v735: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:33:39.720 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:33:39 smithi067 ceph-mon[29730]: pgmap v736: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:33:39.745 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:33:39 smithi082 ceph-mon[32933]: pgmap v736: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:33:39.824 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:33:39 smithi138 ceph-mon[33110]: pgmap v736: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:33:41.720 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:33:41 smithi067 ceph-mon[29730]: pgmap v737: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T21:33:41.745 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:33:41 smithi082 ceph-mon[32933]: pgmap v737: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T21:33:41.824 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:33:41 smithi138 ceph-mon[33110]: pgmap v737: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T21:33:43.721 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:33:43 smithi067 ceph-mon[29730]: pgmap v738: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:33:43.745 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:33:43 smithi082 ceph-mon[32933]: pgmap v738: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:33:43.824 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:33:43 smithi138 ceph-mon[33110]: pgmap v738: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:33:44.509 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:33:44.508+0000 7f90c97fa640 1 -- 172.21.15.67:0/432660756 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f90bc006110 con 0x7f90d41045f0 2024-05-03T21:33:44.509 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:33:44.509+0000 7f90ca7fc640 1 -- 172.21.15.67:0/432660756 <== mon.0 v2:172.21.15.67:3300/0 11 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f90bc006110 con 0x7f90d41045f0 2024-05-03T21:33:45.720 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:33:45 smithi067 ceph-mon[29730]: pgmap v739: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:33:45.745 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:33:45 smithi082 ceph-mon[32933]: pgmap v739: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:33:45.824 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:33:45 smithi138 ceph-mon[33110]: pgmap v739: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:33:47.720 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:33:47 smithi067 ceph-mon[29730]: pgmap v740: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:33:47.745 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:33:47 smithi082 ceph-mon[32933]: pgmap v740: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:33:47.824 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:33:47 smithi138 ceph-mon[33110]: pgmap v740: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:33:49.720 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:33:49 smithi067 ceph-mon[29730]: pgmap v741: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:33:49.745 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:33:49 smithi082 ceph-mon[32933]: pgmap v741: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:33:49.824 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:33:49 smithi138 ceph-mon[33110]: pgmap v741: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:33:51.720 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:33:51 smithi067 ceph-mon[29730]: pgmap v742: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T21:33:51.745 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:33:51 smithi082 ceph-mon[32933]: pgmap v742: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T21:33:51.824 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:33:51 smithi138 ceph-mon[33110]: pgmap v742: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T21:33:53.720 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:33:53 smithi067 ceph-mon[29730]: pgmap v743: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:33:53.745 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:33:53 smithi082 ceph-mon[32933]: pgmap v743: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:33:53.824 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:33:53 smithi138 ceph-mon[33110]: pgmap v743: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:33:55.720 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:33:55 smithi067 ceph-mon[29730]: pgmap v744: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:33:55.745 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:33:55 smithi082 ceph-mon[32933]: pgmap v744: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:33:55.824 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:33:55 smithi138 ceph-mon[33110]: pgmap v744: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:33:57.720 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:33:57 smithi067 ceph-mon[29730]: pgmap v745: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:33:57.745 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:33:57 smithi082 ceph-mon[32933]: pgmap v745: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:33:57.824 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:33:57 smithi138 ceph-mon[33110]: pgmap v745: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:33:59.720 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:33:59 smithi067 ceph-mon[29730]: pgmap v746: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:33:59.745 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:33:59 smithi082 ceph-mon[32933]: pgmap v746: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:33:59.824 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:33:59 smithi138 ceph-mon[33110]: pgmap v746: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:34:01.720 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:34:01 smithi067 ceph-mon[29730]: pgmap v747: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T21:34:01.745 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:34:01 smithi082 ceph-mon[32933]: pgmap v747: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T21:34:01.824 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:34:01 smithi138 ceph-mon[33110]: pgmap v747: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T21:34:03.721 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:34:03 smithi067 ceph-mon[29730]: pgmap v748: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:34:03.745 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:34:03 smithi082 ceph-mon[32933]: pgmap v748: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:34:03.824 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:34:03 smithi138 ceph-mon[33110]: pgmap v748: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:34:05.720 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:34:05 smithi067 ceph-mon[29730]: pgmap v749: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:34:05.745 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:34:05 smithi082 ceph-mon[32933]: pgmap v749: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:34:05.824 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:34:05 smithi138 ceph-mon[33110]: pgmap v749: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:34:06.720 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:34:06 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T21:34:06.744 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:34:06 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T21:34:06.824 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:34:06 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T21:34:07.721 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:34:07 smithi067 ceph-mon[29730]: pgmap v750: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:34:07.721 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:34:07 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T21:34:07.721 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:34:07 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T21:34:07.721 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:34:07 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:34:07.745 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:34:07 smithi082 ceph-mon[32933]: pgmap v750: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:34:07.745 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:34:07 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T21:34:07.745 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:34:07 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T21:34:07.745 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:34:07 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:34:07.824 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:34:07 smithi138 ceph-mon[33110]: pgmap v750: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:34:07.824 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:34:07 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T21:34:07.824 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:34:07 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T21:34:07.824 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:34:07 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:34:09.720 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:34:09 smithi067 ceph-mon[29730]: pgmap v751: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:34:09.745 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:34:09 smithi082 ceph-mon[32933]: pgmap v751: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:34:09.824 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:34:09 smithi138 ceph-mon[33110]: pgmap v751: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:34:11.720 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:34:11 smithi067 ceph-mon[29730]: pgmap v752: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T21:34:11.745 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:34:11 smithi082 ceph-mon[32933]: pgmap v752: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T21:34:11.824 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:34:11 smithi138 ceph-mon[33110]: pgmap v752: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T21:34:13.721 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:34:13 smithi067 ceph-mon[29730]: pgmap v753: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:34:13.745 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:34:13 smithi082 ceph-mon[32933]: pgmap v753: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:34:13.824 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:34:13 smithi138 ceph-mon[33110]: pgmap v753: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:34:15.720 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:34:15 smithi067 ceph-mon[29730]: pgmap v754: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:34:15.745 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:34:15 smithi082 ceph-mon[32933]: pgmap v754: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:34:15.824 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:34:15 smithi138 ceph-mon[33110]: pgmap v754: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:34:17.721 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:34:17 smithi067 ceph-mon[29730]: pgmap v755: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:34:17.745 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:34:17 smithi082 ceph-mon[32933]: pgmap v755: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:34:17.824 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:34:17 smithi138 ceph-mon[33110]: pgmap v755: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:34:19.720 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:34:19 smithi067 ceph-mon[29730]: pgmap v756: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:34:19.745 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:34:19 smithi082 ceph-mon[32933]: pgmap v756: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:34:19.824 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:34:19 smithi138 ceph-mon[33110]: pgmap v756: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:34:21.720 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:34:21 smithi067 ceph-mon[29730]: pgmap v757: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T21:34:21.745 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:34:21 smithi082 ceph-mon[32933]: pgmap v757: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T21:34:21.824 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:34:21 smithi138 ceph-mon[33110]: pgmap v757: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T21:34:23.745 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:34:23 smithi082 ceph-mon[32933]: pgmap v758: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:34:23.824 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:34:23 smithi138 ceph-mon[33110]: pgmap v758: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:34:23.970 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:34:23 smithi067 ceph-mon[29730]: pgmap v758: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:34:25.745 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:34:25 smithi082 ceph-mon[32933]: pgmap v759: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:34:25.773 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:34:25 smithi067 ceph-mon[29730]: pgmap v759: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:34:25.824 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:34:25 smithi138 ceph-mon[33110]: pgmap v759: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:34:27.745 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:34:27 smithi082 ceph-mon[32933]: pgmap v760: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:34:27.824 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:34:27 smithi138 ceph-mon[33110]: pgmap v760: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:34:27.970 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:34:27 smithi067 ceph-mon[29730]: pgmap v760: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:34:29.745 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:34:29 smithi082 ceph-mon[32933]: pgmap v761: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:34:29.824 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:34:29 smithi138 ceph-mon[33110]: pgmap v761: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:34:29.970 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:34:29 smithi067 ceph-mon[29730]: pgmap v761: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:34:31.745 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:34:31 smithi082 ceph-mon[32933]: pgmap v762: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T21:34:31.824 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:34:31 smithi138 ceph-mon[33110]: pgmap v762: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T21:34:31.970 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:34:31 smithi067 ceph-mon[29730]: pgmap v762: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T21:34:33.745 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:34:33 smithi082 ceph-mon[32933]: pgmap v763: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:34:33.824 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:34:33 smithi138 ceph-mon[33110]: pgmap v763: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:34:33.970 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:34:33 smithi067 ceph-mon[29730]: pgmap v763: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:34:35.745 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:34:35 smithi082 ceph-mon[32933]: pgmap v764: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:34:35.824 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:34:35 smithi138 ceph-mon[33110]: pgmap v764: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:34:35.970 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:34:35 smithi067 ceph-mon[29730]: pgmap v764: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:34:37.745 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:34:37 smithi082 ceph-mon[32933]: pgmap v765: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:34:37.824 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:34:37 smithi138 ceph-mon[33110]: pgmap v765: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:34:37.970 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:34:37 smithi067 ceph-mon[29730]: pgmap v765: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:34:39.824 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:34:39 smithi138 ceph-mon[33110]: pgmap v766: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:34:39.970 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:34:39 smithi067 ceph-mon[29730]: pgmap v766: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:34:39.995 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:34:39 smithi082 ceph-mon[32933]: pgmap v766: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:34:41.824 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:34:41 smithi138 ceph-mon[33110]: pgmap v767: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T21:34:41.970 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:34:41 smithi067 ceph-mon[29730]: pgmap v767: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T21:34:41.995 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:34:41 smithi082 ceph-mon[32933]: pgmap v767: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T21:34:43.824 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:34:43 smithi138 ceph-mon[33110]: pgmap v768: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:34:43.970 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:34:43 smithi067 ceph-mon[29730]: pgmap v768: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:34:43.995 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:34:43 smithi082 ceph-mon[32933]: pgmap v768: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:34:45.796 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:34:45 smithi082 ceph-mon[32933]: pgmap v769: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:34:45.824 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:34:45 smithi138 ceph-mon[33110]: pgmap v769: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:34:45.970 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:34:45 smithi067 ceph-mon[29730]: pgmap v769: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:34:47.824 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:34:47 smithi138 ceph-mon[33110]: pgmap v770: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:34:47.970 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:34:47 smithi067 ceph-mon[29730]: pgmap v770: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:34:47.995 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:34:47 smithi082 ceph-mon[32933]: pgmap v770: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:34:49.824 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:34:49 smithi138 ceph-mon[33110]: pgmap v771: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:34:49.970 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:34:49 smithi067 ceph-mon[29730]: pgmap v771: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:34:49.995 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:34:49 smithi082 ceph-mon[32933]: pgmap v771: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:34:51.824 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:34:51 smithi138 ceph-mon[33110]: pgmap v772: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T21:34:51.970 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:34:51 smithi067 ceph-mon[29730]: pgmap v772: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T21:34:51.995 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:34:51 smithi082 ceph-mon[32933]: pgmap v772: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T21:34:53.824 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:34:53 smithi138 ceph-mon[33110]: pgmap v773: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:34:53.970 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:34:53 smithi067 ceph-mon[29730]: pgmap v773: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:34:53.995 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:34:53 smithi082 ceph-mon[32933]: pgmap v773: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:34:55.773 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:34:55 smithi067 ceph-mon[29730]: pgmap v774: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:34:55.824 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:34:55 smithi138 ceph-mon[33110]: pgmap v774: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:34:55.995 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:34:55 smithi082 ceph-mon[32933]: pgmap v774: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:34:57.824 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:34:57 smithi138 ceph-mon[33110]: pgmap v775: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:34:57.970 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:34:57 smithi067 ceph-mon[29730]: pgmap v775: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:34:57.995 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:34:57 smithi082 ceph-mon[32933]: pgmap v775: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:34:59.824 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:34:59 smithi138 ceph-mon[33110]: pgmap v776: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:34:59.970 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:34:59 smithi067 ceph-mon[29730]: pgmap v776: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:34:59.995 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:34:59 smithi082 ceph-mon[32933]: pgmap v776: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:35:01.824 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:35:01 smithi138 ceph-mon[33110]: pgmap v777: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T21:35:01.970 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:35:01 smithi067 ceph-mon[29730]: pgmap v777: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T21:35:01.995 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:35:01 smithi082 ceph-mon[32933]: pgmap v777: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T21:35:02.824 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:35:02 smithi138 ceph-mon[33110]: pgmap v778: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:35:02.970 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:35:02 smithi067 ceph-mon[29730]: pgmap v778: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:35:02.995 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:35:02 smithi082 ceph-mon[32933]: pgmap v778: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:35:05.471 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:35:05 smithi067 ceph-mon[29730]: pgmap v779: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:35:05.495 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:35:05 smithi082 ceph-mon[32933]: pgmap v779: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:35:05.574 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:35:05 smithi138 ceph-mon[33110]: pgmap v779: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:35:07.470 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:35:07 smithi067 ceph-mon[29730]: pgmap v780: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:35:07.471 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:35:07 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T21:35:07.495 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:35:07 smithi082 ceph-mon[32933]: pgmap v780: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:35:07.495 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:35:07 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T21:35:07.574 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:35:07 smithi138 ceph-mon[33110]: pgmap v780: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:35:07.574 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:35:07 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T21:35:08.470 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:35:08 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T21:35:08.471 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:35:08 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T21:35:08.471 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:35:08 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:35:08.495 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:35:08 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T21:35:08.495 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:35:08 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T21:35:08.495 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:35:08 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:35:08.574 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:35:08 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T21:35:08.574 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:35:08 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T21:35:08.574 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:35:08 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:35:09.745 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:35:09 smithi082 ceph-mon[32933]: pgmap v781: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:35:09.824 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:35:09 smithi138 ceph-mon[33110]: pgmap v781: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:35:09.970 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:35:09 smithi067 ceph-mon[29730]: pgmap v781: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:35:11.745 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:35:11 smithi082 ceph-mon[32933]: pgmap v782: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T21:35:11.824 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:35:11 smithi138 ceph-mon[33110]: pgmap v782: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T21:35:11.970 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:35:11 smithi067 ceph-mon[29730]: pgmap v782: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T21:35:13.745 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:35:13 smithi082 ceph-mon[32933]: pgmap v783: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:35:13.824 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:35:13 smithi138 ceph-mon[33110]: pgmap v783: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:35:13.970 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:35:13 smithi067 ceph-mon[29730]: pgmap v783: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:35:15.745 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:35:15 smithi082 ceph-mon[32933]: pgmap v784: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:35:15.824 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:35:15 smithi138 ceph-mon[33110]: pgmap v784: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:35:15.971 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:35:15 smithi067 ceph-mon[29730]: pgmap v784: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:35:17.745 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:35:17 smithi082 ceph-mon[32933]: pgmap v785: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:35:17.824 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:35:17 smithi138 ceph-mon[33110]: pgmap v785: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:35:17.971 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:35:17 smithi067 ceph-mon[29730]: pgmap v785: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:35:19.745 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:35:19 smithi082 ceph-mon[32933]: pgmap v786: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:35:19.824 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:35:19 smithi138 ceph-mon[33110]: pgmap v786: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:35:19.971 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:35:19 smithi067 ceph-mon[29730]: pgmap v786: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:35:21.745 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:35:21 smithi082 ceph-mon[32933]: pgmap v787: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T21:35:21.824 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:35:21 smithi138 ceph-mon[33110]: pgmap v787: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T21:35:21.970 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:35:21 smithi067 ceph-mon[29730]: pgmap v787: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T21:35:23.745 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:35:23 smithi082 ceph-mon[32933]: pgmap v788: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:35:23.824 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:35:23 smithi138 ceph-mon[33110]: pgmap v788: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:35:23.970 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:35:23 smithi067 ceph-mon[29730]: pgmap v788: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:35:25.773 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:35:25 smithi067 ceph-mon[29730]: pgmap v789: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:35:25.824 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:35:25 smithi138 ceph-mon[33110]: pgmap v789: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:35:25.995 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:35:25 smithi082 ceph-mon[32933]: pgmap v789: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:35:27.824 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:35:27 smithi138 ceph-mon[33110]: pgmap v790: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:35:27.970 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:35:27 smithi067 ceph-mon[29730]: pgmap v790: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:35:27.995 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:35:27 smithi082 ceph-mon[32933]: pgmap v790: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:35:29.824 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:35:29 smithi138 ceph-mon[33110]: pgmap v791: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:35:29.970 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:35:29 smithi067 ceph-mon[29730]: pgmap v791: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:35:29.995 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:35:29 smithi082 ceph-mon[32933]: pgmap v791: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:35:31.824 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:35:31 smithi138 ceph-mon[33110]: pgmap v792: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T21:35:31.970 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:35:31 smithi067 ceph-mon[29730]: pgmap v792: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T21:35:31.995 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:35:31 smithi082 ceph-mon[32933]: pgmap v792: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T21:35:33.824 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:35:33 smithi138 ceph-mon[33110]: pgmap v793: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:35:33.970 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:35:33 smithi067 ceph-mon[29730]: pgmap v793: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:35:33.995 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:35:33 smithi082 ceph-mon[32933]: pgmap v793: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:35:35.824 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:35:35 smithi138 ceph-mon[33110]: pgmap v794: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:35:35.970 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:35:35 smithi067 ceph-mon[29730]: pgmap v794: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:35:35.995 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:35:35 smithi082 ceph-mon[32933]: pgmap v794: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:35:37.824 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:35:37 smithi138 ceph-mon[33110]: pgmap v795: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:35:37.970 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:35:37 smithi067 ceph-mon[29730]: pgmap v795: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:35:37.995 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:35:37 smithi082 ceph-mon[32933]: pgmap v795: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:35:39.824 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:35:39 smithi138 ceph-mon[33110]: pgmap v796: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:35:39.970 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:35:39 smithi067 ceph-mon[29730]: pgmap v796: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:35:39.995 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:35:39 smithi082 ceph-mon[32933]: pgmap v796: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:35:41.824 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:35:41 smithi138 ceph-mon[33110]: pgmap v797: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T21:35:41.970 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:35:41 smithi067 ceph-mon[29730]: pgmap v797: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T21:35:41.994 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:35:41 smithi082 ceph-mon[32933]: pgmap v797: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T21:35:43.824 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:35:43 smithi138 ceph-mon[33110]: pgmap v798: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:35:43.971 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:35:43 smithi067 ceph-mon[29730]: pgmap v798: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:35:43.995 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:35:43 smithi082 ceph-mon[32933]: pgmap v798: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:35:45.797 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:35:45 smithi082 ceph-mon[32933]: pgmap v799: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:35:45.824 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:35:45 smithi138 ceph-mon[33110]: pgmap v799: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:35:45.970 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:35:45 smithi067 ceph-mon[29730]: pgmap v799: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:35:47.824 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:35:47 smithi138 ceph-mon[33110]: pgmap v800: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:35:47.869 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:35:47 smithi067 ceph-mon[29730]: pgmap v800: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:35:47.995 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:35:47 smithi082 ceph-mon[32933]: pgmap v800: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:35:49.824 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:35:49 smithi138 ceph-mon[33110]: pgmap v801: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:35:49.970 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:35:49 smithi067 ceph-mon[29730]: pgmap v801: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:35:49.995 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:35:49 smithi082 ceph-mon[32933]: pgmap v801: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:35:51.824 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:35:51 smithi138 ceph-mon[33110]: pgmap v802: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T21:35:51.970 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:35:51 smithi067 ceph-mon[29730]: pgmap v802: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T21:35:51.995 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:35:51 smithi082 ceph-mon[32933]: pgmap v802: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T21:35:53.824 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:35:53 smithi138 ceph-mon[33110]: pgmap v803: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:35:53.970 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:35:53 smithi067 ceph-mon[29730]: pgmap v803: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:35:53.994 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:35:53 smithi082 ceph-mon[32933]: pgmap v803: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:35:54.970 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:35:54 smithi067 ceph-mon[29730]: pgmap v804: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:35:54.995 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:35:54 smithi082 ceph-mon[32933]: pgmap v804: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:35:55.074 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:35:54 smithi138 ceph-mon[33110]: pgmap v804: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:35:57.470 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:35:57 smithi067 ceph-mon[29730]: pgmap v805: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:35:57.495 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:35:57 smithi082 ceph-mon[32933]: pgmap v805: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:35:57.574 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:35:57 smithi138 ceph-mon[33110]: pgmap v805: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:35:59.471 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:35:59 smithi067 ceph-mon[29730]: pgmap v806: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:35:59.495 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:35:59 smithi082 ceph-mon[32933]: pgmap v806: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:35:59.574 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:35:59 smithi138 ceph-mon[33110]: pgmap v806: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:36:01.471 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:36:01 smithi067 ceph-mon[29730]: pgmap v807: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T21:36:01.495 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:36:01 smithi082 ceph-mon[32933]: pgmap v807: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T21:36:01.574 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:36:01 smithi138 ceph-mon[33110]: pgmap v807: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T21:36:03.470 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:36:03 smithi067 ceph-mon[29730]: pgmap v808: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:36:03.494 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:36:03 smithi082 ceph-mon[32933]: pgmap v808: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:36:03.574 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:36:03 smithi138 ceph-mon[33110]: pgmap v808: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:36:05.470 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:36:05 smithi067 ceph-mon[29730]: pgmap v809: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:36:05.495 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:36:05 smithi082 ceph-mon[32933]: pgmap v809: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:36:05.574 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:36:05 smithi138 ceph-mon[33110]: pgmap v809: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:36:07.470 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:36:07 smithi067 ceph-mon[29730]: pgmap v810: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:36:07.495 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:36:07 smithi082 ceph-mon[32933]: pgmap v810: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:36:07.574 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:36:07 smithi138 ceph-mon[33110]: pgmap v810: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:36:08.470 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:36:08 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T21:36:08.494 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:36:08 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T21:36:08.574 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:36:08 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T21:36:09.470 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:36:09 smithi067 ceph-mon[29730]: pgmap v811: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:36:09.471 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:36:09 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T21:36:09.471 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:36:09 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T21:36:09.471 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:36:09 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:36:09.495 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:36:09 smithi082 ceph-mon[32933]: pgmap v811: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:36:09.495 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:36:09 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T21:36:09.495 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:36:09 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T21:36:09.495 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:36:09 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:36:09.574 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:36:09 smithi138 ceph-mon[33110]: pgmap v811: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:36:09.574 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:36:09 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T21:36:09.574 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:36:09 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T21:36:09.574 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:36:09 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:36:11.470 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:36:11 smithi067 ceph-mon[29730]: pgmap v812: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T21:36:11.495 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:36:11 smithi082 ceph-mon[32933]: pgmap v812: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T21:36:11.574 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:36:11 smithi138 ceph-mon[33110]: pgmap v812: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T21:36:13.470 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:36:13 smithi067 ceph-mon[29730]: pgmap v813: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:36:13.495 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:36:13 smithi082 ceph-mon[32933]: pgmap v813: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:36:13.574 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:36:13 smithi138 ceph-mon[33110]: pgmap v813: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:36:15.470 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:36:15 smithi067 ceph-mon[29730]: pgmap v814: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:36:15.495 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:36:15 smithi082 ceph-mon[32933]: pgmap v814: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:36:15.574 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:36:15 smithi138 ceph-mon[33110]: pgmap v814: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:36:17.470 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:36:17 smithi067 ceph-mon[29730]: pgmap v815: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:36:17.495 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:36:17 smithi082 ceph-mon[32933]: pgmap v815: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:36:17.574 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:36:17 smithi138 ceph-mon[33110]: pgmap v815: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:36:19.471 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:36:19 smithi067 ceph-mon[29730]: pgmap v816: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:36:19.495 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:36:19 smithi082 ceph-mon[32933]: pgmap v816: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:36:19.574 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:36:19 smithi138 ceph-mon[33110]: pgmap v816: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:36:21.471 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:36:21 smithi067 ceph-mon[29730]: pgmap v817: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T21:36:21.495 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:36:21 smithi082 ceph-mon[32933]: pgmap v817: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T21:36:21.574 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:36:21 smithi138 ceph-mon[33110]: pgmap v817: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T21:36:23.470 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:36:23 smithi067 ceph-mon[29730]: pgmap v818: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:36:23.495 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:36:23 smithi082 ceph-mon[32933]: pgmap v818: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:36:23.574 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:36:23 smithi138 ceph-mon[33110]: pgmap v818: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:36:25.470 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:36:25 smithi067 ceph-mon[29730]: pgmap v819: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:36:25.495 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:36:25 smithi082 ceph-mon[32933]: pgmap v819: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:36:25.574 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:36:25 smithi138 ceph-mon[33110]: pgmap v819: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:36:27.470 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:36:27 smithi067 ceph-mon[29730]: pgmap v820: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:36:27.495 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:36:27 smithi082 ceph-mon[32933]: pgmap v820: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:36:27.574 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:36:27 smithi138 ceph-mon[33110]: pgmap v820: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:36:29.470 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:36:29 smithi067 ceph-mon[29730]: pgmap v821: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:36:29.495 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:36:29 smithi082 ceph-mon[32933]: pgmap v821: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:36:29.574 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:36:29 smithi138 ceph-mon[33110]: pgmap v821: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:36:31.471 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:36:31 smithi067 ceph-mon[29730]: pgmap v822: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T21:36:31.495 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:36:31 smithi082 ceph-mon[32933]: pgmap v822: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T21:36:31.574 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:36:31 smithi138 ceph-mon[33110]: pgmap v822: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T21:36:33.495 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:36:33 smithi082 ceph-mon[32933]: pgmap v823: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:36:33.574 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:36:33 smithi138 ceph-mon[33110]: pgmap v823: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:36:33.720 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:36:33 smithi067 ceph-mon[29730]: pgmap v823: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:36:35.495 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:36:35 smithi082 ceph-mon[32933]: pgmap v824: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:36:35.574 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:36:35 smithi138 ceph-mon[33110]: pgmap v824: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:36:35.720 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:36:35 smithi067 ceph-mon[29730]: pgmap v824: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:36:37.495 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:36:37 smithi082 ceph-mon[32933]: pgmap v825: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:36:37.574 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:36:37 smithi138 ceph-mon[33110]: pgmap v825: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:36:37.721 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:36:37 smithi067 ceph-mon[29730]: pgmap v825: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:36:39.495 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:36:39 smithi082 ceph-mon[32933]: pgmap v826: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:36:39.574 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:36:39 smithi138 ceph-mon[33110]: pgmap v826: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:36:39.720 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:36:39 smithi067 ceph-mon[29730]: pgmap v826: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:36:41.495 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:36:41 smithi082 ceph-mon[32933]: pgmap v827: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T21:36:41.574 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:36:41 smithi138 ceph-mon[33110]: pgmap v827: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T21:36:41.720 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:36:41 smithi067 ceph-mon[29730]: pgmap v827: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T21:36:43.495 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:36:43 smithi082 ceph-mon[32933]: pgmap v828: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:36:43.574 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:36:43 smithi138 ceph-mon[33110]: pgmap v828: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:36:43.720 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:36:43 smithi067 ceph-mon[29730]: pgmap v828: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T21:36:44.430 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:36:44.428+0000 7f90d91bc640 1 -- 172.21.15.67:0/432660756 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f90c00b2920 msgr2=0x7f90c00b4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).tick idle (900018952) for more than 900000000 us, fault. 2024-05-03T21:36:44.430 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:36:44.428+0000 7f90d91bc640 1 --2- 172.21.15.67:0/432660756 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f90c00b2920 0x7f90c00b4de0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f90d4103a90 tx=0x7f90cc009720 comp rx=0 tx=0).stop 2024-05-03T21:36:44.430 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:36:44.428+0000 7f90ca7fc640 1 -- 172.21.15.67:0/432660756 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f90c00b2920 msgr2=0x7f90c00b4de0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-05-03T21:36:44.430 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:36:44.428+0000 7f90ca7fc640 1 --2- 172.21.15.67:0/432660756 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f90c00b2920 0x7f90c00b4de0 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:36:44.430 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:36:44.428+0000 7f90ca7fc640 1 --2- 172.21.15.67:0/432660756 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f90c00bc9a0 0x7f90c00bed90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:36:44.430 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:36:44.428+0000 7f90ca7fc640 1 -- 172.21.15.67:0/432660756 --> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] -- mgr_command(tid 0: {"prefix": "fs subvolumegroup pin", "vol_name": "cephfs", "group_name": "qa", "pin_type": "random", "pin_setting": "0.10", "target": ["mon-mgr", ""]}) -- 0x7f90c00bf370 con 0x7f90c00bc9a0 2024-05-03T21:36:44.431 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:36:44.428+0000 7f90d91bc640 1 --2- 172.21.15.67:0/432660756 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f90c00bc9a0 0x7f90c00bed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:36:44.431 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:36:44.428+0000 7f90d91bc640 1 --2- 172.21.15.67:0/432660756 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f90c00bc9a0 0x7f90c00bed90 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f90cc004010 tx=0x7f90cc015000 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T21:36:44.534 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:36:44.533+0000 7f90c97fa640 1 -- 172.21.15.67:0/432660756 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f90bc006360 con 0x7f90d41045f0 2024-05-03T21:36:44.535 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:36:44.533+0000 7f90ca7fc640 1 -- 172.21.15.67:0/432660756 <== mon.0 v2:172.21.15.67:3300/0 12 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f90bc006360 con 0x7f90d41045f0 2024-05-03T21:36:44.634 INFO:teuthology.orchestra.run.smithi067.stderr:Error ETIMEDOUT: Connection timed out. Please check the client's permission and connection. 2024-05-03T21:36:44.640 DEBUG:teuthology.orchestra.run:got remote process result: 110 2024-05-03T21:36:44.641 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/teuthology/task/exec.py", line 66, in task remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi067 with status 110: "sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs subvolumegroup pin cephfs qa random 0.10'" 2024-05-03T21:36:44.874 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=d73e359b211c4acda846f7ec9a52f3d7 Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/teuthology/task/exec.py", line 66, in task remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi067 with status 110: "sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs subvolumegroup pin cephfs qa random 0.10'" 2024-05-03T21:36:44.876 DEBUG:teuthology.run_tasks:Unwinding manager kclient 2024-05-03T21:36:44.886 INFO:tasks.kclient:Unmounting kernel clients... 2024-05-03T21:36:44.886 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-05-03T21:36:44.886 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/proc/self/mounts of=/dev/stdout 2024-05-03T21:36:44.909 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-05-03T21:36:44.909 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/proc/self/mounts of=/dev/stdout 2024-05-03T21:36:44.968 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9a047fe7599a549042d06a3fb5b6813a203eabea shell --fsid 713bc57a-0991-11ef-bc97-c7b262605968 -- ceph osd blocklist ls 2024-05-03T21:36:45.267 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/713bc57a-0991-11ef-bc97-c7b262605968/mon.a/config 2024-05-03T21:36:45.391 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:36:45 smithi067 ceph-mon[29730]: pgmap v829: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:36:45.391 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:36:45 smithi067 ceph-mon[29730]: from='client.15138 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup pin", "vol_name": "cephfs", "group_name": "qa", "pin_type": "random", "pin_setting": "0.10", "target": ["mon-mgr", ""]}]: dispatch 2024-05-03T21:36:45.495 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:36:45 smithi082 ceph-mon[32933]: pgmap v829: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:36:45.495 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:36:45 smithi082 ceph-mon[32933]: from='client.15138 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup pin", "vol_name": "cephfs", "group_name": "qa", "pin_type": "random", "pin_setting": "0.10", "target": ["mon-mgr", ""]}]: dispatch 2024-05-03T21:36:45.574 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:36:45 smithi138 ceph-mon[33110]: pgmap v829: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-03T21:36:45.574 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:36:45 smithi138 ceph-mon[33110]: from='client.15138 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup pin", "vol_name": "cephfs", "group_name": "qa", "pin_type": "random", "pin_setting": "0.10", "target": ["mon-mgr", ""]}]: dispatch 2024-05-03T21:36:47.149 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:36:47.130+0000 7fe71cb71700 1 -- 172.21.15.67:0/2125852255 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe718102d60 msgr2=0x7fe71810f250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:36:47.149 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:36:47.130+0000 7fe71cb71700 1 --2- 172.21.15.67:0/2125852255 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe718102d60 0x7fe71810f250 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7fe71000b0a0 tx=0x7fe71000b3b0 comp rx=0 tx=0).stop 2024-05-03T21:36:47.149 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:36:47.130+0000 7fe71cb71700 1 -- 172.21.15.67:0/2125852255 shutdown_connections 2024-05-03T21:36:47.149 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:36:47.130+0000 7fe71cb71700 1 --2- 172.21.15.67:0/2125852255 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe718102d60 0x7fe71810f250 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:36:47.149 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:36:47.130+0000 7fe71cb71700 1 --2- 172.21.15.67:0/2125852255 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fe718108410 0x7fe7181087f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:36:47.150 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:36:47.130+0000 7fe71cb71700 1 --2- 172.21.15.67:0/2125852255 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fe7181023a0 0x7fe718102820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:36:47.150 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:36:47.130+0000 7fe71cb71700 1 -- 172.21.15.67:0/2125852255 >> 172.21.15.67:0/2125852255 conn(0x7fe718079350 msgr2=0x7fe718079760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T21:36:47.150 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:36:47.130+0000 7fe71cb71700 1 -- 172.21.15.67:0/2125852255 shutdown_connections 2024-05-03T21:36:47.150 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:36:47.130+0000 7fe71cb71700 1 -- 172.21.15.67:0/2125852255 wait complete. 2024-05-03T21:36:47.150 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:36:47.131+0000 7fe71cb71700 1 Processor -- start 2024-05-03T21:36:47.150 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:36:47.131+0000 7fe71cb71700 1 -- start start 2024-05-03T21:36:47.150 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:36:47.131+0000 7fe71cb71700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe7181023a0 0x7fe71819f8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:36:47.150 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:36:47.131+0000 7fe71cb71700 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fe718102d60 0x7fe71819fdf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:36:47.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:36:47.131+0000 7fe71cb71700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fe718108410 0x7fe718199a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:36:47.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:36:47.131+0000 7fe71cb71700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fe7181127c0 con 0x7fe7181023a0 2024-05-03T21:36:47.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:36:47.131+0000 7fe71cb71700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7fe718112640 con 0x7fe718108410 2024-05-03T21:36:47.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:36:47.131+0000 7fe71cb71700 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7fe718112940 con 0x7fe718102d60 2024-05-03T21:36:47.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:36:47.131+0000 7fe71659c700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe7181023a0 0x7fe71819f8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:36:47.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:36:47.132+0000 7fe71659c700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe7181023a0 0x7fe71819f8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:44862/0 (socket says 172.21.15.67:44862) 2024-05-03T21:36:47.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:36:47.132+0000 7fe71659c700 1 -- 172.21.15.67:0/1121544209 learned_addr learned my addr 172.21.15.67:0/1121544209 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-03T21:36:47.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:36:47.132+0000 7fe716d9d700 1 --2- 172.21.15.67:0/1121544209 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fe718108410 0x7fe718199a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:36:47.152 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:36:47.132+0000 7fe715d9b700 1 --2- 172.21.15.67:0/1121544209 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fe718102d60 0x7fe71819fdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:36:47.152 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:36:47.132+0000 7fe71659c700 1 -- 172.21.15.67:0/1121544209 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fe718102d60 msgr2=0x7fe71819fdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:36:47.152 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:36:47.132+0000 7fe71659c700 1 --2- 172.21.15.67:0/1121544209 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fe718102d60 0x7fe71819fdf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:36:47.152 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:36:47.132+0000 7fe71659c700 1 -- 172.21.15.67:0/1121544209 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fe718108410 msgr2=0x7fe718199a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:36:47.152 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:36:47.132+0000 7fe71659c700 1 --2- 172.21.15.67:0/1121544209 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fe718108410 0x7fe718199a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:36:47.152 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:36:47.132+0000 7fe71659c700 1 -- 172.21.15.67:0/1121544209 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe71819a1d0 con 0x7fe7181023a0 2024-05-03T21:36:47.152 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:36:47.132+0000 7fe71659c700 1 --2- 172.21.15.67:0/1121544209 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe7181023a0 0x7fe71819f8b0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7fe70800c920 tx=0x7fe70800cc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T21:36:47.152 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:36:47.132+0000 7fe7077fe700 1 -- 172.21.15.67:0/1121544209 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe708004a30 con 0x7fe7181023a0 2024-05-03T21:36:47.152 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:36:47.132+0000 7fe71cb71700 1 -- 172.21.15.67:0/1121544209 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe71819a4c0 con 0x7fe7181023a0 2024-05-03T21:36:47.153 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:36:47.132+0000 7fe71cb71700 1 -- 172.21.15.67:0/1121544209 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe7181a6780 con 0x7fe7181023a0 2024-05-03T21:36:47.153 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:36:47.132+0000 7fe7077fe700 1 -- 172.21.15.67:0/1121544209 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(28 keys) ==== 1170+0+0 (secure 0 0 0) 0x7fe708005520 con 0x7fe7181023a0 2024-05-03T21:36:47.153 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:36:47.132+0000 7fe7077fe700 1 -- 172.21.15.67:0/1121544209 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe70801d060 con 0x7fe7181023a0 2024-05-03T21:36:47.153 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:36:47.133+0000 7fe71cb71700 1 -- 172.21.15.67:0/1121544209 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe71806a190 con 0x7fe7181023a0 2024-05-03T21:36:47.153 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:36:47.135+0000 7fe7077fe700 1 -- 172.21.15.67:0/1121544209 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 17) ==== 149703+0+0 (secure 0 0 0) 0x7fe708004cd0 con 0x7fe7181023a0 2024-05-03T21:36:47.153 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:36:47.135+0000 7fe7077fe700 1 --2- 172.21.15.67:0/1121544209 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7fe7000b2910 0x7fe7000b4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:36:47.153 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:36:47.136+0000 7fe715d9b700 1 --2- 172.21.15.67:0/1121544209 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7fe7000b2910 0x7fe7000b4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:36:47.153 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:36:47.136+0000 7fe7077fe700 1 -- 172.21.15.67:0/1121544209 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 8211+0+0 (secure 0 0 0) 0x7fe7080e2210 con 0x7fe7181023a0 2024-05-03T21:36:47.154 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:36:47.136+0000 7fe715d9b700 1 --2- 172.21.15.67:0/1121544209 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7fe7000b2910 0x7fe7000b4dd0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7fe7180798c0 tx=0x7fe70c00b410 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T21:36:47.154 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:36:47.144+0000 7fe7077fe700 1 -- 172.21.15.67:0/1121544209 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191839 (secure 0 0 0) 0x7fe708014030 con 0x7fe7181023a0 2024-05-03T21:36:47.310 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:36:47 smithi067 ceph-mon[29730]: pgmap v830: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 5 op/s 2024-05-03T21:36:47.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:36:47.374+0000 7fe71cb71700 1 -- 172.21.15.67:0/1121544209 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7fe7181a6a60 con 0x7fe7181023a0 2024-05-03T21:36:47.377 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:36:47.375+0000 7fe7077fe700 1 -- 172.21.15.67:0/1121544209 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 10 entries v84) ==== 81+0+588 (secure 0 0 0) 0x7fe7080a9de0 con 0x7fe7181023a0 2024-05-03T21:36:47.377 INFO:teuthology.orchestra.run.smithi067.stdout:172.21.15.67:0/1239643865 2024-05-04T21:11:33.721416+0000 2024-05-03T21:36:47.377 INFO:teuthology.orchestra.run.smithi067.stdout:172.21.15.67:6801/2576113038 2024-05-04T21:11:33.721416+0000 2024-05-03T21:36:47.377 INFO:teuthology.orchestra.run.smithi067.stdout:172.21.15.67:6800/2576113038 2024-05-04T21:11:33.721416+0000 2024-05-03T21:36:47.377 INFO:teuthology.orchestra.run.smithi067.stdout:172.21.15.67:6800/1698171509 2024-05-04T21:11:07.333651+0000 2024-05-03T21:36:47.377 INFO:teuthology.orchestra.run.smithi067.stdout:172.21.15.67:0/2217893923 2024-05-04T21:11:33.721416+0000 2024-05-03T21:36:47.377 INFO:teuthology.orchestra.run.smithi067.stdout:172.21.15.67:0/20864811 2024-05-04T21:11:07.333651+0000 2024-05-03T21:36:47.378 INFO:teuthology.orchestra.run.smithi067.stdout:172.21.15.67:6801/1698171509 2024-05-04T21:11:07.333651+0000 2024-05-03T21:36:47.378 INFO:teuthology.orchestra.run.smithi067.stdout:172.21.15.67:0/773376553 2024-05-04T21:11:33.721416+0000 2024-05-03T21:36:47.378 INFO:teuthology.orchestra.run.smithi067.stdout:172.21.15.67:0/847549114 2024-05-04T21:11:07.333651+0000 2024-05-03T21:36:47.378 INFO:teuthology.orchestra.run.smithi067.stdout:172.21.15.67:0/1291778035 2024-05-04T21:11:07.333651+0000 2024-05-03T21:36:47.381 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:36:47.379+0000 7fe71cb71700 1 -- 172.21.15.67:0/1121544209 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7fe7000b2910 msgr2=0x7fe7000b4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:36:47.381 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:36:47.379+0000 7fe71cb71700 1 --2- 172.21.15.67:0/1121544209 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7fe7000b2910 0x7fe7000b4dd0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7fe7180798c0 tx=0x7fe70c00b410 comp rx=0 tx=0).stop 2024-05-03T21:36:47.381 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:36:47.379+0000 7fe71cb71700 1 -- 172.21.15.67:0/1121544209 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe7181023a0 msgr2=0x7fe71819f8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:36:47.381 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:36:47.379+0000 7fe71cb71700 1 --2- 172.21.15.67:0/1121544209 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe7181023a0 0x7fe71819f8b0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7fe70800c920 tx=0x7fe70800cc30 comp rx=0 tx=0).stop 2024-05-03T21:36:47.381 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:36:47.379+0000 7fe71cb71700 1 -- 172.21.15.67:0/1121544209 shutdown_connections 2024-05-03T21:36:47.381 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:36:47.379+0000 7fe71cb71700 1 --2- 172.21.15.67:0/1121544209 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7fe7000b2910 0x7fe7000b4dd0 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:36:47.382 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:36:47.379+0000 7fe71cb71700 1 --2- 172.21.15.67:0/1121544209 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe7181023a0 0x7fe71819f8b0 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:36:47.382 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:36:47.379+0000 7fe71cb71700 1 --2- 172.21.15.67:0/1121544209 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fe718108410 0x7fe718199a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:36:47.382 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:36:47.379+0000 7fe71cb71700 1 --2- 172.21.15.67:0/1121544209 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fe718102d60 0x7fe71819fdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:36:47.382 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:36:47.379+0000 7fe71cb71700 1 -- 172.21.15.67:0/1121544209 >> 172.21.15.67:0/1121544209 conn(0x7fe718079350 msgr2=0x7fe718101570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T21:36:47.382 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:36:47.379+0000 7fe71cb71700 1 -- 172.21.15.67:0/1121544209 shutdown_connections 2024-05-03T21:36:47.382 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:36:47.379+0000 7fe71cb71700 1 -- 172.21.15.67:0/1121544209 wait complete. 2024-05-03T21:36:47.383 INFO:teuthology.orchestra.run.smithi067.stderr:listed 10 entries 2024-05-03T21:36:47.495 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:36:47 smithi082 ceph-mon[32933]: pgmap v830: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 5 op/s 2024-05-03T21:36:47.574 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:36:47 smithi138 ceph-mon[33110]: pgmap v830: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 5 op/s 2024-05-03T21:36:48.040 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-05-03T21:36:48.040 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-03T21:36:48.040 DEBUG:teuthology.orchestra.run.smithi067:> sudo umount /home/ubuntu/cephtest/mnt.0 2024-05-03T21:36:48.076 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi067.front.sepia.ceph.com 2024-05-03T21:36:48.076 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-03T21:36:48.076 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-05-03T21:36:48.142 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-03T21:36:48.142 DEBUG:teuthology.orchestra.run.smithi067:> ip netns list 2024-05-03T21:36:48.198 INFO:teuthology.orchestra.run.smithi067.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-05-03T21:36:48.198 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-03T21:36:48.199 DEBUG:teuthology.orchestra.run.smithi067:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.0 2024-05-03T21:36:48.261 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:36:48 smithi067 ceph-mon[29730]: from='client.? 172.21.15.67:0/1121544209' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-05-03T21:36:48.264 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-03T21:36:48.264 DEBUG:teuthology.orchestra.run.smithi067:> sudo ip link delete ceph-brx 2024-05-03T21:36:48.360 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2024-05-03T21:36:48.370 INFO:tasks.cephadm:Teardown begin 2024-05-03T21:36:48.371 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-05-03T21:36:48.371 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9a047fe7599a549042d06a3fb5b6813a203eabea shell --fsid 713bc57a-0991-11ef-bc97-c7b262605968 -- ceph fs dump --format=json 2024-05-03T21:36:48.574 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:36:48 smithi138 ceph-mon[33110]: from='client.? 172.21.15.67:0/1121544209' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-05-03T21:36:48.621 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/713bc57a-0991-11ef-bc97-c7b262605968/mon.a/config 2024-05-03T21:36:48.745 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:36:48 smithi082 ceph-mon[32933]: from='client.? 172.21.15.67:0/1121544209' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-05-03T21:36:49.428 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:36:49 smithi067 ceph-mon[29730]: pgmap v831: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-05-03T21:36:49.574 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:36:49 smithi138 ceph-mon[33110]: pgmap v831: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-05-03T21:36:49.745 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:36:49 smithi082 ceph-mon[32933]: pgmap v831: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-05-03T21:36:50.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:36:50.253+0000 7f5df6eb2700 1 -- 172.21.15.67:0/1265237355 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5df0102d60 msgr2=0x7f5df010f250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:36:50.257 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:36:50.253+0000 7f5df6eb2700 1 --2- 172.21.15.67:0/1265237355 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5df0102d60 0x7f5df010f250 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f5de800b0a0 tx=0x7f5de800b3b0 comp rx=0 tx=0).stop 2024-05-03T21:36:50.257 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:36:50.253+0000 7f5df6eb2700 1 -- 172.21.15.67:0/1265237355 shutdown_connections 2024-05-03T21:36:50.257 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:36:50.253+0000 7f5df6eb2700 1 --2- 172.21.15.67:0/1265237355 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5df0102d60 0x7f5df010f250 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:36:50.257 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:36:50.253+0000 7f5df6eb2700 1 --2- 172.21.15.67:0/1265237355 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f5df0108410 0x7f5df01087f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:36:50.257 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:36:50.253+0000 7f5df6eb2700 1 --2- 172.21.15.67:0/1265237355 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f5df01023a0 0x7f5df0102820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:36:50.257 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:36:50.253+0000 7f5df6eb2700 1 -- 172.21.15.67:0/1265237355 >> 172.21.15.67:0/1265237355 conn(0x7f5df0079350 msgr2=0x7f5df0079760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T21:36:50.257 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:36:50.253+0000 7f5df6eb2700 1 -- 172.21.15.67:0/1265237355 shutdown_connections 2024-05-03T21:36:50.258 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:36:50.253+0000 7f5df6eb2700 1 -- 172.21.15.67:0/1265237355 wait complete. 2024-05-03T21:36:50.258 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:36:50.254+0000 7f5df6eb2700 1 Processor -- start 2024-05-03T21:36:50.258 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:36:50.254+0000 7f5df6eb2700 1 -- start start 2024-05-03T21:36:50.258 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:36:50.254+0000 7f5df6eb2700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f5df01023a0 0x7f5df019f920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:36:50.258 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:36:50.254+0000 7f5df6eb2700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5df0102d60 0x7f5df019fe60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:36:50.258 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:36:50.254+0000 7f5df6eb2700 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f5df0108410 0x7f5df0199ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:36:50.258 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:36:50.254+0000 7f5df6eb2700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f5df01127f0 con 0x7f5df0102d60 2024-05-03T21:36:50.258 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:36:50.254+0000 7f5df6eb2700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f5df0112670 con 0x7f5df01023a0 2024-05-03T21:36:50.259 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:36:50.254+0000 7f5df6eb2700 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f5df0112970 con 0x7f5df0108410 2024-05-03T21:36:50.259 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:36:50.254+0000 7f5deffff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5df0102d60 0x7f5df019fe60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:36:50.259 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:36:50.254+0000 7f5deffff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5df0102d60 0x7f5df019fe60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:44884/0 (socket says 172.21.15.67:44884) 2024-05-03T21:36:50.259 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:36:50.254+0000 7f5deffff700 1 -- 172.21.15.67:0/1025463687 learned_addr learned my addr 172.21.15.67:0/1025463687 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-03T21:36:50.259 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:36:50.254+0000 7f5df4c4e700 1 --2- 172.21.15.67:0/1025463687 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f5df01023a0 0x7f5df019f920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:36:50.259 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:36:50.254+0000 7f5df544f700 1 --2- 172.21.15.67:0/1025463687 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f5df0108410 0x7f5df0199ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:36:50.259 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:36:50.255+0000 7f5deffff700 1 -- 172.21.15.67:0/1025463687 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f5df0108410 msgr2=0x7f5df0199ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:36:50.259 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:36:50.255+0000 7f5deffff700 1 --2- 172.21.15.67:0/1025463687 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f5df0108410 0x7f5df0199ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:36:50.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:36:50.255+0000 7f5deffff700 1 -- 172.21.15.67:0/1025463687 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f5df01023a0 msgr2=0x7f5df019f920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:36:50.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:36:50.255+0000 7f5deffff700 1 --2- 172.21.15.67:0/1025463687 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f5df01023a0 0x7f5df019f920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:36:50.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:36:50.255+0000 7f5deffff700 1 -- 172.21.15.67:0/1025463687 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5df019a240 con 0x7f5df0102d60 2024-05-03T21:36:50.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:36:50.255+0000 7f5deffff700 1 --2- 172.21.15.67:0/1025463687 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5df0102d60 0x7f5df019fe60 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f5ddc00d980 tx=0x7f5ddc00dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T21:36:50.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:36:50.255+0000 7f5dedffb700 1 -- 172.21.15.67:0/1025463687 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ddc017070 con 0x7f5df0102d60 2024-05-03T21:36:50.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:36:50.255+0000 7f5df6eb2700 1 -- 172.21.15.67:0/1025463687 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5df019a530 con 0x7f5df0102d60 2024-05-03T21:36:50.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:36:50.255+0000 7f5dedffb700 1 -- 172.21.15.67:0/1025463687 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(28 keys) ==== 1170+0+0 (secure 0 0 0) 0x7f5ddc005020 con 0x7f5df0102d60 2024-05-03T21:36:50.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:36:50.255+0000 7f5dedffb700 1 -- 172.21.15.67:0/1025463687 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ddc00fc60 con 0x7f5df0102d60 2024-05-03T21:36:50.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:36:50.255+0000 7f5df6eb2700 1 -- 172.21.15.67:0/1025463687 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5df0103fb0 con 0x7f5df0102d60 2024-05-03T21:36:50.262 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:36:50.256+0000 7f5df6eb2700 1 -- 172.21.15.67:0/1025463687 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5df006a190 con 0x7f5df0102d60 2024-05-03T21:36:50.262 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:36:50.257+0000 7f5dedffb700 1 -- 172.21.15.67:0/1025463687 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 17) ==== 149703+0+0 (secure 0 0 0) 0x7f5ddc0051d0 con 0x7f5df0102d60 2024-05-03T21:36:50.262 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:36:50.258+0000 7f5dedffb700 1 --2- 172.21.15.67:0/1025463687 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f5de00b2990 0x7f5de00b4e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:36:50.262 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:36:50.258+0000 7f5df4c4e700 1 --2- 172.21.15.67:0/1025463687 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f5de00b2990 0x7f5de00b4e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:36:50.262 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:36:50.258+0000 7f5dedffb700 1 -- 172.21.15.67:0/1025463687 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 8211+0+0 (secure 0 0 0) 0x7f5ddc0e2150 con 0x7f5df0102d60 2024-05-03T21:36:50.262 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:36:50.259+0000 7f5df4c4e700 1 --2- 172.21.15.67:0/1025463687 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f5de00b2990 0x7f5de00b4e50 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f5de4005950 tx=0x7f5de400b410 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T21:36:50.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:36:50.263+0000 7f5dedffb700 1 -- 172.21.15.67:0/1025463687 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191839 (secure 0 0 0) 0x7f5ddc020020 con 0x7f5df0102d60 2024-05-03T21:36:50.533 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:36:50.531+0000 7f5df6eb2700 1 -- 172.21.15.67:0/1025463687 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f5df00640b0 con 0x7f5df0102d60 2024-05-03T21:36:50.534 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:36:50.532+0000 7f5dedffb700 1 -- 172.21.15.67:0/1025463687 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+11868 (secure 0 0 0) 0x7f5ddc0a9d20 con 0x7f5df0102d60 2024-05-03T21:36:50.534 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-05-03T21:36:50.534 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14541,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6835/4224826546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":4224826546},{"type":"v1","addr":"172.21.15.67:6835","nonce":4224826546}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14553,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6837/2485231621","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6836","nonce":2485231621},{"type":"v1","addr":"172.21.15.67:6837","nonce":2485231621}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14598,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6839/2106078262","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6838","nonce":2106078262},{"type":"v1","addr":"172.21.15.67:6839","nonce":2106078262}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14610,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6841/1559196215","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6840","nonce":1559196215},{"type":"v1","addr":"172.21.15.67:6841","nonce":1559196215}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24376,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.138:6833/3028022602","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6832","nonce":3028022602},{"type":"v1","addr":"172.21.15.138:6833","nonce":3028022602}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24400,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.138:6835/3342864697","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6834","nonce":3342864697},{"type":"v1","addr":"172.21.15.138:6835","nonce":3342864697}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7}],"filesystems":[{"mdsmap":{"epoch":26,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-05-03T21:18:44.665825+0000","modified":"2024-05-03T21:21:43.713458+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24473,"mds_1":24449,"mds_2":24425},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24406":{"gid":24406,"name":"c","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.138:6837/2203273249","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6836","nonce":2203273249},{"type":"v1","addr":"172.21.15.138:6837","nonce":2203273249}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24407":{"gid":24407,"name":"k","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.82:6833/1127287628","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6832","nonce":1127287628},{"type":"v1","addr":"172.21.15.82:6833","nonce":1127287628}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24418":{"gid":24418,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.138:6839/334382443","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6838","nonce":334382443},{"type":"v1","addr":"172.21.15.138:6839","nonce":334382443}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24425":{"gid":24425,"name":"e","rank":2,"incarnation":21,"state":"up:active","state_seq":14,"addr":"172.21.15.82:6835/280518761","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6834","nonce":280518761},{"type":"v1","addr":"172.21.15.82:6835","nonce":280518761}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24449":{"gid":24449,"name":"h","rank":1,"incarnation":19,"state":"up:active","state_seq":11,"addr":"172.21.15.82:6837/1275987852","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6836","nonce":1275987852},{"type":"v1","addr":"172.21.15.82:6837","nonce":1275987852}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24473":{"gid":24473,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":5,"addr":"172.21.15.82:6839/3644499578","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6838","nonce":3644499578},{"type":"v1","addr":"172.21.15.82:6839","nonce":3644499578}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24473,"qdb_cluster":[24425,24449,24473]},"id":1}]} 2024-05-03T21:36:50.536 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:36:50.534+0000 7f5df6eb2700 1 -- 172.21.15.67:0/1025463687 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f5de00b2990 msgr2=0x7f5de00b4e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:36:50.536 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:36:50.534+0000 7f5df6eb2700 1 --2- 172.21.15.67:0/1025463687 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f5de00b2990 0x7f5de00b4e50 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f5de4005950 tx=0x7f5de400b410 comp rx=0 tx=0).stop 2024-05-03T21:36:50.536 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:36:50.534+0000 7f5df6eb2700 1 -- 172.21.15.67:0/1025463687 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5df0102d60 msgr2=0x7f5df019fe60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:36:50.537 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:36:50.534+0000 7f5df6eb2700 1 --2- 172.21.15.67:0/1025463687 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5df0102d60 0x7f5df019fe60 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f5ddc00d980 tx=0x7f5ddc00dc90 comp rx=0 tx=0).stop 2024-05-03T21:36:50.537 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:36:50.534+0000 7f5df6eb2700 1 -- 172.21.15.67:0/1025463687 shutdown_connections 2024-05-03T21:36:50.537 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:36:50.534+0000 7f5df6eb2700 1 --2- 172.21.15.67:0/1025463687 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f5de00b2990 0x7f5de00b4e50 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:36:50.537 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:36:50.534+0000 7f5df6eb2700 1 --2- 172.21.15.67:0/1025463687 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f5df0108410 0x7f5df0199ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:36:50.537 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:36:50.534+0000 7f5df6eb2700 1 --2- 172.21.15.67:0/1025463687 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f5df01023a0 0x7f5df019f920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:36:50.537 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:36:50.534+0000 7f5df6eb2700 1 --2- 172.21.15.67:0/1025463687 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5df0102d60 0x7f5df019fe60 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:36:50.537 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:36:50.534+0000 7f5df6eb2700 1 -- 172.21.15.67:0/1025463687 >> 172.21.15.67:0/1025463687 conn(0x7f5df0079350 msgr2=0x7f5df0101570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T21:36:50.538 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:36:50.535+0000 7f5df6eb2700 1 -- 172.21.15.67:0/1025463687 shutdown_connections 2024-05-03T21:36:50.538 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:36:50.535+0000 7f5df6eb2700 1 -- 172.21.15.67:0/1025463687 wait complete. 2024-05-03T21:36:50.538 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 26 2024-05-03T21:36:51.197 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9a047fe7599a549042d06a3fb5b6813a203eabea shell --fsid 713bc57a-0991-11ef-bc97-c7b262605968 -- ceph fs dump --format=json 2024-05-03T21:36:51.446 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/713bc57a-0991-11ef-bc97-c7b262605968/mon.a/config 2024-05-03T21:36:51.470 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:36:51 smithi067 ceph-mon[29730]: pgmap v832: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-05-03T21:36:51.470 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:36:51 smithi067 ceph-mon[29730]: from='client.? 172.21.15.67:0/1025463687' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-03T21:36:51.574 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:36:51 smithi138 ceph-mon[33110]: pgmap v832: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-05-03T21:36:51.574 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:36:51 smithi138 ceph-mon[33110]: from='client.? 172.21.15.67:0/1025463687' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-03T21:36:51.745 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:36:51 smithi082 ceph-mon[32933]: pgmap v832: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-05-03T21:36:51.745 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:36:51 smithi082 ceph-mon[32933]: from='client.? 172.21.15.67:0/1025463687' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-03T21:36:52.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:36:52.993+0000 7fb924ae6700 1 -- 172.21.15.67:0/1264546 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb920108850 msgr2=0x7fb9201103d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:36:52.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:36:52.993+0000 7fb924ae6700 1 --2- 172.21.15.67:0/1264546 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb920108850 0x7fb9201103d0 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7fb9100099c0 tx=0x7fb910009cd0 comp rx=0 tx=0).stop 2024-05-03T21:36:52.996 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:36:52.993+0000 7fb924ae6700 1 -- 172.21.15.67:0/1264546 shutdown_connections 2024-05-03T21:36:52.996 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:36:52.993+0000 7fb924ae6700 1 --2- 172.21.15.67:0/1264546 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb920108850 0x7fb9201103d0 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:36:52.996 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:36:52.993+0000 7fb924ae6700 1 --2- 172.21.15.67:0/1264546 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fb920103750 0x7fb920103b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:36:52.996 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:36:52.993+0000 7fb924ae6700 1 --2- 172.21.15.67:0/1264546 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fb920104100 0x7fb920108120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:36:52.996 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:36:52.993+0000 7fb924ae6700 1 -- 172.21.15.67:0/1264546 >> 172.21.15.67:0/1264546 conn(0x7fb92007a600 msgr2=0x7fb92007aa10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T21:36:52.996 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:36:52.993+0000 7fb924ae6700 1 -- 172.21.15.67:0/1264546 shutdown_connections 2024-05-03T21:36:52.996 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:36:52.993+0000 7fb924ae6700 1 -- 172.21.15.67:0/1264546 wait complete. 2024-05-03T21:36:52.998 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:36:52.994+0000 7fb924ae6700 1 Processor -- start 2024-05-03T21:36:52.999 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:36:52.994+0000 7fb924ae6700 1 -- start start 2024-05-03T21:36:52.999 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:36:52.994+0000 7fb924ae6700 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fb920103750 0x7fb92019f830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:36:52.999 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:36:52.994+0000 7fb924ae6700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb920104100 0x7fb92019fd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:36:52.999 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:36:52.994+0000 7fb924ae6700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fb920108850 0x7fb9201999f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:36:52.999 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:36:52.994+0000 7fb924ae6700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fb9201138b0 con 0x7fb920104100 2024-05-03T21:36:52.999 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:36:52.994+0000 7fb924ae6700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7fb920113730 con 0x7fb920108850 2024-05-03T21:36:52.999 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:36:52.994+0000 7fb924ae6700 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7fb920113a30 con 0x7fb920103750 2024-05-03T21:36:52.999 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:36:52.994+0000 7fb91dd9b700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb920104100 0x7fb92019fd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:36:52.999 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:36:52.994+0000 7fb91dd9b700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb920104100 0x7fb92019fd70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:44896/0 (socket says 172.21.15.67:44896) 2024-05-03T21:36:53.000 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:36:52.994+0000 7fb91dd9b700 1 -- 172.21.15.67:0/2382173410 learned_addr learned my addr 172.21.15.67:0/2382173410 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-03T21:36:53.000 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:36:52.994+0000 7fb91e59c700 1 --2- 172.21.15.67:0/2382173410 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fb920103750 0x7fb92019f830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:36:53.000 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:36:52.994+0000 7fb91ed9d700 1 --2- 172.21.15.67:0/2382173410 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fb920108850 0x7fb9201999f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:36:53.000 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:36:52.995+0000 7fb91dd9b700 1 -- 172.21.15.67:0/2382173410 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fb920103750 msgr2=0x7fb92019f830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:36:53.000 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:36:52.995+0000 7fb91dd9b700 1 --2- 172.21.15.67:0/2382173410 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fb920103750 0x7fb92019f830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:36:53.000 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:36:52.995+0000 7fb91dd9b700 1 -- 172.21.15.67:0/2382173410 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fb920108850 msgr2=0x7fb9201999f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:36:53.000 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:36:52.995+0000 7fb91dd9b700 1 --2- 172.21.15.67:0/2382173410 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fb920108850 0x7fb9201999f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:36:53.000 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:36:52.995+0000 7fb91dd9b700 1 -- 172.21.15.67:0/2382173410 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb92019a150 con 0x7fb920104100 2024-05-03T21:36:53.001 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:36:52.995+0000 7fb91dd9b700 1 --2- 172.21.15.67:0/2382173410 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb920104100 0x7fb92019fd70 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7fb90800d950 tx=0x7fb90800dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T21:36:53.001 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:36:52.995+0000 7fb9177fe700 1 -- 172.21.15.67:0/2382173410 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb908016070 con 0x7fb920104100 2024-05-03T21:36:53.001 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:36:52.995+0000 7fb924ae6700 1 -- 172.21.15.67:0/2382173410 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb92019a440 con 0x7fb920104100 2024-05-03T21:36:53.001 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:36:52.995+0000 7fb9177fe700 1 -- 172.21.15.67:0/2382173410 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(28 keys) ==== 1170+0+0 (secure 0 0 0) 0x7fb90800fd90 con 0x7fb920104100 2024-05-03T21:36:53.001 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:36:52.995+0000 7fb924ae6700 1 -- 172.21.15.67:0/2382173410 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb9201a6700 con 0x7fb920104100 2024-05-03T21:36:53.001 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:36:52.995+0000 7fb9177fe700 1 -- 172.21.15.67:0/2382173410 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb90800e610 con 0x7fb920104100 2024-05-03T21:36:53.001 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:36:52.996+0000 7fb924ae6700 1 -- 172.21.15.67:0/2382173410 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb92006a190 con 0x7fb920104100 2024-05-03T21:36:53.002 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:36:52.998+0000 7fb9177fe700 1 -- 172.21.15.67:0/2382173410 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 17) ==== 149703+0+0 (secure 0 0 0) 0x7fb908019070 con 0x7fb920104100 2024-05-03T21:36:53.002 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:36:52.998+0000 7fb9177fe700 1 --2- 172.21.15.67:0/2382173410 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7fb90c0b2910 0x7fb90c0b4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:36:53.003 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:36:52.998+0000 7fb91e59c700 1 --2- 172.21.15.67:0/2382173410 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7fb90c0b2910 0x7fb90c0b4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:36:53.003 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:36:52.998+0000 7fb9177fe700 1 -- 172.21.15.67:0/2382173410 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 8211+0+0 (secure 0 0 0) 0x7fb9080e2250 con 0x7fb920104100 2024-05-03T21:36:53.003 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:36:52.999+0000 7fb91e59c700 1 --2- 172.21.15.67:0/2382173410 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7fb90c0b2910 0x7fb90c0b4dd0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7fb9180060b0 tx=0x7fb91800c040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T21:36:53.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:36:53.003+0000 7fb9177fe700 1 -- 172.21.15.67:0/2382173410 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191839 (secure 0 0 0) 0x7fb9080a9dc0 con 0x7fb920104100 2024-05-03T21:36:53.276 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:36:53.273+0000 7fb924ae6700 1 -- 172.21.15.67:0/2382173410 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb92019af20 con 0x7fb920104100 2024-05-03T21:36:53.276 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:36:53.274+0000 7fb9177fe700 1 -- 172.21.15.67:0/2382173410 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+11868 (secure 0 0 0) 0x7fb9080a96c0 con 0x7fb920104100 2024-05-03T21:36:53.276 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-05-03T21:36:53.276 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14541,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6835/4224826546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":4224826546},{"type":"v1","addr":"172.21.15.67:6835","nonce":4224826546}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14553,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6837/2485231621","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6836","nonce":2485231621},{"type":"v1","addr":"172.21.15.67:6837","nonce":2485231621}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14598,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6839/2106078262","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6838","nonce":2106078262},{"type":"v1","addr":"172.21.15.67:6839","nonce":2106078262}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14610,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6841/1559196215","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6840","nonce":1559196215},{"type":"v1","addr":"172.21.15.67:6841","nonce":1559196215}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24376,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.138:6833/3028022602","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6832","nonce":3028022602},{"type":"v1","addr":"172.21.15.138:6833","nonce":3028022602}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24400,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.138:6835/3342864697","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6834","nonce":3342864697},{"type":"v1","addr":"172.21.15.138:6835","nonce":3342864697}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7}],"filesystems":[{"mdsmap":{"epoch":26,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-05-03T21:18:44.665825+0000","modified":"2024-05-03T21:21:43.713458+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24473,"mds_1":24449,"mds_2":24425},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24406":{"gid":24406,"name":"c","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.138:6837/2203273249","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6836","nonce":2203273249},{"type":"v1","addr":"172.21.15.138:6837","nonce":2203273249}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24407":{"gid":24407,"name":"k","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.82:6833/1127287628","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6832","nonce":1127287628},{"type":"v1","addr":"172.21.15.82:6833","nonce":1127287628}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24418":{"gid":24418,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.138:6839/334382443","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6838","nonce":334382443},{"type":"v1","addr":"172.21.15.138:6839","nonce":334382443}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24425":{"gid":24425,"name":"e","rank":2,"incarnation":21,"state":"up:active","state_seq":14,"addr":"172.21.15.82:6835/280518761","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6834","nonce":280518761},{"type":"v1","addr":"172.21.15.82:6835","nonce":280518761}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24449":{"gid":24449,"name":"h","rank":1,"incarnation":19,"state":"up:active","state_seq":11,"addr":"172.21.15.82:6837/1275987852","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6836","nonce":1275987852},{"type":"v1","addr":"172.21.15.82:6837","nonce":1275987852}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24473":{"gid":24473,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":5,"addr":"172.21.15.82:6839/3644499578","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6838","nonce":3644499578},{"type":"v1","addr":"172.21.15.82:6839","nonce":3644499578}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24473,"qdb_cluster":[24425,24449,24473]},"id":1}]} 2024-05-03T21:36:53.278 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:36:53.277+0000 7fb924ae6700 1 -- 172.21.15.67:0/2382173410 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7fb90c0b2910 msgr2=0x7fb90c0b4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:36:53.278 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:36:53.277+0000 7fb924ae6700 1 --2- 172.21.15.67:0/2382173410 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7fb90c0b2910 0x7fb90c0b4dd0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7fb9180060b0 tx=0x7fb91800c040 comp rx=0 tx=0).stop 2024-05-03T21:36:53.279 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:36:53.277+0000 7fb924ae6700 1 -- 172.21.15.67:0/2382173410 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb920104100 msgr2=0x7fb92019fd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:36:53.279 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:36:53.277+0000 7fb924ae6700 1 --2- 172.21.15.67:0/2382173410 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb920104100 0x7fb92019fd70 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7fb90800d950 tx=0x7fb90800dc60 comp rx=0 tx=0).stop 2024-05-03T21:36:53.279 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:36:53.277+0000 7fb924ae6700 1 -- 172.21.15.67:0/2382173410 shutdown_connections 2024-05-03T21:36:53.279 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:36:53.277+0000 7fb924ae6700 1 --2- 172.21.15.67:0/2382173410 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7fb90c0b2910 0x7fb90c0b4dd0 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:36:53.279 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:36:53.277+0000 7fb924ae6700 1 --2- 172.21.15.67:0/2382173410 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fb920108850 0x7fb9201999f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:36:53.279 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:36:53.277+0000 7fb924ae6700 1 --2- 172.21.15.67:0/2382173410 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fb920103750 0x7fb92019f830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:36:53.279 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:36:53.277+0000 7fb924ae6700 1 --2- 172.21.15.67:0/2382173410 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb920104100 0x7fb92019fd70 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:36:53.279 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:36:53.277+0000 7fb924ae6700 1 -- 172.21.15.67:0/2382173410 >> 172.21.15.67:0/2382173410 conn(0x7fb92007a600 msgr2=0x7fb920102590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T21:36:53.280 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:36:53.277+0000 7fb924ae6700 1 -- 172.21.15.67:0/2382173410 shutdown_connections 2024-05-03T21:36:53.280 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:36:53.277+0000 7fb924ae6700 1 -- 172.21.15.67:0/2382173410 wait complete. 2024-05-03T21:36:53.280 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 26 2024-05-03T21:36:53.574 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:36:53 smithi138 ceph-mon[33110]: pgmap v833: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 255 B/s wr, 6 op/s 2024-05-03T21:36:53.651 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:36:53 smithi067 ceph-mon[29730]: pgmap v833: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 255 B/s wr, 6 op/s 2024-05-03T21:36:53.745 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:36:53 smithi082 ceph-mon[32933]: pgmap v833: 97 pgs: 97 active+clean; 637 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 255 B/s wr, 6 op/s 2024-05-03T21:36:54.002 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9a047fe7599a549042d06a3fb5b6813a203eabea shell --fsid 713bc57a-0991-11ef-bc97-c7b262605968 -- ceph osd dump --format=json 2024-05-03T21:36:54.257 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/713bc57a-0991-11ef-bc97-c7b262605968/mon.a/config 2024-05-03T21:36:54.555 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:36:54 smithi067 ceph-mon[29730]: from='client.? 172.21.15.67:0/2382173410' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-03T21:36:54.574 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:36:54 smithi138 ceph-mon[33110]: from='client.? 172.21.15.67:0/2382173410' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-03T21:36:54.745 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:36:54 smithi082 ceph-mon[32933]: from='client.? 172.21.15.67:0/2382173410' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-03T21:36:55.553 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:36:55 smithi067 ceph-mon[29730]: pgmap v834: 97 pgs: 97 active+clean; 638 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 426 B/s wr, 6 op/s 2024-05-03T21:36:55.574 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:36:55 smithi138 ceph-mon[33110]: pgmap v834: 97 pgs: 97 active+clean; 638 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 426 B/s wr, 6 op/s 2024-05-03T21:36:55.745 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:36:55 smithi082 ceph-mon[32933]: pgmap v834: 97 pgs: 97 active+clean; 638 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 426 B/s wr, 6 op/s 2024-05-03T21:36:55.766 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:36:55.764+0000 7fb40ef03700 1 -- 172.21.15.67:0/4069839537 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb408108850 msgr2=0x7fb4081103d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:36:55.766 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:36:55.764+0000 7fb40ef03700 1 --2- 172.21.15.67:0/4069839537 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb408108850 0x7fb4081103d0 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7fb40000b0d0 tx=0x7fb40000b3e0 comp rx=0 tx=0).stop 2024-05-03T21:36:55.767 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:36:55.765+0000 7fb40ef03700 1 -- 172.21.15.67:0/4069839537 shutdown_connections 2024-05-03T21:36:55.767 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:36:55.765+0000 7fb40ef03700 1 --2- 172.21.15.67:0/4069839537 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb408108850 0x7fb4081103d0 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:36:55.767 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:36:55.765+0000 7fb40ef03700 1 --2- 172.21.15.67:0/4069839537 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fb408103750 0x7fb408103b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:36:55.767 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:36:55.765+0000 7fb40ef03700 1 --2- 172.21.15.67:0/4069839537 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fb408104100 0x7fb408108120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:36:55.767 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:36:55.765+0000 7fb40ef03700 1 -- 172.21.15.67:0/4069839537 >> 172.21.15.67:0/4069839537 conn(0x7fb40807a600 msgr2=0x7fb40807aa10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T21:36:55.767 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:36:55.765+0000 7fb40ef03700 1 -- 172.21.15.67:0/4069839537 shutdown_connections 2024-05-03T21:36:55.767 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:36:55.765+0000 7fb40ef03700 1 -- 172.21.15.67:0/4069839537 wait complete. 2024-05-03T21:36:55.769 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:36:55.765+0000 7fb40ef03700 1 Processor -- start 2024-05-03T21:36:55.769 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:36:55.765+0000 7fb40ef03700 1 -- start start 2024-05-03T21:36:55.769 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:36:55.766+0000 7fb40ef03700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fb408103750 0x7fb40819f870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:36:55.770 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:36:55.766+0000 7fb40ef03700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb408104100 0x7fb40819fdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:36:55.770 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:36:55.766+0000 7fb40ef03700 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fb408108850 0x7fb408199a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:36:55.770 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:36:55.766+0000 7fb40ef03700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fb4081138d0 con 0x7fb408104100 2024-05-03T21:36:55.770 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:36:55.766+0000 7fb40ef03700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7fb408113750 con 0x7fb408103750 2024-05-03T21:36:55.770 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:36:55.766+0000 7fb40ef03700 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7fb408113a50 con 0x7fb408108850 2024-05-03T21:36:55.770 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:36:55.766+0000 7fb407fff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb408104100 0x7fb40819fdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:36:55.770 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:36:55.766+0000 7fb407fff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb408104100 0x7fb40819fdb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:38962/0 (socket says 172.21.15.67:38962) 2024-05-03T21:36:55.770 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:36:55.766+0000 7fb407fff700 1 -- 172.21.15.67:0/3419067302 learned_addr learned my addr 172.21.15.67:0/3419067302 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-03T21:36:55.771 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:36:55.766+0000 7fb40d4a0700 1 --2- 172.21.15.67:0/3419067302 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fb408108850 0x7fb408199a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:36:55.771 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:36:55.766+0000 7fb40cc9f700 1 --2- 172.21.15.67:0/3419067302 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fb408103750 0x7fb40819f870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:36:55.771 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:36:55.766+0000 7fb407fff700 1 -- 172.21.15.67:0/3419067302 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fb408108850 msgr2=0x7fb408199a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:36:55.771 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:36:55.766+0000 7fb407fff700 1 --2- 172.21.15.67:0/3419067302 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fb408108850 0x7fb408199a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:36:55.771 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:36:55.766+0000 7fb407fff700 1 -- 172.21.15.67:0/3419067302 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fb408103750 msgr2=0x7fb40819f870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:36:55.771 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:36:55.766+0000 7fb407fff700 1 --2- 172.21.15.67:0/3419067302 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fb408103750 0x7fb40819f870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:36:55.771 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:36:55.766+0000 7fb407fff700 1 -- 172.21.15.67:0/3419067302 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb40819a190 con 0x7fb408104100 2024-05-03T21:36:55.771 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:36:55.766+0000 7fb407fff700 1 --2- 172.21.15.67:0/3419067302 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb408104100 0x7fb40819fdb0 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7fb3f400c970 tx=0x7fb3f400cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T21:36:55.771 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:36:55.767+0000 7fb405ffb700 1 -- 172.21.15.67:0/3419067302 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3f4007aa0 con 0x7fb408104100 2024-05-03T21:36:55.772 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:36:55.767+0000 7fb405ffb700 1 -- 172.21.15.67:0/3419067302 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(28 keys) ==== 1170+0+0 (secure 0 0 0) 0x7fb3f4004d30 con 0x7fb408104100 2024-05-03T21:36:55.772 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:36:55.767+0000 7fb40ef03700 1 -- 172.21.15.67:0/3419067302 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb40819a480 con 0x7fb408104100 2024-05-03T21:36:55.772 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:36:55.767+0000 7fb405ffb700 1 -- 172.21.15.67:0/3419067302 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3f4005690 con 0x7fb408104100 2024-05-03T21:36:55.772 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:36:55.767+0000 7fb40ef03700 1 -- 172.21.15.67:0/3419067302 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb4081a6740 con 0x7fb408104100 2024-05-03T21:36:55.772 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:36:55.768+0000 7fb40ef03700 1 -- 172.21.15.67:0/3419067302 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb40806a190 con 0x7fb408104100 2024-05-03T21:36:55.773 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:36:55.769+0000 7fb405ffb700 1 -- 172.21.15.67:0/3419067302 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 17) ==== 149703+0+0 (secure 0 0 0) 0x7fb3f4004750 con 0x7fb408104100 2024-05-03T21:36:55.773 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:36:55.770+0000 7fb405ffb700 1 --2- 172.21.15.67:0/3419067302 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7fb3f80b2990 0x7fb3f80b4e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:36:55.773 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:36:55.770+0000 7fb40cc9f700 1 --2- 172.21.15.67:0/3419067302 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7fb3f80b2990 0x7fb3f80b4e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:36:55.773 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:36:55.770+0000 7fb405ffb700 1 -- 172.21.15.67:0/3419067302 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 8211+0+0 (secure 0 0 0) 0x7fb3f40e23a0 con 0x7fb408104100 2024-05-03T21:36:55.773 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:36:55.770+0000 7fb40cc9f700 1 --2- 172.21.15.67:0/3419067302 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7fb3f80b2990 0x7fb3f80b4e50 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7fb3fc005950 tx=0x7fb3fc005ed0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T21:36:55.775 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:36:55.773+0000 7fb405ffb700 1 -- 172.21.15.67:0/3419067302 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191839 (secure 0 0 0) 0x7fb3f40aa090 con 0x7fb408104100 2024-05-03T21:36:55.997 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:36:55.995+0000 7fb40ef03700 1 -- 172.21.15.67:0/3419067302 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fb40819af60 con 0x7fb408104100 2024-05-03T21:36:55.998 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:36:55.997+0000 7fb405ffb700 1 -- 172.21.15.67:0/3419067302 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v84) ==== 74+0+22637 (secure 0 0 0) 0x7fb40819af60 con 0x7fb408104100 2024-05-03T21:36:55.999 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-05-03T21:36:56.001 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":84,"fsid":"713bc57a-0991-11ef-bc97-c7b262605968","created":"2024-05-03T21:10:34.070178+0000","modified":"2024-05-03T21:19:34.967839+0000","last_up_change":"2024-05-03T21:17:39.993919+0000","last_in_change":"2024-05-03T21:17:19.978727+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-03T21:13:59.810446+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-05-03T21:18:37.953919+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"82","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-05-03T21:18:41.571707+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"82","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7939f31c-6fb7-42bd-af96-88a448d11bd2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":3692483807},{"type":"v1","addr":"172.21.15.67:6803","nonce":3692483807}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":3692483807},{"type":"v1","addr":"172.21.15.67:6805","nonce":3692483807}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":3692483807},{"type":"v1","addr":"172.21.15.67:6809","nonce":3692483807}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":3692483807},{"type":"v1","addr":"172.21.15.67:6807","nonce":3692483807}]},"public_addr":"172.21.15.67:6803/3692483807","cluster_addr":"172.21.15.67:6805/3692483807","heartbeat_back_addr":"172.21.15.67:6809/3692483807","heartbeat_front_addr":"172.21.15.67:6807/3692483807","state":["exists","up"]},{"osd":1,"uuid":"71858193-05a6-4c2c-9bb7-bc3d811d634e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6800","nonce":1139376397},{"type":"v1","addr":"172.21.15.82:6801","nonce":1139376397}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6802","nonce":1139376397},{"type":"v1","addr":"172.21.15.82:6803","nonce":1139376397}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6806","nonce":1139376397},{"type":"v1","addr":"172.21.15.82:6807","nonce":1139376397}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6804","nonce":1139376397},{"type":"v1","addr":"172.21.15.82:6805","nonce":1139376397}]},"public_addr":"172.21.15.82:6801/1139376397","cluster_addr":"172.21.15.82:6803/1139376397","heartbeat_back_addr":"172.21.15.82:6807/1139376397","heartbeat_front_addr":"172.21.15.82:6805/1139376397","state":["exists","up"]},{"osd":2,"uuid":"f9ece78d-6a19-4d46-9022-6ed184074745","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6800","nonce":2774330123},{"type":"v1","addr":"172.21.15.138:6801","nonce":2774330123}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6802","nonce":2774330123},{"type":"v1","addr":"172.21.15.138:6803","nonce":2774330123}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6806","nonce":2774330123},{"type":"v1","addr":"172.21.15.138:6807","nonce":2774330123}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6804","nonce":2774330123},{"type":"v1","addr":"172.21.15.138:6805","nonce":2774330123}]},"public_addr":"172.21.15.138:6801/2774330123","cluster_addr":"172.21.15.138:6803/2774330123","heartbeat_back_addr":"172.21.15.138:6807/2774330123","heartbeat_front_addr":"172.21.15.138:6805/2774330123","state":["exists","up"]},{"osd":3,"uuid":"3616df71-e7b3-4687-aaba-37b0b7766a3c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":27,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":3915934652},{"type":"v1","addr":"172.21.15.67:6811","nonce":3915934652}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":3915934652},{"type":"v1","addr":"172.21.15.67:6813","nonce":3915934652}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":3915934652},{"type":"v1","addr":"172.21.15.67:6817","nonce":3915934652}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":3915934652},{"type":"v1","addr":"172.21.15.67:6815","nonce":3915934652}]},"public_addr":"172.21.15.67:6811/3915934652","cluster_addr":"172.21.15.67:6813/3915934652","heartbeat_back_addr":"172.21.15.67:6817/3915934652","heartbeat_front_addr":"172.21.15.67:6815/3915934652","state":["exists","up"]},{"osd":4,"uuid":"c3fd0ffd-e23f-4279-b679-9d95a144cf32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6808","nonce":4031818951},{"type":"v1","addr":"172.21.15.82:6809","nonce":4031818951}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6810","nonce":4031818951},{"type":"v1","addr":"172.21.15.82:6811","nonce":4031818951}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6814","nonce":4031818951},{"type":"v1","addr":"172.21.15.82:6815","nonce":4031818951}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6812","nonce":4031818951},{"type":"v1","addr":"172.21.15.82:6813","nonce":4031818951}]},"public_addr":"172.21.15.82:6809/4031818951","cluster_addr":"172.21.15.82:6811/4031818951","heartbeat_back_addr":"172.21.15.82:6815/4031818951","heartbeat_front_addr":"172.21.15.82:6813/4031818951","state":["exists","up"]},{"osd":5,"uuid":"540ef0d9-a6dc-43c6-b09e-e87340ba24cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6808","nonce":2521889499},{"type":"v1","addr":"172.21.15.138:6809","nonce":2521889499}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6810","nonce":2521889499},{"type":"v1","addr":"172.21.15.138:6811","nonce":2521889499}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6814","nonce":2521889499},{"type":"v1","addr":"172.21.15.138:6815","nonce":2521889499}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6812","nonce":2521889499},{"type":"v1","addr":"172.21.15.138:6813","nonce":2521889499}]},"public_addr":"172.21.15.138:6809/2521889499","cluster_addr":"172.21.15.138:6811/2521889499","heartbeat_back_addr":"172.21.15.138:6815/2521889499","heartbeat_front_addr":"172.21.15.138:6813/2521889499","state":["exists","up"]},{"osd":6,"uuid":"49a89fd0-958a-4be9-a8db-6022f946153e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6818","nonce":3269682731},{"type":"v1","addr":"172.21.15.67:6819","nonce":3269682731}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":3269682731},{"type":"v1","addr":"172.21.15.67:6821","nonce":3269682731}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":3269682731},{"type":"v1","addr":"172.21.15.67:6825","nonce":3269682731}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":3269682731},{"type":"v1","addr":"172.21.15.67:6823","nonce":3269682731}]},"public_addr":"172.21.15.67:6819/3269682731","cluster_addr":"172.21.15.67:6821/3269682731","heartbeat_back_addr":"172.21.15.67:6825/3269682731","heartbeat_front_addr":"172.21.15.67:6823/3269682731","state":["exists","up"]},{"osd":7,"uuid":"8fa4cd64-ca6b-48ae-8db0-d369ce6102ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":50,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6816","nonce":3008589976},{"type":"v1","addr":"172.21.15.82:6817","nonce":3008589976}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6818","nonce":3008589976},{"type":"v1","addr":"172.21.15.82:6819","nonce":3008589976}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6822","nonce":3008589976},{"type":"v1","addr":"172.21.15.82:6823","nonce":3008589976}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6820","nonce":3008589976},{"type":"v1","addr":"172.21.15.82:6821","nonce":3008589976}]},"public_addr":"172.21.15.82:6817/3008589976","cluster_addr":"172.21.15.82:6819/3008589976","heartbeat_back_addr":"172.21.15.82:6823/3008589976","heartbeat_front_addr":"172.21.15.82:6821/3008589976","state":["exists","up"]},{"osd":8,"uuid":"fae39087-6c38-4771-a2b9-3f4fecb08edd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":57,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6816","nonce":2223920070},{"type":"v1","addr":"172.21.15.138:6817","nonce":2223920070}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6818","nonce":2223920070},{"type":"v1","addr":"172.21.15.138:6819","nonce":2223920070}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6822","nonce":2223920070},{"type":"v1","addr":"172.21.15.138:6823","nonce":2223920070}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6820","nonce":2223920070},{"type":"v1","addr":"172.21.15.138:6821","nonce":2223920070}]},"public_addr":"172.21.15.138:6817/2223920070","cluster_addr":"172.21.15.138:6819/2223920070","heartbeat_back_addr":"172.21.15.138:6823/2223920070","heartbeat_front_addr":"172.21.15.138:6821/2223920070","state":["exists","up"]},{"osd":9,"uuid":"3d387dbf-0163-41b7-98af-4fc45ed73ad4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":63,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":3939417129},{"type":"v1","addr":"172.21.15.67:6827","nonce":3939417129}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":3939417129},{"type":"v1","addr":"172.21.15.67:6829","nonce":3939417129}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":3939417129},{"type":"v1","addr":"172.21.15.67:6833","nonce":3939417129}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":3939417129},{"type":"v1","addr":"172.21.15.67:6831","nonce":3939417129}]},"public_addr":"172.21.15.67:6827/3939417129","cluster_addr":"172.21.15.67:6829/3939417129","heartbeat_back_addr":"172.21.15.67:6833/3939417129","heartbeat_front_addr":"172.21.15.67:6831/3939417129","state":["exists","up"]},{"osd":10,"uuid":"db0305a0-a83b-474b-837a-fbc239ac1738","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6824","nonce":2505507029},{"type":"v1","addr":"172.21.15.82:6825","nonce":2505507029}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6826","nonce":2505507029},{"type":"v1","addr":"172.21.15.82:6827","nonce":2505507029}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6830","nonce":2505507029},{"type":"v1","addr":"172.21.15.82:6831","nonce":2505507029}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6828","nonce":2505507029},{"type":"v1","addr":"172.21.15.82:6829","nonce":2505507029}]},"public_addr":"172.21.15.82:6825/2505507029","cluster_addr":"172.21.15.82:6827/2505507029","heartbeat_back_addr":"172.21.15.82:6831/2505507029","heartbeat_front_addr":"172.21.15.82:6829/2505507029","state":["exists","up"]},{"osd":11,"uuid":"479c2747-f28e-4704-87e4-d0c31782e2de","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":74,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6824","nonce":2528685018},{"type":"v1","addr":"172.21.15.138:6825","nonce":2528685018}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6826","nonce":2528685018},{"type":"v1","addr":"172.21.15.138:6827","nonce":2528685018}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6830","nonce":2528685018},{"type":"v1","addr":"172.21.15.138:6831","nonce":2528685018}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6828","nonce":2528685018},{"type":"v1","addr":"172.21.15.138:6829","nonce":2528685018}]},"public_addr":"172.21.15.138:6825/2528685018","cluster_addr":"172.21.15.138:6827/2528685018","heartbeat_back_addr":"172.21.15.138:6831/2528685018","heartbeat_front_addr":"172.21.15.138:6829/2528685018","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-03T21:13:07.768773+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-03T21:13:31.870232+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-03T21:13:55.394206+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-03T21:14:20.540281+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-03T21:14:44.318523+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-03T21:15:08.490829+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-03T21:15:34.144734+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-03T21:15:58.717240+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-03T21:16:22.743013+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-03T21:16:49.099588+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-03T21:17:13.321126+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-03T21:17:38.001978+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.0","mappings":[{"from":8,"to":4}]},{"pgid":"2.a","mappings":[{"from":2,"to":11}]},{"pgid":"2.b","mappings":[{"from":2,"to":4}]},{"pgid":"2.10","mappings":[{"from":8,"to":11}]},{"pgid":"3.1","mappings":[{"from":4,"to":9}]},{"pgid":"3.1b","mappings":[{"from":5,"to":9}]},{"pgid":"3.21","mappings":[{"from":4,"to":8}]},{"pgid":"3.3c","mappings":[{"from":4,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.67:0/1239643865":"2024-05-04T21:11:33.721416+0000","172.21.15.67:6801/2576113038":"2024-05-04T21:11:33.721416+0000","172.21.15.67:6800/2576113038":"2024-05-04T21:11:33.721416+0000","172.21.15.67:6800/1698171509":"2024-05-04T21:11:07.333651+0000","172.21.15.67:0/2217893923":"2024-05-04T21:11:33.721416+0000","172.21.15.67:0/20864811":"2024-05-04T21:11:07.333651+0000","172.21.15.67:6801/1698171509":"2024-05-04T21:11:07.333651+0000","172.21.15.67:0/773376553":"2024-05-04T21:11:33.721416+0000","172.21.15.67:0/847549114":"2024-05-04T21:11:07.333651+0000","172.21.15.67:0/1291778035":"2024-05-04T21:11:07.333651+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-03T21:36:56.002 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:36:55.999+0000 7fb40ef03700 1 -- 172.21.15.67:0/3419067302 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7fb3f80b2990 msgr2=0x7fb3f80b4e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:36:56.002 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:36:55.999+0000 7fb40ef03700 1 --2- 172.21.15.67:0/3419067302 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7fb3f80b2990 0x7fb3f80b4e50 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7fb3fc005950 tx=0x7fb3fc005ed0 comp rx=0 tx=0).stop 2024-05-03T21:36:56.002 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:36:55.999+0000 7fb40ef03700 1 -- 172.21.15.67:0/3419067302 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb408104100 msgr2=0x7fb40819fdb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:36:56.003 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:36:55.999+0000 7fb40ef03700 1 --2- 172.21.15.67:0/3419067302 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb408104100 0x7fb40819fdb0 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7fb3f400c970 tx=0x7fb3f400cc80 comp rx=0 tx=0).stop 2024-05-03T21:36:56.003 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:36:55.999+0000 7fb40ef03700 1 -- 172.21.15.67:0/3419067302 shutdown_connections 2024-05-03T21:36:56.003 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:36:55.999+0000 7fb40ef03700 1 --2- 172.21.15.67:0/3419067302 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7fb3f80b2990 0x7fb3f80b4e50 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:36:56.003 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:36:55.999+0000 7fb40ef03700 1 --2- 172.21.15.67:0/3419067302 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fb408108850 0x7fb408199a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:36:56.003 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:36:55.999+0000 7fb40ef03700 1 --2- 172.21.15.67:0/3419067302 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fb408103750 0x7fb40819f870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:36:56.003 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:36:55.999+0000 7fb40ef03700 1 --2- 172.21.15.67:0/3419067302 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb408104100 0x7fb40819fdb0 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:36:56.003 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:36:55.999+0000 7fb40ef03700 1 -- 172.21.15.67:0/3419067302 >> 172.21.15.67:0/3419067302 conn(0x7fb40807a600 msgr2=0x7fb4081024e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T21:36:56.003 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:36:55.999+0000 7fb40ef03700 1 -- 172.21.15.67:0/3419067302 shutdown_connections 2024-05-03T21:36:56.003 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:36:55.999+0000 7fb40ef03700 1 -- 172.21.15.67:0/3419067302 wait complete. 2024-05-03T21:36:56.574 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:36:56 smithi138 ceph-mon[33110]: from='client.? 172.21.15.67:0/3419067302' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-03T21:36:56.641 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:36:56 smithi067 ceph-mon[29730]: from='client.? 172.21.15.67:0/3419067302' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-03T21:36:56.675 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9a047fe7599a549042d06a3fb5b6813a203eabea shell --fsid 713bc57a-0991-11ef-bc97-c7b262605968 -- ceph fs fail cephfs 2024-05-03T21:36:56.745 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:36:56 smithi082 ceph-mon[32933]: from='client.? 172.21.15.67:0/3419067302' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-03T21:36:56.931 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/713bc57a-0991-11ef-bc97-c7b262605968/mon.a/config 2024-05-03T21:36:57.307 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:36:57 smithi067 ceph-mon[29730]: pgmap v835: 97 pgs: 97 active+clean; 638 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 426 B/s wr, 6 op/s 2024-05-03T21:36:57.574 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:36:57 smithi138 ceph-mon[33110]: pgmap v835: 97 pgs: 97 active+clean; 638 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 426 B/s wr, 6 op/s 2024-05-03T21:36:57.745 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:36:57 smithi082 ceph-mon[32933]: pgmap v835: 97 pgs: 97 active+clean; 638 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 426 B/s wr, 6 op/s 2024-05-03T21:36:58.604 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:36:58.599+0000 7ff363a2e700 1 -- 172.21.15.67:0/3288547962 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff35c103540 msgr2=0x7ff35c1039c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:36:58.604 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:36:58.599+0000 7ff363a2e700 1 --2- 172.21.15.67:0/3288547962 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff35c103540 0x7ff35c1039c0 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7ff3580099c0 tx=0x7ff358009cd0 comp rx=0 tx=0).stop 2024-05-03T21:36:58.604 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:36:58.599+0000 7ff363a2e700 1 -- 172.21.15.67:0/3288547962 shutdown_connections 2024-05-03T21:36:58.604 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:36:58.599+0000 7ff363a2e700 1 --2- 172.21.15.67:0/3288547962 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ff35c103f00 0x7ff35c1103b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:36:58.605 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:36:58.599+0000 7ff363a2e700 1 --2- 172.21.15.67:0/3288547962 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7ff35c1095e0 0x7ff35c1099c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:36:58.605 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:36:58.599+0000 7ff363a2e700 1 --2- 172.21.15.67:0/3288547962 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff35c103540 0x7ff35c1039c0 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:36:58.605 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:36:58.599+0000 7ff363a2e700 1 -- 172.21.15.67:0/3288547962 >> 172.21.15.67:0/3288547962 conn(0x7ff35c07a650 msgr2=0x7ff35c07aa60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T21:36:58.605 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:36:58.599+0000 7ff363a2e700 1 -- 172.21.15.67:0/3288547962 shutdown_connections 2024-05-03T21:36:58.605 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:36:58.599+0000 7ff363a2e700 1 -- 172.21.15.67:0/3288547962 wait complete. 2024-05-03T21:36:58.605 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:36:58.600+0000 7ff363a2e700 1 Processor -- start 2024-05-03T21:36:58.606 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:36:58.600+0000 7ff363a2e700 1 -- start start 2024-05-03T21:36:58.606 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:36:58.600+0000 7ff363a2e700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ff35c103540 0x7ff35c19f920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:36:58.606 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:36:58.600+0000 7ff363a2e700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff35c103f00 0x7ff35c19fe60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:36:58.606 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:36:58.600+0000 7ff363a2e700 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7ff35c1095e0 0x7ff35c199b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:36:58.606 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:36:58.600+0000 7ff363a2e700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7ff35c113950 con 0x7ff35c103f00 2024-05-03T21:36:58.607 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:36:58.600+0000 7ff363a2e700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7ff35c1137d0 con 0x7ff35c103540 2024-05-03T21:36:58.607 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:36:58.600+0000 7ff363a2e700 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7ff35c113ad0 con 0x7ff35c1095e0 2024-05-03T21:36:58.607 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:36:58.600+0000 7ff360fc9700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff35c103f00 0x7ff35c19fe60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:36:58.607 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:36:58.600+0000 7ff360fc9700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff35c103f00 0x7ff35c19fe60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:38994/0 (socket says 172.21.15.67:38994) 2024-05-03T21:36:58.607 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:36:58.600+0000 7ff360fc9700 1 -- 172.21.15.67:0/769676616 learned_addr learned my addr 172.21.15.67:0/769676616 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-03T21:36:58.608 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:36:58.600+0000 7ff3617ca700 1 --2- 172.21.15.67:0/769676616 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ff35c103540 0x7ff35c19f920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:36:58.608 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:36:58.601+0000 7ff361fcb700 1 --2- 172.21.15.67:0/769676616 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7ff35c1095e0 0x7ff35c199b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:36:58.608 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:36:58.601+0000 7ff360fc9700 1 -- 172.21.15.67:0/769676616 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7ff35c1095e0 msgr2=0x7ff35c199b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:36:58.608 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:36:58.601+0000 7ff360fc9700 1 --2- 172.21.15.67:0/769676616 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7ff35c1095e0 0x7ff35c199b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:36:58.608 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:36:58.601+0000 7ff360fc9700 1 -- 172.21.15.67:0/769676616 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ff35c103540 msgr2=0x7ff35c19f920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:36:58.608 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:36:58.601+0000 7ff360fc9700 1 --2- 172.21.15.67:0/769676616 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ff35c103540 0x7ff35c19f920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:36:58.608 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:36:58.601+0000 7ff360fc9700 1 -- 172.21.15.67:0/769676616 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff35c19a290 con 0x7ff35c103f00 2024-05-03T21:36:58.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:36:58.601+0000 7ff360fc9700 1 --2- 172.21.15.67:0/769676616 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff35c103f00 0x7ff35c19fe60 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7ff3580059d0 tx=0x7ff35800bb00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T21:36:58.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:36:58.601+0000 7ff3527fc700 1 -- 172.21.15.67:0/769676616 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff358027070 con 0x7ff35c103f00 2024-05-03T21:36:58.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:36:58.601+0000 7ff363a2e700 1 -- 172.21.15.67:0/769676616 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff35c19a520 con 0x7ff35c103f00 2024-05-03T21:36:58.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:36:58.601+0000 7ff3527fc700 1 -- 172.21.15.67:0/769676616 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(28 keys) ==== 1170+0+0 (secure 0 0 0) 0x7ff358003f70 con 0x7ff35c103f00 2024-05-03T21:36:58.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:36:58.601+0000 7ff3527fc700 1 -- 172.21.15.67:0/769676616 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff358022dd0 con 0x7ff35c103f00 2024-05-03T21:36:58.610 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:36:58.602+0000 7ff363a2e700 1 -- 172.21.15.67:0/769676616 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff35c1a67f0 con 0x7ff35c103f00 2024-05-03T21:36:58.611 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:36:58.603+0000 7ff363a2e700 1 -- 172.21.15.67:0/769676616 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff35c104c50 con 0x7ff35c103f00 2024-05-03T21:36:58.611 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:36:58.604+0000 7ff3527fc700 1 -- 172.21.15.67:0/769676616 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 17) ==== 149703+0+0 (secure 0 0 0) 0x7ff35802a070 con 0x7ff35c103f00 2024-05-03T21:36:58.611 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:36:58.605+0000 7ff3527fc700 1 --2- 172.21.15.67:0/769676616 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7ff3480b2990 0x7ff3480b4e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:36:58.612 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:36:58.605+0000 7ff3617ca700 1 --2- 172.21.15.67:0/769676616 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7ff3480b2990 0x7ff3480b4e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:36:58.612 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:36:58.605+0000 7ff3527fc700 1 -- 172.21.15.67:0/769676616 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 8211+0+0 (secure 0 0 0) 0x7ff3580e2e60 con 0x7ff35c103f00 2024-05-03T21:36:58.612 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:36:58.605+0000 7ff3617ca700 1 --2- 172.21.15.67:0/769676616 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7ff3480b2990 0x7ff3480b4e50 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7ff34c007be0 tx=0x7ff34c0073d0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T21:36:58.612 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:36:58.609+0000 7ff3527fc700 1 -- 172.21.15.67:0/769676616 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191839 (secure 0 0 0) 0x7ff358013070 con 0x7ff35c103f00 2024-05-03T21:36:58.878 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:36:58.876+0000 7ff363a2e700 1 -- 172.21.15.67:0/769676616 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7ff35c02e510 con 0x7ff35c103f00 2024-05-03T21:36:58.880 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:36:58.879+0000 7ff3527fc700 1 -- 172.21.15.67:0/769676616 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v27) ==== 161+0+0 (secure 0 0 0) 0x7ff3580aab50 con 0x7ff35c103f00 2024-05-03T21:36:58.883 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:36:58.881+0000 7ff363a2e700 1 -- 172.21.15.67:0/769676616 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7ff3480b2990 msgr2=0x7ff3480b4e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:36:58.883 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:36:58.881+0000 7ff363a2e700 1 --2- 172.21.15.67:0/769676616 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7ff3480b2990 0x7ff3480b4e50 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7ff34c007be0 tx=0x7ff34c0073d0 comp rx=0 tx=0).stop 2024-05-03T21:36:58.883 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:36:58.881+0000 7ff363a2e700 1 -- 172.21.15.67:0/769676616 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff35c103f00 msgr2=0x7ff35c19fe60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:36:58.884 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:36:58.881+0000 7ff363a2e700 1 --2- 172.21.15.67:0/769676616 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff35c103f00 0x7ff35c19fe60 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7ff3580059d0 tx=0x7ff35800bb00 comp rx=0 tx=0).stop 2024-05-03T21:36:58.884 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:36:58.881+0000 7ff363a2e700 1 -- 172.21.15.67:0/769676616 shutdown_connections 2024-05-03T21:36:58.884 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:36:58.881+0000 7ff363a2e700 1 --2- 172.21.15.67:0/769676616 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7ff3480b2990 0x7ff3480b4e50 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:36:58.884 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:36:58.881+0000 7ff363a2e700 1 --2- 172.21.15.67:0/769676616 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7ff35c1095e0 0x7ff35c199b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:36:58.884 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:36:58.881+0000 7ff363a2e700 1 --2- 172.21.15.67:0/769676616 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ff35c103540 0x7ff35c19f920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:36:58.884 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:36:58.881+0000 7ff363a2e700 1 --2- 172.21.15.67:0/769676616 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff35c103f00 0x7ff35c19fe60 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:36:58.884 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:36:58.881+0000 7ff363a2e700 1 -- 172.21.15.67:0/769676616 >> 172.21.15.67:0/769676616 conn(0x7ff35c07a650 msgr2=0x7ff35c1020b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T21:36:58.884 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:36:58.881+0000 7ff363a2e700 1 -- 172.21.15.67:0/769676616 shutdown_connections 2024-05-03T21:36:58.884 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:36:58.881+0000 7ff363a2e700 1 -- 172.21.15.67:0/769676616 wait complete. 2024-05-03T21:36:58.885 INFO:teuthology.orchestra.run.smithi067.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-05-03T21:36:58.971 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:36:58 smithi067 ceph-713bc57a-0991-11ef-bc97-c7b262605968-mon-a[29726]: 2024-05-03T21:36:58.877+0000 7f4b75adb700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-05-03T21:36:59.245 INFO:journalctl@ceph.mds.b.smithi082.stdout:May 03 21:36:58 smithi082 ceph-713bc57a-0991-11ef-bc97-c7b262605968-mds-b[63361]: ignoring --setuser ceph since I am not root 2024-05-03T21:36:59.245 INFO:journalctl@ceph.mds.b.smithi082.stdout:May 03 21:36:58 smithi082 ceph-713bc57a-0991-11ef-bc97-c7b262605968-mds-b[63361]: ignoring --setgroup ceph since I am not root 2024-05-03T21:36:59.245 INFO:journalctl@ceph.mds.b.smithi082.stdout:May 03 21:36:58 smithi082 ceph-mds[63365]: -- 172.21.15.82:0/2918877923 <== mon.2 v2:172.21.15.82:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55cc0bc13a40 con 0x55cc0bbe6800 2024-05-03T21:36:59.246 INFO:journalctl@ceph.mds.b.smithi082.stdout:May 03 21:36:58 smithi082 ceph-713bc57a-0991-11ef-bc97-c7b262605968-mds-b[63361]: starting mds.b at 2024-05-03T21:36:59.246 INFO:journalctl@ceph.mds.k.smithi082.stdout:May 03 21:36:58 smithi082 ceph-713bc57a-0991-11ef-bc97-c7b262605968-mds-k[62403]: ignoring --setuser ceph since I am not root 2024-05-03T21:36:59.246 INFO:journalctl@ceph.mds.k.smithi082.stdout:May 03 21:36:58 smithi082 ceph-713bc57a-0991-11ef-bc97-c7b262605968-mds-k[62403]: ignoring --setgroup ceph since I am not root 2024-05-03T21:36:59.246 INFO:journalctl@ceph.mds.k.smithi082.stdout:May 03 21:36:58 smithi082 ceph-mds[62407]: -- 172.21.15.82:0/3385067486 <== mon.2 v2:172.21.15.82:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55f7c7069a40 con 0x55f7c703c800 2024-05-03T21:36:59.246 INFO:journalctl@ceph.mds.k.smithi082.stdout:May 03 21:36:58 smithi082 ceph-713bc57a-0991-11ef-bc97-c7b262605968-mds-k[62403]: starting mds.k at 2024-05-03T21:36:59.247 INFO:journalctl@ceph.mds.h.smithi082.stdout:May 03 21:36:58 smithi082 ceph-713bc57a-0991-11ef-bc97-c7b262605968-mds-h[63050]: ignoring --setuser ceph since I am not root 2024-05-03T21:36:59.247 INFO:journalctl@ceph.mds.h.smithi082.stdout:May 03 21:36:58 smithi082 ceph-713bc57a-0991-11ef-bc97-c7b262605968-mds-h[63050]: ignoring --setgroup ceph since I am not root 2024-05-03T21:36:59.247 INFO:journalctl@ceph.mds.h.smithi082.stdout:May 03 21:36:58 smithi082 ceph-mds[63054]: -- 172.21.15.82:0/201552001 <== mon.0 v2:172.21.15.67:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x558eb62c3a40 con 0x558eb6297000 2024-05-03T21:36:59.247 INFO:journalctl@ceph.mds.h.smithi082.stdout:May 03 21:36:58 smithi082 ceph-713bc57a-0991-11ef-bc97-c7b262605968-mds-h[63050]: starting mds.h at 2024-05-03T21:36:59.247 INFO:journalctl@ceph.mds.e.smithi082.stdout:May 03 21:36:58 smithi082 ceph-713bc57a-0991-11ef-bc97-c7b262605968-mds-e[62726]: ignoring --setuser ceph since I am not root 2024-05-03T21:36:59.248 INFO:journalctl@ceph.mds.e.smithi082.stdout:May 03 21:36:58 smithi082 ceph-713bc57a-0991-11ef-bc97-c7b262605968-mds-e[62726]: ignoring --setgroup ceph since I am not root 2024-05-03T21:36:59.248 INFO:journalctl@ceph.mds.e.smithi082.stdout:May 03 21:36:58 smithi082 ceph-mds[62730]: -- 172.21.15.82:0/2678947320 <== mon.1 v2:172.21.15.138:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x561dea4fda40 con 0x561dea4d1000 2024-05-03T21:36:59.248 INFO:journalctl@ceph.mds.e.smithi082.stdout:May 03 21:36:58 smithi082 ceph-713bc57a-0991-11ef-bc97-c7b262605968-mds-e[62726]: starting mds.e at 2024-05-03T21:36:59.272 INFO:journalctl@ceph.mds.c.smithi138.stdout:May 03 21:36:58 smithi138 ceph-713bc57a-0991-11ef-bc97-c7b262605968-mds-c[63433]: ignoring --setuser ceph since I am not root 2024-05-03T21:36:59.272 INFO:journalctl@ceph.mds.c.smithi138.stdout:May 03 21:36:58 smithi138 ceph-713bc57a-0991-11ef-bc97-c7b262605968-mds-c[63433]: ignoring --setgroup ceph since I am not root 2024-05-03T21:36:59.273 INFO:journalctl@ceph.mds.c.smithi138.stdout:May 03 21:36:58 smithi138 ceph-mds[63437]: -- 172.21.15.138:0/809207976 <== mon.0 v2:172.21.15.67:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x560ed27cda40 con 0x560ed27a1000 2024-05-03T21:36:59.273 INFO:journalctl@ceph.mds.c.smithi138.stdout:May 03 21:36:58 smithi138 ceph-713bc57a-0991-11ef-bc97-c7b262605968-mds-c[63433]: starting mds.c at 2024-05-03T21:36:59.273 INFO:journalctl@ceph.mds.i.smithi138.stdout:May 03 21:36:58 smithi138 ceph-713bc57a-0991-11ef-bc97-c7b262605968-mds-i[63754]: ignoring --setuser ceph since I am not root 2024-05-03T21:36:59.273 INFO:journalctl@ceph.mds.i.smithi138.stdout:May 03 21:36:58 smithi138 ceph-713bc57a-0991-11ef-bc97-c7b262605968-mds-i[63754]: ignoring --setgroup ceph since I am not root 2024-05-03T21:36:59.273 INFO:journalctl@ceph.mds.i.smithi138.stdout:May 03 21:36:58 smithi138 ceph-mds[63758]: -- 172.21.15.138:0/4209007289 <== mon.1 v2:172.21.15.138:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x564223aafa40 con 0x564223a82800 2024-05-03T21:36:59.273 INFO:journalctl@ceph.mds.i.smithi138.stdout:May 03 21:36:58 smithi138 ceph-713bc57a-0991-11ef-bc97-c7b262605968-mds-i[63754]: starting mds.i at 2024-05-03T21:36:59.470 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:36:59 smithi067 ceph-mon[29730]: pgmap v836: 97 pgs: 97 active+clean; 638 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 426 B/s wr, 6 op/s 2024-05-03T21:36:59.471 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:36:59 smithi067 ceph-mon[29730]: from='client.? 172.21.15.67:0/769676616' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-05-03T21:36:59.471 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:36:59 smithi067 ceph-mon[29730]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-05-03T21:36:59.471 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:36:59 smithi067 ceph-mon[29730]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-05-03T21:36:59.471 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:36:59 smithi067 ceph-mon[29730]: from='client.? 172.21.15.67:0/769676616' entity='client.admin' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2024-05-03T21:36:59.471 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:36:59 smithi067 ceph-mon[29730]: osdmap e85: 12 total, 12 up, 12 in 2024-05-03T21:36:59.471 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:36:59 smithi067 ceph-mon[29730]: fsmap cephfs:0/3 6 up:standby, 3 failed 2024-05-03T21:36:59.501 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9a047fe7599a549042d06a3fb5b6813a203eabea shell --fsid 713bc57a-0991-11ef-bc97-c7b262605968 -- ceph fs rm cephfs --yes-i-really-mean-it 2024-05-03T21:36:59.574 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:36:59 smithi138 ceph-mon[33110]: pgmap v836: 97 pgs: 97 active+clean; 638 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 426 B/s wr, 6 op/s 2024-05-03T21:36:59.574 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:36:59 smithi138 ceph-mon[33110]: from='client.? 172.21.15.67:0/769676616' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-05-03T21:36:59.574 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:36:59 smithi138 ceph-mon[33110]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-05-03T21:36:59.574 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:36:59 smithi138 ceph-mon[33110]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-05-03T21:36:59.575 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:36:59 smithi138 ceph-mon[33110]: from='client.? 172.21.15.67:0/769676616' entity='client.admin' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2024-05-03T21:36:59.575 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:36:59 smithi138 ceph-mon[33110]: osdmap e85: 12 total, 12 up, 12 in 2024-05-03T21:36:59.575 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:36:59 smithi138 ceph-mon[33110]: fsmap cephfs:0/3 6 up:standby, 3 failed 2024-05-03T21:36:59.745 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:36:59 smithi082 ceph-mon[32933]: pgmap v836: 97 pgs: 97 active+clean; 638 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 426 B/s wr, 6 op/s 2024-05-03T21:36:59.745 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:36:59 smithi082 ceph-mon[32933]: from='client.? 172.21.15.67:0/769676616' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-05-03T21:36:59.745 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:36:59 smithi082 ceph-mon[32933]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-05-03T21:36:59.745 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:36:59 smithi082 ceph-mon[32933]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-05-03T21:36:59.746 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:36:59 smithi082 ceph-mon[32933]: from='client.? 172.21.15.67:0/769676616' entity='client.admin' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2024-05-03T21:36:59.746 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:36:59 smithi082 ceph-mon[32933]: osdmap e85: 12 total, 12 up, 12 in 2024-05-03T21:36:59.746 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:36:59 smithi082 ceph-mon[32933]: fsmap cephfs:0/3 6 up:standby, 3 failed 2024-05-03T21:36:59.753 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/713bc57a-0991-11ef-bc97-c7b262605968/mon.a/config 2024-05-03T21:37:01.077 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:37:00 smithi067 ceph-mon[29730]: mds.? [v2:172.21.15.138:6836/2397906387,v1:172.21.15.138:6838/2397906387] up:boot 2024-05-03T21:37:01.077 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:37:00 smithi067 ceph-mon[29730]: mds.? [v2:172.21.15.138:6837/316998271,v1:172.21.15.138:6839/316998271] up:boot 2024-05-03T21:37:01.077 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:37:00 smithi067 ceph-mon[29730]: mds.? [v2:172.21.15.82:6832/1542033966,v1:172.21.15.82:6834/1542033966] up:boot 2024-05-03T21:37:01.077 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:37:00 smithi067 ceph-mon[29730]: mds.? [v2:172.21.15.82:6833/1933446460,v1:172.21.15.82:6835/1933446460] up:boot 2024-05-03T21:37:01.078 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:37:00 smithi067 ceph-mon[29730]: mds.? [v2:172.21.15.82:6836/3027292830,v1:172.21.15.82:6838/3027292830] up:boot 2024-05-03T21:37:01.078 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:37:00 smithi067 ceph-mon[29730]: mds.? [v2:172.21.15.82:6837/3092360121,v1:172.21.15.82:6839/3092360121] up:boot 2024-05-03T21:37:01.078 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:37:00 smithi067 ceph-mon[29730]: fsmap cephfs:0/3 12 up:standby, 3 failed 2024-05-03T21:37:01.078 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:37:00 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-05-03T21:37:01.078 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:37:00 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-05-03T21:37:01.078 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:37:00 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-05-03T21:37:01.078 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:37:00 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-05-03T21:37:01.079 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:37:00 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-05-03T21:37:01.079 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:37:00 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-05-03T21:37:01.079 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:37:00 smithi067 ceph-mon[29730]: pgmap v838: 97 pgs: 97 active+clean; 638 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 511 B/s wr, 6 op/s 2024-05-03T21:37:01.245 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:37:00 smithi082 ceph-mon[32933]: mds.? [v2:172.21.15.138:6836/2397906387,v1:172.21.15.138:6838/2397906387] up:boot 2024-05-03T21:37:01.245 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:37:00 smithi082 ceph-mon[32933]: mds.? [v2:172.21.15.138:6837/316998271,v1:172.21.15.138:6839/316998271] up:boot 2024-05-03T21:37:01.245 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:37:00 smithi082 ceph-mon[32933]: mds.? [v2:172.21.15.82:6832/1542033966,v1:172.21.15.82:6834/1542033966] up:boot 2024-05-03T21:37:01.245 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:37:00 smithi082 ceph-mon[32933]: mds.? [v2:172.21.15.82:6833/1933446460,v1:172.21.15.82:6835/1933446460] up:boot 2024-05-03T21:37:01.246 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:37:00 smithi082 ceph-mon[32933]: mds.? [v2:172.21.15.82:6836/3027292830,v1:172.21.15.82:6838/3027292830] up:boot 2024-05-03T21:37:01.246 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:37:00 smithi082 ceph-mon[32933]: mds.? [v2:172.21.15.82:6837/3092360121,v1:172.21.15.82:6839/3092360121] up:boot 2024-05-03T21:37:01.246 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:37:00 smithi082 ceph-mon[32933]: fsmap cephfs:0/3 12 up:standby, 3 failed 2024-05-03T21:37:01.246 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:37:00 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-05-03T21:37:01.246 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:37:00 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-05-03T21:37:01.246 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:37:00 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-05-03T21:37:01.246 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:37:00 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-05-03T21:37:01.246 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:37:00 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-05-03T21:37:01.247 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:37:00 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-05-03T21:37:01.247 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:37:00 smithi082 ceph-mon[32933]: pgmap v838: 97 pgs: 97 active+clean; 638 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 511 B/s wr, 6 op/s 2024-05-03T21:37:01.315 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:37:01.313+0000 7f0706ddb700 1 -- 172.21.15.67:0/1921784175 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0700104130 msgr2=0x7f0700108140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:37:01.315 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:37:01.313+0000 7f0706ddb700 1 --2- 172.21.15.67:0/1921784175 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0700104130 0x7f0700108140 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f06f40099c0 tx=0x7f06f4009cd0 comp rx=0 tx=0).stop 2024-05-03T21:37:01.316 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:37:01.313+0000 7f0706ddb700 1 -- 172.21.15.67:0/1921784175 shutdown_connections 2024-05-03T21:37:01.316 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:37:01.313+0000 7f0706ddb700 1 --2- 172.21.15.67:0/1921784175 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f0700108870 0x7f07001103f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:37:01.316 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:37:01.313+0000 7f0706ddb700 1 --2- 172.21.15.67:0/1921784175 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f0700103780 0x7f0700103b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:37:01.316 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:37:01.313+0000 7f0706ddb700 1 --2- 172.21.15.67:0/1921784175 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0700104130 0x7f0700108140 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:37:01.316 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:37:01.313+0000 7f0706ddb700 1 -- 172.21.15.67:0/1921784175 >> 172.21.15.67:0/1921784175 conn(0x7f070007a610 msgr2=0x7f070007aa20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T21:37:01.316 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:37:01.313+0000 7f0706ddb700 1 -- 172.21.15.67:0/1921784175 shutdown_connections 2024-05-03T21:37:01.316 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:37:01.313+0000 7f0706ddb700 1 -- 172.21.15.67:0/1921784175 wait complete. 2024-05-03T21:37:01.316 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:37:01.314+0000 7f0706ddb700 1 Processor -- start 2024-05-03T21:37:01.317 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:37:01.314+0000 7f0706ddb700 1 -- start start 2024-05-03T21:37:01.318 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:37:01.314+0000 7f0706ddb700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0700103780 0x7f070019f940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:37:01.319 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:37:01.314+0000 7f0706ddb700 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f0700104130 0x7f070019fe80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:37:01.319 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:37:01.314+0000 7f0706ddb700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f0700108870 0x7f0700199b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:37:01.319 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:37:01.314+0000 7f0706ddb700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f07001138e0 con 0x7f0700103780 2024-05-03T21:37:01.319 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:37:01.314+0000 7f0706ddb700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f0700113760 con 0x7f0700108870 2024-05-03T21:37:01.319 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:37:01.314+0000 7f0706ddb700 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f0700113a60 con 0x7f0700104130 2024-05-03T21:37:01.319 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:37:01.314+0000 7f0704b77700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0700103780 0x7f070019f940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:37:01.319 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:37:01.314+0000 7f0704b77700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0700103780 0x7f070019f940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:39014/0 (socket says 172.21.15.67:39014) 2024-05-03T21:37:01.319 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:37:01.314+0000 7f0704b77700 1 -- 172.21.15.67:0/1016565407 learned_addr learned my addr 172.21.15.67:0/1016565407 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-03T21:37:01.320 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:37:01.314+0000 7f06fffff700 1 --2- 172.21.15.67:0/1016565407 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f0700104130 0x7f070019fe80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:37:01.320 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:37:01.314+0000 7f0705378700 1 --2- 172.21.15.67:0/1016565407 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f0700108870 0x7f0700199b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:37:01.320 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:37:01.315+0000 7f0704b77700 1 -- 172.21.15.67:0/1016565407 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f0700104130 msgr2=0x7f070019fe80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:37:01.320 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:37:01.315+0000 7f0704b77700 1 --2- 172.21.15.67:0/1016565407 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f0700104130 0x7f070019fe80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:37:01.320 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:37:01.315+0000 7f0704b77700 1 -- 172.21.15.67:0/1016565407 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f0700108870 msgr2=0x7f0700199b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:37:01.320 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:37:01.315+0000 7f0704b77700 1 --2- 172.21.15.67:0/1016565407 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f0700108870 0x7f0700199b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:37:01.320 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:37:01.315+0000 7f0704b77700 1 -- 172.21.15.67:0/1016565407 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f070019a260 con 0x7f0700103780 2024-05-03T21:37:01.321 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:37:01.315+0000 7f06fffff700 1 --2- 172.21.15.67:0/1016565407 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f0700104130 0x7f070019fe80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-03T21:37:01.321 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:37:01.315+0000 7f0704b77700 1 --2- 172.21.15.67:0/1016565407 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0700103780 0x7f070019f940 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f06f000c970 tx=0x7f06f000cd30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T21:37:01.321 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:37:01.315+0000 7f06fdffb700 1 -- 172.21.15.67:0/1016565407 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06f0007aa0 con 0x7f0700103780 2024-05-03T21:37:01.321 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:37:01.315+0000 7f06fdffb700 1 -- 172.21.15.67:0/1016565407 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(28 keys) ==== 1170+0+0 (secure 0 0 0) 0x7f06f0004d30 con 0x7f0700103780 2024-05-03T21:37:01.321 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:37:01.315+0000 7f06fdffb700 1 -- 172.21.15.67:0/1016565407 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06f000f450 con 0x7f0700103780 2024-05-03T21:37:01.321 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:37:01.315+0000 7f0706ddb700 1 -- 172.21.15.67:0/1016565407 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f070019a550 con 0x7f0700103780 2024-05-03T21:37:01.321 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:37:01.315+0000 7f0706ddb700 1 -- 172.21.15.67:0/1016565407 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f07001a6810 con 0x7f0700103780 2024-05-03T21:37:01.321 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:37:01.316+0000 7f0706ddb700 1 -- 172.21.15.67:0/1016565407 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f070006a190 con 0x7f0700103780 2024-05-03T21:37:01.322 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:37:01.318+0000 7f06fdffb700 1 -- 172.21.15.67:0/1016565407 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 17) ==== 149703+0+0 (secure 0 0 0) 0x7f06f0004750 con 0x7f0700103780 2024-05-03T21:37:01.322 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:37:01.318+0000 7f06fdffb700 1 --2- 172.21.15.67:0/1016565407 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f06e80b2990 0x7f06e80b4e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:37:01.323 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:37:01.318+0000 7f06fffff700 1 --2- 172.21.15.67:0/1016565407 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f06e80b2990 0x7f06e80b4e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:37:01.323 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:37:01.318+0000 7f06fdffb700 1 -- 172.21.15.67:0/1016565407 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 8469+0+0 (secure 0 0 0) 0x7f06f00e2620 con 0x7f0700103780 2024-05-03T21:37:01.323 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:37:01.319+0000 7f06fffff700 1 --2- 172.21.15.67:0/1016565407 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f06e80b2990 0x7f06e80b4e50 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f0700103060 tx=0x7f06f4005fd0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T21:37:01.323 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:37:01.322+0000 7f06fdffb700 1 -- 172.21.15.67:0/1016565407 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191839 (secure 0 0 0) 0x7f06f00aa180 con 0x7f0700103780 2024-05-03T21:37:01.324 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:37:00 smithi138 ceph-mon[33110]: mds.? [v2:172.21.15.138:6836/2397906387,v1:172.21.15.138:6838/2397906387] up:boot 2024-05-03T21:37:01.324 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:37:00 smithi138 ceph-mon[33110]: mds.? [v2:172.21.15.138:6837/316998271,v1:172.21.15.138:6839/316998271] up:boot 2024-05-03T21:37:01.324 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:37:00 smithi138 ceph-mon[33110]: mds.? [v2:172.21.15.82:6832/1542033966,v1:172.21.15.82:6834/1542033966] up:boot 2024-05-03T21:37:01.324 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:37:00 smithi138 ceph-mon[33110]: mds.? [v2:172.21.15.82:6833/1933446460,v1:172.21.15.82:6835/1933446460] up:boot 2024-05-03T21:37:01.324 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:37:00 smithi138 ceph-mon[33110]: mds.? [v2:172.21.15.82:6836/3027292830,v1:172.21.15.82:6838/3027292830] up:boot 2024-05-03T21:37:01.324 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:37:00 smithi138 ceph-mon[33110]: mds.? [v2:172.21.15.82:6837/3092360121,v1:172.21.15.82:6839/3092360121] up:boot 2024-05-03T21:37:01.324 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:37:00 smithi138 ceph-mon[33110]: fsmap cephfs:0/3 12 up:standby, 3 failed 2024-05-03T21:37:01.324 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:37:00 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-05-03T21:37:01.325 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:37:00 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-05-03T21:37:01.325 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:37:00 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-05-03T21:37:01.325 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:37:00 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-05-03T21:37:01.325 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:37:00 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-05-03T21:37:01.325 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:37:00 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-05-03T21:37:01.325 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:37:00 smithi138 ceph-mon[33110]: pgmap v838: 97 pgs: 97 active+clean; 638 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 511 B/s wr, 6 op/s 2024-05-03T21:37:01.584 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:37:01.583+0000 7f0706ddb700 1 -- 172.21.15.67:0/1016565407 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) -- 0x7f070019b020 con 0x7f0700103780 2024-05-03T21:37:01.894 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:37:01.893+0000 7f06fdffb700 1 -- 172.21.15.67:0/1016565407 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v29) ==== 104+0+0 (secure 0 0 0) 0x7f070019b020 con 0x7f0700103780 2024-05-03T21:37:01.897 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:37:01.895+0000 7f0706ddb700 1 -- 172.21.15.67:0/1016565407 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f06e80b2990 msgr2=0x7f06e80b4e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:37:01.897 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:37:01.895+0000 7f0706ddb700 1 --2- 172.21.15.67:0/1016565407 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f06e80b2990 0x7f06e80b4e50 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f0700103060 tx=0x7f06f4005fd0 comp rx=0 tx=0).stop 2024-05-03T21:37:01.897 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:37:01.895+0000 7f0706ddb700 1 -- 172.21.15.67:0/1016565407 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0700103780 msgr2=0x7f070019f940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:37:01.897 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:37:01.895+0000 7f0706ddb700 1 --2- 172.21.15.67:0/1016565407 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0700103780 0x7f070019f940 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f06f000c970 tx=0x7f06f000cd30 comp rx=0 tx=0).stop 2024-05-03T21:37:01.897 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:37:01.895+0000 7f0706ddb700 1 -- 172.21.15.67:0/1016565407 shutdown_connections 2024-05-03T21:37:01.898 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:37:01.895+0000 7f0706ddb700 1 --2- 172.21.15.67:0/1016565407 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f06e80b2990 0x7f06e80b4e50 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:37:01.898 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:37:01.895+0000 7f0706ddb700 1 --2- 172.21.15.67:0/1016565407 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0700103780 0x7f070019f940 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:37:01.898 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:37:01.895+0000 7f0706ddb700 1 --2- 172.21.15.67:0/1016565407 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f0700108870 0x7f0700199b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:37:01.898 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:37:01.895+0000 7f0706ddb700 1 --2- 172.21.15.67:0/1016565407 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f0700104130 0x7f070019fe80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:37:01.898 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:37:01.895+0000 7f0706ddb700 1 -- 172.21.15.67:0/1016565407 >> 172.21.15.67:0/1016565407 conn(0x7f070007a610 msgr2=0x7f0700101770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T21:37:01.898 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:37:01.895+0000 7f0706ddb700 1 -- 172.21.15.67:0/1016565407 shutdown_connections 2024-05-03T21:37:01.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:37:01.895+0000 7f0706ddb700 1 -- 172.21.15.67:0/1016565407 wait complete. 2024-05-03T21:37:02.211 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:37:01 smithi067 ceph-mon[29730]: from='client.? 172.21.15.67:0/1016565407' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-05-03T21:37:02.244 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:37:01 smithi082 ceph-mon[32933]: from='client.? 172.21.15.67:0/1016565407' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-05-03T21:37:02.324 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:37:01 smithi138 ceph-mon[33110]: from='client.? 172.21.15.67:0/1016565407' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-05-03T21:37:02.521 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9a047fe7599a549042d06a3fb5b6813a203eabea shell --fsid 713bc57a-0991-11ef-bc97-c7b262605968 -- ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-05-03T21:37:02.775 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/713bc57a-0991-11ef-bc97-c7b262605968/mon.a/config 2024-05-03T21:37:03.143 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:37:02 smithi067 ceph-mon[29730]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-05-03T21:37:03.144 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:37:02 smithi067 ceph-mon[29730]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-05-03T21:37:03.144 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:37:02 smithi067 ceph-mon[29730]: Cluster is now healthy 2024-05-03T21:37:03.144 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:37:02 smithi067 ceph-mon[29730]: from='client.? 172.21.15.67:0/1016565407' entity='client.admin' cmd='[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]': finished 2024-05-03T21:37:03.144 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:37:02 smithi067 ceph-mon[29730]: fsmap 12 up:standby 2024-05-03T21:37:03.144 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:37:02 smithi067 ceph-mon[29730]: pgmap v839: 97 pgs: 97 active+clean; 638 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 204 B/s wr, 5 op/s 2024-05-03T21:37:03.245 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:37:02 smithi082 ceph-mon[32933]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-05-03T21:37:03.245 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:37:02 smithi082 ceph-mon[32933]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-05-03T21:37:03.245 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:37:02 smithi082 ceph-mon[32933]: Cluster is now healthy 2024-05-03T21:37:03.245 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:37:02 smithi082 ceph-mon[32933]: from='client.? 172.21.15.67:0/1016565407' entity='client.admin' cmd='[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]': finished 2024-05-03T21:37:03.245 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:37:02 smithi082 ceph-mon[32933]: fsmap 12 up:standby 2024-05-03T21:37:03.246 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:37:02 smithi082 ceph-mon[32933]: pgmap v839: 97 pgs: 97 active+clean; 638 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 204 B/s wr, 5 op/s 2024-05-03T21:37:03.324 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:37:02 smithi138 ceph-mon[33110]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-05-03T21:37:03.324 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:37:02 smithi138 ceph-mon[33110]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-05-03T21:37:03.324 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:37:02 smithi138 ceph-mon[33110]: Cluster is now healthy 2024-05-03T21:37:03.324 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:37:02 smithi138 ceph-mon[33110]: from='client.? 172.21.15.67:0/1016565407' entity='client.admin' cmd='[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]': finished 2024-05-03T21:37:03.325 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:37:02 smithi138 ceph-mon[33110]: fsmap 12 up:standby 2024-05-03T21:37:03.325 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:37:02 smithi138 ceph-mon[33110]: pgmap v839: 97 pgs: 97 active+clean; 638 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 204 B/s wr, 5 op/s 2024-05-03T21:37:04.407 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:37:04.405+0000 7f231359e700 1 -- 172.21.15.67:0/1584702437 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2314103fc0 msgr2=0x7f2314108010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:37:04.408 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:37:04.405+0000 7f231359e700 1 --2- 172.21.15.67:0/1584702437 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2314103fc0 0x7f2314108010 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f23080099c0 tx=0x7f2308009cd0 comp rx=0 tx=0).stop 2024-05-03T21:37:04.408 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:37:04.405+0000 7f231359e700 1 -- 172.21.15.67:0/1584702437 shutdown_connections 2024-05-03T21:37:04.408 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:37:04.405+0000 7f231359e700 1 --2- 172.21.15.67:0/1584702437 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f23141086f0 0x7f23141102c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:37:04.408 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:37:04.405+0000 7f231359e700 1 --2- 172.21.15.67:0/1584702437 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f2314103610 0x7f23141039f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:37:04.408 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:37:04.405+0000 7f231359e700 1 --2- 172.21.15.67:0/1584702437 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2314103fc0 0x7f2314108010 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:37:04.408 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:37:04.405+0000 7f231359e700 1 -- 172.21.15.67:0/1584702437 >> 172.21.15.67:0/1584702437 conn(0x7f23140feec0 msgr2=0x7f23141012e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T21:37:04.408 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:37:04.406+0000 7f231359e700 1 -- 172.21.15.67:0/1584702437 shutdown_connections 2024-05-03T21:37:04.408 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:37:04.406+0000 7f231359e700 1 -- 172.21.15.67:0/1584702437 wait complete. 2024-05-03T21:37:04.410 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:37:04.406+0000 7f231359e700 1 Processor -- start 2024-05-03T21:37:04.410 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:37:04.406+0000 7f231359e700 1 -- start start 2024-05-03T21:37:04.411 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:37:04.407+0000 7f231359e700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f2314103610 0x7f231419f7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:37:04.411 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:37:04.407+0000 7f231359e700 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f2314103fc0 0x7f231419fd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:37:04.411 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:37:04.407+0000 7f231359e700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f23141086f0 0x7f23141999f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:37:04.411 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:37:04.407+0000 7f231359e700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f2314113810 con 0x7f23141086f0 2024-05-03T21:37:04.411 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:37:04.407+0000 7f231359e700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f2314113690 con 0x7f2314103610 2024-05-03T21:37:04.411 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:37:04.407+0000 7f231359e700 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f2314113990 con 0x7f2314103fc0 2024-05-03T21:37:04.411 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:37:04.407+0000 7f2312d9d700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f23141086f0 0x7f23141999f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:37:04.411 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:37:04.407+0000 7f2312d9d700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f23141086f0 0x7f23141999f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:39038/0 (socket says 172.21.15.67:39038) 2024-05-03T21:37:04.412 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:37:04.407+0000 7f2312d9d700 1 -- 172.21.15.67:0/2059452743 learned_addr learned my addr 172.21.15.67:0/2059452743 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-03T21:37:04.412 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:37:04.407+0000 7f231259c700 1 --2- 172.21.15.67:0/2059452743 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f2314103610 0x7f231419f7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:37:04.412 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:37:04.407+0000 7f2311d9b700 1 --2- 172.21.15.67:0/2059452743 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f2314103fc0 0x7f231419fd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:37:04.412 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:37:04.407+0000 7f2312d9d700 1 -- 172.21.15.67:0/2059452743 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f2314103fc0 msgr2=0x7f231419fd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:37:04.412 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:37:04.407+0000 7f2312d9d700 1 --2- 172.21.15.67:0/2059452743 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f2314103fc0 0x7f231419fd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:37:04.412 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:37:04.407+0000 7f2312d9d700 1 -- 172.21.15.67:0/2059452743 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f2314103610 msgr2=0x7f231419f7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:37:04.412 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:37:04.407+0000 7f2312d9d700 1 --2- 172.21.15.67:0/2059452743 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f2314103610 0x7f231419f7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:37:04.412 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:37:04.407+0000 7f2312d9d700 1 -- 172.21.15.67:0/2059452743 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f231419a150 con 0x7f23141086f0 2024-05-03T21:37:04.412 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:37:04.407+0000 7f2311d9b700 1 --2- 172.21.15.67:0/2059452743 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f2314103fc0 0x7f231419fd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-03T21:37:04.413 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:37:04.407+0000 7f2312d9d700 1 --2- 172.21.15.67:0/2059452743 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f23141086f0 0x7f23141999f0 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f230c00ada0 tx=0x7f230c00c330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T21:37:04.413 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:37:04.408+0000 7f23037fe700 1 -- 172.21.15.67:0/2059452743 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f230c017020 con 0x7f23141086f0 2024-05-03T21:37:04.413 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:37:04.408+0000 7f231359e700 1 -- 172.21.15.67:0/2059452743 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f231419a440 con 0x7f23141086f0 2024-05-03T21:37:04.413 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:37:04.408+0000 7f231359e700 1 -- 172.21.15.67:0/2059452743 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f23141a66b0 con 0x7f23141086f0 2024-05-03T21:37:04.413 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:37:04.408+0000 7f23037fe700 1 -- 172.21.15.67:0/2059452743 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(28 keys) ==== 1170+0+0 (secure 0 0 0) 0x7f230c00a0d0 con 0x7f23141086f0 2024-05-03T21:37:04.413 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:37:04.408+0000 7f23037fe700 1 -- 172.21.15.67:0/2059452743 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f230c005520 con 0x7f23141086f0 2024-05-03T21:37:04.413 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:37:04.409+0000 7f231359e700 1 -- 172.21.15.67:0/2059452743 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f22f4005330 con 0x7f23141086f0 2024-05-03T21:37:04.414 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:37:04.410+0000 7f23037fe700 1 -- 172.21.15.67:0/2059452743 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 17) ==== 149703+0+0 (secure 0 0 0) 0x7f230c004ad0 con 0x7f23141086f0 2024-05-03T21:37:04.414 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:37:04.411+0000 7f23037fe700 1 --2- 172.21.15.67:0/2059452743 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f22fc0b2910 0x7f22fc0b4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:37:04.414 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:37:04.411+0000 7f231259c700 1 --2- 172.21.15.67:0/2059452743 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f22fc0b2910 0x7f22fc0b4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:37:04.415 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:37:04.411+0000 7f23037fe700 1 -- 172.21.15.67:0/2059452743 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 8469+0+0 (secure 0 0 0) 0x7f230c0e1a90 con 0x7f23141086f0 2024-05-03T21:37:04.415 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:37:04.411+0000 7f231259c700 1 --2- 172.21.15.67:0/2059452743 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f22fc0b2910 0x7f22fc0b4dd0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f2314108420 tx=0x7f230400a380 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T21:37:04.417 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:37:04.415+0000 7f23037fe700 1 -- 172.21.15.67:0/2059452743 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191839 (secure 0 0 0) 0x7f230c021020 con 0x7f23141086f0 2024-05-03T21:37:04.629 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:37:04.628+0000 7f231359e700 1 -- 172.21.15.67:0/2059452743 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) -- 0x7f22f4005620 con 0x7f23141086f0 2024-05-03T21:37:05.190 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:37:05.188+0000 7f23037fe700 1 -- 172.21.15.67:0/2059452743 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v86) ==== 181+0+0 (secure 0 0 0) 0x7f230c0a95f0 con 0x7f23141086f0 2024-05-03T21:37:05.194 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:37:05.192+0000 7f231359e700 1 -- 172.21.15.67:0/2059452743 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f22fc0b2910 msgr2=0x7f22fc0b4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:37:05.194 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:37:05.192+0000 7f231359e700 1 --2- 172.21.15.67:0/2059452743 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f22fc0b2910 0x7f22fc0b4dd0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f2314108420 tx=0x7f230400a380 comp rx=0 tx=0).stop 2024-05-03T21:37:05.194 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:37:05.192+0000 7f231359e700 1 -- 172.21.15.67:0/2059452743 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f23141086f0 msgr2=0x7f23141999f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:37:05.194 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:37:05.192+0000 7f231359e700 1 --2- 172.21.15.67:0/2059452743 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f23141086f0 0x7f23141999f0 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f230c00ada0 tx=0x7f230c00c330 comp rx=0 tx=0).stop 2024-05-03T21:37:05.194 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:37:05.192+0000 7f231359e700 1 -- 172.21.15.67:0/2059452743 shutdown_connections 2024-05-03T21:37:05.195 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:37:05.192+0000 7f231359e700 1 --2- 172.21.15.67:0/2059452743 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f22fc0b2910 0x7f22fc0b4dd0 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:37:05.195 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:37:05.192+0000 7f231359e700 1 --2- 172.21.15.67:0/2059452743 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f23141086f0 0x7f23141999f0 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:37:05.195 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:37:05.192+0000 7f231359e700 1 --2- 172.21.15.67:0/2059452743 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f2314103610 0x7f231419f7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:37:05.195 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:37:05.192+0000 7f231359e700 1 --2- 172.21.15.67:0/2059452743 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f2314103fc0 0x7f231419fd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:37:05.195 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:37:05.192+0000 7f231359e700 1 -- 172.21.15.67:0/2059452743 >> 172.21.15.67:0/2059452743 conn(0x7f23140feec0 msgr2=0x7f23140ff8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T21:37:05.196 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:37:05.193+0000 7f231359e700 1 -- 172.21.15.67:0/2059452743 shutdown_connections 2024-05-03T21:37:05.196 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:37:05.193+0000 7f231359e700 1 -- 172.21.15.67:0/2059452743 wait complete. 2024-05-03T21:37:05.196 INFO:teuthology.orchestra.run.smithi067.stderr:pool 'cephfs_metadata' removed 2024-05-03T21:37:05.470 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:37:05 smithi067 ceph-mon[29730]: pgmap v840: 97 pgs: 97 active+clean; 638 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-05-03T21:37:05.471 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:37:05 smithi067 ceph-mon[29730]: from='client.? 172.21.15.67:0/2059452743' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-05-03T21:37:05.494 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:37:05 smithi082 ceph-mon[32933]: pgmap v840: 97 pgs: 97 active+clean; 638 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-05-03T21:37:05.495 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:37:05 smithi082 ceph-mon[32933]: from='client.? 172.21.15.67:0/2059452743' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-05-03T21:37:05.574 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:37:05 smithi138 ceph-mon[33110]: pgmap v840: 97 pgs: 97 active+clean; 638 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-05-03T21:37:05.574 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:37:05 smithi138 ceph-mon[33110]: from='client.? 172.21.15.67:0/2059452743' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-05-03T21:37:05.880 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9a047fe7599a549042d06a3fb5b6813a203eabea shell --fsid 713bc57a-0991-11ef-bc97-c7b262605968 -- ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-05-03T21:37:06.137 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/713bc57a-0991-11ef-bc97-c7b262605968/mon.a/config 2024-05-03T21:37:06.464 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:37:06 smithi067 ceph-mon[29730]: from='client.? 172.21.15.67:0/2059452743' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]': finished 2024-05-03T21:37:06.464 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:37:06 smithi067 ceph-mon[29730]: osdmap e86: 12 total, 12 up, 12 in 2024-05-03T21:37:06.495 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:37:06 smithi082 ceph-mon[32933]: from='client.? 172.21.15.67:0/2059452743' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]': finished 2024-05-03T21:37:06.495 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:37:06 smithi082 ceph-mon[32933]: osdmap e86: 12 total, 12 up, 12 in 2024-05-03T21:37:06.574 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:37:06 smithi138 ceph-mon[33110]: from='client.? 172.21.15.67:0/2059452743' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]': finished 2024-05-03T21:37:06.574 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:37:06 smithi138 ceph-mon[33110]: osdmap e86: 12 total, 12 up, 12 in 2024-05-03T21:37:07.459 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:37:07 smithi067 ceph-mon[29730]: pgmap v842: 65 pgs: 65 active+clean; 577 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-03T21:37:07.494 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:37:07 smithi082 ceph-mon[32933]: pgmap v842: 65 pgs: 65 active+clean; 577 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-03T21:37:07.574 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:37:07 smithi138 ceph-mon[33110]: pgmap v842: 65 pgs: 65 active+clean; 577 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-03T21:37:07.680 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:37:07.678+0000 7f5ab013b700 1 -- 172.21.15.67:0/3337380836 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5aa8103540 msgr2=0x7f5aa81039c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:37:07.680 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:37:07.678+0000 7f5ab013b700 1 --2- 172.21.15.67:0/3337380836 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5aa8103540 0x7f5aa81039c0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f5a980099c0 tx=0x7f5a98009cd0 comp rx=0 tx=0).stop 2024-05-03T21:37:07.680 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:37:07.678+0000 7f5ab013b700 1 -- 172.21.15.67:0/3337380836 shutdown_connections 2024-05-03T21:37:07.680 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:37:07.678+0000 7f5ab013b700 1 --2- 172.21.15.67:0/3337380836 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f5aa8103f00 0x7f5aa81103b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:37:07.681 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:37:07.678+0000 7f5ab013b700 1 --2- 172.21.15.67:0/3337380836 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f5aa81095e0 0x7f5aa81099c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:37:07.681 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:37:07.678+0000 7f5ab013b700 1 --2- 172.21.15.67:0/3337380836 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5aa8103540 0x7f5aa81039c0 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:37:07.681 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:37:07.678+0000 7f5ab013b700 1 -- 172.21.15.67:0/3337380836 >> 172.21.15.67:0/3337380836 conn(0x7f5aa807a650 msgr2=0x7f5aa807aa60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T21:37:07.681 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:37:07.678+0000 7f5ab013b700 1 -- 172.21.15.67:0/3337380836 shutdown_connections 2024-05-03T21:37:07.681 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:37:07.678+0000 7f5ab013b700 1 -- 172.21.15.67:0/3337380836 wait complete. 2024-05-03T21:37:07.683 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:37:07.679+0000 7f5ab013b700 1 Processor -- start 2024-05-03T21:37:07.683 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:37:07.679+0000 7f5ab013b700 1 -- start start 2024-05-03T21:37:07.683 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:37:07.679+0000 7f5ab013b700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5aa8103540 0x7f5aa8199dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:37:07.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:37:07.679+0000 7f5ab013b700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f5aa8103f00 0x7f5aa819a300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:37:07.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:37:07.679+0000 7f5ab013b700 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f5aa81095e0 0x7f5aa819e690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:37:07.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:37:07.679+0000 7f5ab013b700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f5aa81139d0 con 0x7f5aa8103540 2024-05-03T21:37:07.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:37:07.679+0000 7f5ab013b700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f5aa8113850 con 0x7f5aa8103f00 2024-05-03T21:37:07.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:37:07.679+0000 7f5ab013b700 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f5aa8113b50 con 0x7f5aa81095e0 2024-05-03T21:37:07.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:37:07.679+0000 7f5aaded7700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5aa8103540 0x7f5aa8199dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:37:07.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:37:07.679+0000 7f5aaded7700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5aa8103540 0x7f5aa8199dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:45786/0 (socket says 172.21.15.67:45786) 2024-05-03T21:37:07.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:37:07.679+0000 7f5aaded7700 1 -- 172.21.15.67:0/1254285272 learned_addr learned my addr 172.21.15.67:0/1254285272 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-03T21:37:07.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:37:07.679+0000 7f5aae6d8700 1 --2- 172.21.15.67:0/1254285272 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f5aa81095e0 0x7f5aa819e690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:37:07.685 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:37:07.680+0000 7f5aad6d6700 1 --2- 172.21.15.67:0/1254285272 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f5aa8103f00 0x7f5aa819a300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:37:07.685 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:37:07.680+0000 7f5aaded7700 1 -- 172.21.15.67:0/1254285272 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f5aa81095e0 msgr2=0x7f5aa819e690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:37:07.685 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:37:07.680+0000 7f5aaded7700 1 --2- 172.21.15.67:0/1254285272 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f5aa81095e0 0x7f5aa819e690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:37:07.685 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:37:07.680+0000 7f5aaded7700 1 -- 172.21.15.67:0/1254285272 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f5aa8103f00 msgr2=0x7f5aa819a300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:37:07.685 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:37:07.680+0000 7f5aaded7700 1 --2- 172.21.15.67:0/1254285272 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f5aa8103f00 0x7f5aa819a300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:37:07.685 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:37:07.680+0000 7f5aaded7700 1 -- 172.21.15.67:0/1254285272 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5aa819edd0 con 0x7f5aa8103540 2024-05-03T21:37:07.685 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:37:07.680+0000 7f5aae6d8700 1 --2- 172.21.15.67:0/1254285272 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f5aa81095e0 0x7f5aa819e690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-03T21:37:07.685 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:37:07.680+0000 7f5aaded7700 1 --2- 172.21.15.67:0/1254285272 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5aa8103540 0x7f5aa8199dc0 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f5aa400c920 tx=0x7f5aa400cce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T21:37:07.685 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:37:07.680+0000 7f5a9effd700 1 -- 172.21.15.67:0/1254285272 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5aa4007b00 con 0x7f5aa8103540 2024-05-03T21:37:07.686 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:37:07.680+0000 7f5ab013b700 1 -- 172.21.15.67:0/1254285272 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5aa819f0c0 con 0x7f5aa8103540 2024-05-03T21:37:07.686 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:37:07.680+0000 7f5a9effd700 1 -- 172.21.15.67:0/1254285272 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(28 keys) ==== 1170+0+0 (secure 0 0 0) 0x7f5aa4004d30 con 0x7f5aa8103540 2024-05-03T21:37:07.686 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:37:07.680+0000 7f5a9effd700 1 -- 172.21.15.67:0/1254285272 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5aa40056e0 con 0x7f5aa8103540 2024-05-03T21:37:07.686 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:37:07.680+0000 7f5ab013b700 1 -- 172.21.15.67:0/1254285272 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5aa81a6940 con 0x7f5aa8103540 2024-05-03T21:37:07.686 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:37:07.681+0000 7f5ab013b700 1 -- 172.21.15.67:0/1254285272 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5aa8104c50 con 0x7f5aa8103540 2024-05-03T21:37:07.687 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:37:07.682+0000 7f5a9effd700 1 -- 172.21.15.67:0/1254285272 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 17) ==== 149703+0+0 (secure 0 0 0) 0x7f5aa4007ca0 con 0x7f5aa8103540 2024-05-03T21:37:07.687 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:37:07.683+0000 7f5a9effd700 1 --2- 172.21.15.67:0/1254285272 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f5a940b2910 0x7f5a940b4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T21:37:07.687 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:37:07.683+0000 7f5aad6d6700 1 --2- 172.21.15.67:0/1254285272 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f5a940b2910 0x7f5a940b4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T21:37:07.687 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:37:07.684+0000 7f5a9effd700 1 -- 172.21.15.67:0/1254285272 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 7915+0+0 (secure 0 0 0) 0x7f5aa40e22a0 con 0x7f5aa8103540 2024-05-03T21:37:07.687 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:37:07.684+0000 7f5aad6d6700 1 --2- 172.21.15.67:0/1254285272 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f5a940b2910 0x7f5a940b4dd0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f5aa807abe0 tx=0x7f5a98005fd0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T21:37:07.688 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:37:07.687+0000 7f5a9effd700 1 -- 172.21.15.67:0/1254285272 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191839 (secure 0 0 0) 0x7f5aa4014030 con 0x7f5aa8103540 2024-05-03T21:37:07.905 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:37:07.901+0000 7f5ab013b700 1 -- 172.21.15.67:0/1254285272 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) -- 0x7f5aa819b120 con 0x7f5aa8103540 2024-05-03T21:37:08.210 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:37:08.208+0000 7f5a9effd700 1 -- 172.21.15.67:0/1254285272 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v87) ==== 169+0+0 (secure 0 0 0) 0x7f5aa40a9f50 con 0x7f5aa8103540 2024-05-03T21:37:08.212 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:37:08.210+0000 7f5ab013b700 1 -- 172.21.15.67:0/1254285272 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f5a940b2910 msgr2=0x7f5a940b4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:37:08.213 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:37:08.210+0000 7f5ab013b700 1 --2- 172.21.15.67:0/1254285272 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f5a940b2910 0x7f5a940b4dd0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f5aa807abe0 tx=0x7f5a98005fd0 comp rx=0 tx=0).stop 2024-05-03T21:37:08.213 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:37:08.210+0000 7f5ab013b700 1 -- 172.21.15.67:0/1254285272 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5aa8103540 msgr2=0x7f5aa8199dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T21:37:08.213 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:37:08.210+0000 7f5ab013b700 1 --2- 172.21.15.67:0/1254285272 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5aa8103540 0x7f5aa8199dc0 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f5aa400c920 tx=0x7f5aa400cce0 comp rx=0 tx=0).stop 2024-05-03T21:37:08.213 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:37:08.211+0000 7f5ab013b700 1 -- 172.21.15.67:0/1254285272 shutdown_connections 2024-05-03T21:37:08.213 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:37:08.211+0000 7f5ab013b700 1 --2- 172.21.15.67:0/1254285272 >> [v2:172.21.15.67:6800/604451592,v1:172.21.15.67:6801/604451592] conn(0x7f5a940b2910 0x7f5a940b4dd0 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:37:08.213 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:37:08.211+0000 7f5ab013b700 1 --2- 172.21.15.67:0/1254285272 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5aa8103540 0x7f5aa8199dc0 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:37:08.213 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:37:08.211+0000 7f5ab013b700 1 --2- 172.21.15.67:0/1254285272 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f5aa81095e0 0x7f5aa819e690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:37:08.214 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:37:08.211+0000 7f5ab013b700 1 --2- 172.21.15.67:0/1254285272 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f5aa8103f00 0x7f5aa819a300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T21:37:08.214 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:37:08.211+0000 7f5ab013b700 1 -- 172.21.15.67:0/1254285272 >> 172.21.15.67:0/1254285272 conn(0x7f5aa807a650 msgr2=0x7f5aa8102000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T21:37:08.214 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:37:08.211+0000 7f5ab013b700 1 -- 172.21.15.67:0/1254285272 shutdown_connections 2024-05-03T21:37:08.214 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T21:37:08.211+0000 7f5ab013b700 1 -- 172.21.15.67:0/1254285272 wait complete. 2024-05-03T21:37:08.214 INFO:teuthology.orchestra.run.smithi067.stderr:pool 'cephfs_data' removed 2024-05-03T21:37:08.470 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:37:08 smithi067 ceph-mon[29730]: from='client.? 172.21.15.67:0/1254285272' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2024-05-03T21:37:08.494 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:37:08 smithi082 ceph-mon[32933]: from='client.? 172.21.15.67:0/1254285272' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2024-05-03T21:37:08.573 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:37:08 smithi138 ceph-mon[33110]: from='client.? 172.21.15.67:0/1254285272' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2024-05-03T21:37:08.794 DEBUG:teuthology.orchestra.run.smithi067:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-05-03T21:37:08.822 DEBUG:teuthology.orchestra.run.smithi082:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-05-03T21:37:08.849 DEBUG:teuthology.orchestra.run.smithi138:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-05-03T21:37:08.875 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2024-05-03T21:37:08.875 DEBUG:teuthology.orchestra.run.smithi067:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-05-03T21:37:08.890 DEBUG:teuthology.orchestra.run.smithi082:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-05-03T21:37:08.905 DEBUG:teuthology.orchestra.run.smithi138:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-05-03T21:37:08.930 INFO:tasks.cephadm:Stopping all daemons... 2024-05-03T21:37:08.930 INFO:tasks.cephadm.mon.a:Stopping mon.a... 2024-05-03T21:37:08.930 DEBUG:teuthology.orchestra.run.smithi067:> sudo systemctl stop ceph-713bc57a-0991-11ef-bc97-c7b262605968@mon.a 2024-05-03T21:37:09.125 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:37:08 smithi067 systemd[1]: Stopping Ceph mon.a for 713bc57a-0991-11ef-bc97-c7b262605968... 2024-05-03T21:37:09.434 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:37:09 smithi067 ceph-mon[29730]: pgmap v843: 65 pgs: 65 active+clean; 577 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-03T21:37:09.434 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:37:09 smithi067 ceph-mon[29730]: from='client.? 172.21.15.67:0/1254285272' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]': finished 2024-05-03T21:37:09.434 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:37:09 smithi067 ceph-mon[29730]: osdmap e87: 12 total, 12 up, 12 in 2024-05-03T21:37:09.434 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:37:09 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T21:37:09.434 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:37:09 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-05-03T21:37:09.434 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:37:09 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-05-03T21:37:09.435 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:37:09 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-05-03T21:37:09.435 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:37:09 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-05-03T21:37:09.435 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:37:09 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T21:37:09.435 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:37:09 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T21:37:09.435 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:37:09 smithi067 ceph-mon[29730]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:37:09.435 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:37:09 smithi067 ceph-713bc57a-0991-11ef-bc97-c7b262605968-mon-a[29726]: 2024-05-03T21:37:09.227+0000 7f4b7b2e6700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-05-03T21:37:09.435 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:37:09 smithi067 ceph-713bc57a-0991-11ef-bc97-c7b262605968-mon-a[29726]: 2024-05-03T21:37:09.227+0000 7f4b7b2e6700 -1 mon.a@0(leader) e3 *** Got Signal Terminated *** 2024-05-03T21:37:09.435 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:37:09 smithi067 podman[87307]: 2024-05-03 21:37:09.432829152 +0000 UTC m=+0.299884996 container died a7aeb235f8119c2c2a3a62abe67c53ebef7f232f2064adc35acb1147ad41d33f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9a047fe7599a549042d06a3fb5b6813a203eabea, name=ceph-713bc57a-0991-11ef-bc97-c7b262605968-mon-a, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-rishabh-testing-20240501.193033-9a047fe, org.label-schema.build-date=20240430, org.label-schema.vendor=CentOS, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, io.buildah.version=1.33.5, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0) 2024-05-03T21:37:09.495 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:37:09 smithi082 ceph-mon[32933]: pgmap v843: 65 pgs: 65 active+clean; 577 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-03T21:37:09.495 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:37:09 smithi082 ceph-mon[32933]: from='client.? 172.21.15.67:0/1254285272' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]': finished 2024-05-03T21:37:09.495 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:37:09 smithi082 ceph-mon[32933]: osdmap e87: 12 total, 12 up, 12 in 2024-05-03T21:37:09.495 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:37:09 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T21:37:09.496 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:37:09 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-05-03T21:37:09.496 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:37:09 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-05-03T21:37:09.496 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:37:09 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-05-03T21:37:09.496 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:37:09 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-05-03T21:37:09.496 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:37:09 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T21:37:09.496 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:37:09 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T21:37:09.496 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:37:09 smithi082 ceph-mon[32933]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:37:09.574 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:37:09 smithi138 ceph-mon[33110]: pgmap v843: 65 pgs: 65 active+clean; 577 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-03T21:37:09.574 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:37:09 smithi138 ceph-mon[33110]: from='client.? 172.21.15.67:0/1254285272' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]': finished 2024-05-03T21:37:09.574 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:37:09 smithi138 ceph-mon[33110]: osdmap e87: 12 total, 12 up, 12 in 2024-05-03T21:37:09.575 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:37:09 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T21:37:09.575 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:37:09 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-05-03T21:37:09.575 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:37:09 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-05-03T21:37:09.575 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:37:09 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-05-03T21:37:09.575 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:37:09 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-05-03T21:37:09.575 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:37:09 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T21:37:09.575 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:37:09 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T21:37:09.575 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:37:09 smithi138 ceph-mon[33110]: from='mgr.14150 172.21.15.67:0/2930277331' entity='mgr.x' 2024-05-03T21:37:09.970 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:37:09 smithi067 podman[87307]: 2024-05-03 21:37:09.811569701 +0000 UTC m=+0.678625553 container cleanup a7aeb235f8119c2c2a3a62abe67c53ebef7f232f2064adc35acb1147ad41d33f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9a047fe7599a549042d06a3fb5b6813a203eabea, name=ceph-713bc57a-0991-11ef-bc97-c7b262605968-mon-a, GIT_BRANCH=HEAD, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, ceph=True, org.label-schema.build-date=20240430, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, RELEASE=wip-rishabh-testing-20240501.193033-9a047fe) 2024-05-03T21:37:09.971 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 21:37:09 smithi067 bash[87307]: ceph-713bc57a-0991-11ef-bc97-c7b262605968-mon-a 2024-05-03T21:37:10.233 DEBUG:teuthology.orchestra.run.smithi067:> sudo pkill -f 'journalctl -f -n 0 -u ceph-713bc57a-0991-11ef-bc97-c7b262605968@mon.a.service' 2024-05-03T21:37:10.279 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-03T21:37:10.280 INFO:tasks.cephadm.mon.a:Stopped mon.a 2024-05-03T21:37:10.280 INFO:tasks.cephadm.mon.c:Stopping mon.b... 2024-05-03T21:37:10.280 DEBUG:teuthology.orchestra.run.smithi082:> sudo systemctl stop ceph-713bc57a-0991-11ef-bc97-c7b262605968@mon.b 2024-05-03T21:37:10.599 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:37:10 smithi082 systemd[1]: Stopping Ceph mon.b for 713bc57a-0991-11ef-bc97-c7b262605968... 2024-05-03T21:37:10.890 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:37:10 smithi082 ceph-713bc57a-0991-11ef-bc97-c7b262605968-mon-b[32928]: 2024-05-03T21:37:10.596+0000 7fdbf9413700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-05-03T21:37:10.891 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:37:10 smithi082 ceph-713bc57a-0991-11ef-bc97-c7b262605968-mon-b[32928]: 2024-05-03T21:37:10.596+0000 7fdbf9413700 -1 mon.b@2(peon) e3 *** Got Signal Terminated *** 2024-05-03T21:37:10.891 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:37:10 smithi082 podman[67973]: 2024-05-03 21:37:10.638374975 +0000 UTC m=+0.158474079 container died 072030cb27ff879c71ca965ad2737a32f1aa1ab1aff9871ee66645c81ad90028 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:68e15ae152ef5c805c733e76da8fc1d3dd4ee669d5ba025f457e05725baa24dd, name=ceph-713bc57a-0991-11ef-bc97-c7b262605968-mon-b, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-rishabh-testing-20240501.193033-9a047fe, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.build-date=20240430, ceph=True) 2024-05-03T21:37:11.245 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:37:11 smithi082 podman[67973]: 2024-05-03 21:37:11.097535845 +0000 UTC m=+0.617634957 container cleanup 072030cb27ff879c71ca965ad2737a32f1aa1ab1aff9871ee66645c81ad90028 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:68e15ae152ef5c805c733e76da8fc1d3dd4ee669d5ba025f457e05725baa24dd, name=ceph-713bc57a-0991-11ef-bc97-c7b262605968-mon-b, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, RELEASE=wip-rishabh-testing-20240501.193033-9a047fe, ceph=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240430, GIT_CLEAN=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD) 2024-05-03T21:37:11.245 INFO:journalctl@ceph.mon.b.smithi082.stdout:May 03 21:37:11 smithi082 bash[67973]: ceph-713bc57a-0991-11ef-bc97-c7b262605968-mon-b 2024-05-03T21:37:11.513 DEBUG:teuthology.orchestra.run.smithi082:> sudo pkill -f 'journalctl -f -n 0 -u ceph-713bc57a-0991-11ef-bc97-c7b262605968@mon.b.service' 2024-05-03T21:37:11.555 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-03T21:37:11.555 INFO:tasks.cephadm.mon.c:Stopped mon.b 2024-05-03T21:37:11.556 INFO:tasks.cephadm.mon.c:Stopping mon.c... 2024-05-03T21:37:11.556 DEBUG:teuthology.orchestra.run.smithi138:> sudo systemctl stop ceph-713bc57a-0991-11ef-bc97-c7b262605968@mon.c 2024-05-03T21:37:11.837 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:37:11 smithi138 systemd[1]: Stopping Ceph mon.c for 713bc57a-0991-11ef-bc97-c7b262605968... 2024-05-03T21:37:12.324 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:37:11 smithi138 ceph-713bc57a-0991-11ef-bc97-c7b262605968-mon-c[33106]: 2024-05-03T21:37:11.834+0000 7f4aa473e700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-05-03T21:37:12.324 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:37:11 smithi138 ceph-713bc57a-0991-11ef-bc97-c7b262605968-mon-c[33106]: 2024-05-03T21:37:11.834+0000 7f4aa473e700 -1 mon.c@1(peon) e3 *** Got Signal Terminated *** 2024-05-03T21:37:12.325 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:37:12 smithi138 podman[68393]: 2024-05-03 21:37:12.070577608 +0000 UTC m=+0.322076274 container died b833bba2b5cf358958269b1691b789d1894d99c30b42d1dbccfc1a54ea56a1fb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:68e15ae152ef5c805c733e76da8fc1d3dd4ee669d5ba025f457e05725baa24dd, name=ceph-713bc57a-0991-11ef-bc97-c7b262605968-mon-c, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.schema-version=1.0, RELEASE=wip-rishabh-testing-20240501.193033-9a047fe, org.label-schema.build-date=20240430, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, ceph=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-05-03T21:37:12.813 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:37:12 smithi138 podman[68393]: 2024-05-03 21:37:12.527844621 +0000 UTC m=+0.779343329 container cleanup b833bba2b5cf358958269b1691b789d1894d99c30b42d1dbccfc1a54ea56a1fb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:68e15ae152ef5c805c733e76da8fc1d3dd4ee669d5ba025f457e05725baa24dd, name=ceph-713bc57a-0991-11ef-bc97-c7b262605968-mon-c, ceph=True, GIT_CLEAN=True, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240430, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-rishabh-testing-20240501.193033-9a047fe, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b) 2024-05-03T21:37:12.813 INFO:journalctl@ceph.mon.c.smithi138.stdout:May 03 21:37:12 smithi138 bash[68393]: ceph-713bc57a-0991-11ef-bc97-c7b262605968-mon-c 2024-05-03T21:37:12.945 DEBUG:teuthology.orchestra.run.smithi138:> sudo pkill -f 'journalctl -f -n 0 -u ceph-713bc57a-0991-11ef-bc97-c7b262605968@mon.c.service' 2024-05-03T21:37:12.987 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-03T21:37:12.987 INFO:tasks.cephadm.mon.c:Stopped mon.c 2024-05-03T21:37:12.988 INFO:tasks.cephadm.mgr.x:Stopping mgr.x... 2024-05-03T21:37:12.988 DEBUG:teuthology.orchestra.run.smithi067:> sudo systemctl stop ceph-713bc57a-0991-11ef-bc97-c7b262605968@mgr.x 2024-05-03T21:37:13.284 INFO:journalctl@ceph.mgr.x.smithi067.stdout:May 03 21:37:13 smithi067 systemd[1]: Stopping Ceph mgr.x for 713bc57a-0991-11ef-bc97-c7b262605968... 2024-05-03T21:37:13.551 INFO:journalctl@ceph.mgr.x.smithi067.stdout:May 03 21:37:13 smithi067 ceph-713bc57a-0991-11ef-bc97-c7b262605968-mgr-x[29964]: 2024-05-03T21:37:13.281+0000 7f141dfc0700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mgr -n mgr.x -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-03T21:37:13.552 INFO:journalctl@ceph.mgr.x.smithi067.stdout:May 03 21:37:13 smithi067 podman[87428]: 2024-05-03 21:37:13.313482231 +0000 UTC m=+0.118536849 container died c7ff7c041cc782e833bd5f17e8627af26a5d2cd4b083a8b7ac36999ce800629b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9a047fe7599a549042d06a3fb5b6813a203eabea, name=ceph-713bc57a-0991-11ef-bc97-c7b262605968-mgr-x, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240430, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, RELEASE=wip-rishabh-testing-20240501.193033-9a047fe, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2) 2024-05-03T21:37:13.885 INFO:journalctl@ceph.mgr.x.smithi067.stdout:May 03 21:37:13 smithi067 podman[87428]: 2024-05-03 21:37:13.641121415 +0000 UTC m=+0.446176026 container cleanup c7ff7c041cc782e833bd5f17e8627af26a5d2cd4b083a8b7ac36999ce800629b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9a047fe7599a549042d06a3fb5b6813a203eabea, name=ceph-713bc57a-0991-11ef-bc97-c7b262605968-mgr-x, io.buildah.version=1.33.5, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, RELEASE=wip-rishabh-testing-20240501.193033-9a047fe, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.build-date=20240430, org.label-schema.schema-version=1.0) 2024-05-03T21:37:13.885 INFO:journalctl@ceph.mgr.x.smithi067.stdout:May 03 21:37:13 smithi067 bash[87428]: ceph-713bc57a-0991-11ef-bc97-c7b262605968-mgr-x 2024-05-03T21:37:14.017 DEBUG:teuthology.orchestra.run.smithi067:> sudo pkill -f 'journalctl -f -n 0 -u ceph-713bc57a-0991-11ef-bc97-c7b262605968@mgr.x.service' 2024-05-03T21:37:14.051 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-03T21:37:14.052 INFO:tasks.cephadm.mgr.x:Stopped mgr.x 2024-05-03T21:37:14.052 INFO:tasks.cephadm.mgr.y:Stopping mgr.y... 2024-05-03T21:37:14.052 DEBUG:teuthology.orchestra.run.smithi082:> sudo systemctl stop ceph-713bc57a-0991-11ef-bc97-c7b262605968@mgr.y 2024-05-03T21:37:14.365 INFO:journalctl@ceph.mgr.y.smithi082.stdout:May 03 21:37:14 smithi082 systemd[1]: Stopping Ceph mgr.y for 713bc57a-0991-11ef-bc97-c7b262605968... 2024-05-03T21:37:14.654 INFO:journalctl@ceph.mgr.y.smithi082.stdout:May 03 21:37:14 smithi082 podman[68099]: 2024-05-03 21:37:14.364407846 +0000 UTC m=+0.121052660 container died cea5d1d3105e675ee00f76cede266bc1e9010f7b555cb401dd7a0ff360c447c8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:68e15ae152ef5c805c733e76da8fc1d3dd4ee669d5ba025f457e05725baa24dd, name=ceph-713bc57a-0991-11ef-bc97-c7b262605968-mgr-y, GIT_CLEAN=True, org.label-schema.build-date=20240430, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , ceph=True, io.buildah.version=1.33.5, RELEASE=wip-rishabh-testing-20240501.193033-9a047fe, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=) 2024-05-03T21:37:14.995 INFO:journalctl@ceph.mgr.y.smithi082.stdout:May 03 21:37:14 smithi082 podman[68099]: 2024-05-03 21:37:14.726988539 +0000 UTC m=+0.483633395 container cleanup cea5d1d3105e675ee00f76cede266bc1e9010f7b555cb401dd7a0ff360c447c8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:68e15ae152ef5c805c733e76da8fc1d3dd4ee669d5ba025f457e05725baa24dd, name=ceph-713bc57a-0991-11ef-bc97-c7b262605968-mgr-y, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, ceph=True, RELEASE=wip-rishabh-testing-20240501.193033-9a047fe, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240430, io.buildah.version=1.33.5) 2024-05-03T21:37:14.995 INFO:journalctl@ceph.mgr.y.smithi082.stdout:May 03 21:37:14 smithi082 bash[68099]: ceph-713bc57a-0991-11ef-bc97-c7b262605968-mgr-y 2024-05-03T21:37:15.191 DEBUG:teuthology.orchestra.run.smithi082:> sudo pkill -f 'journalctl -f -n 0 -u ceph-713bc57a-0991-11ef-bc97-c7b262605968@mgr.y.service' 2024-05-03T21:37:15.225 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-03T21:37:15.225 INFO:tasks.cephadm.mgr.y:Stopped mgr.y 2024-05-03T21:37:15.226 INFO:tasks.cephadm.mgr.z:Stopping mgr.z... 2024-05-03T21:37:15.226 DEBUG:teuthology.orchestra.run.smithi138:> sudo systemctl stop ceph-713bc57a-0991-11ef-bc97-c7b262605968@mgr.z 2024-05-03T21:37:15.529 INFO:journalctl@ceph.mgr.z.smithi138.stdout:May 03 21:37:15 smithi138 systemd[1]: Stopping Ceph mgr.z for 713bc57a-0991-11ef-bc97-c7b262605968... 2024-05-03T21:37:15.817 INFO:journalctl@ceph.mgr.z.smithi138.stdout:May 03 21:37:15 smithi138 podman[68528]: 2024-05-03 21:37:15.536112338 +0000 UTC m=+0.122764700 container died b67b795589f4586dbbb621dcdb239448062167b48f6ea916c9cce4134e9f0324 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:68e15ae152ef5c805c733e76da8fc1d3dd4ee669d5ba025f457e05725baa24dd, name=ceph-713bc57a-0991-11ef-bc97-c7b262605968-mgr-z, RELEASE=wip-rishabh-testing-20240501.193033-9a047fe, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, ceph=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.build-date=20240430) 2024-05-03T21:37:16.074 INFO:journalctl@ceph.mgr.z.smithi138.stdout:May 03 21:37:15 smithi138 podman[68528]: 2024-05-03 21:37:15.89039138 +0000 UTC m=+0.477043739 container cleanup b67b795589f4586dbbb621dcdb239448062167b48f6ea916c9cce4134e9f0324 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:68e15ae152ef5c805c733e76da8fc1d3dd4ee669d5ba025f457e05725baa24dd, name=ceph-713bc57a-0991-11ef-bc97-c7b262605968-mgr-z, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-rishabh-testing-20240501.193033-9a047fe, org.label-schema.build-date=20240430, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-05-03T21:37:16.074 INFO:journalctl@ceph.mgr.z.smithi138.stdout:May 03 21:37:15 smithi138 bash[68528]: ceph-713bc57a-0991-11ef-bc97-c7b262605968-mgr-z 2024-05-03T21:37:16.354 DEBUG:teuthology.orchestra.run.smithi138:> sudo pkill -f 'journalctl -f -n 0 -u ceph-713bc57a-0991-11ef-bc97-c7b262605968@mgr.z.service' 2024-05-03T21:37:16.389 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-03T21:37:16.390 INFO:tasks.cephadm.mgr.z:Stopped mgr.z 2024-05-03T21:37:16.390 INFO:tasks.cephadm.osd.0:Stopping osd.0... 2024-05-03T21:37:16.390 DEBUG:teuthology.orchestra.run.smithi067:> sudo systemctl stop ceph-713bc57a-0991-11ef-bc97-c7b262605968@osd.0 2024-05-03T21:37:16.704 INFO:journalctl@ceph.osd.0.smithi067.stdout:May 03 21:37:16 smithi067 systemd[1]: Stopping Ceph osd.0 for 713bc57a-0991-11ef-bc97-c7b262605968... 2024-05-03T21:37:16.970 INFO:journalctl@ceph.osd.0.smithi067.stdout:May 03 21:37:16 smithi067 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-0[41836]: 2024-05-03T21:37:16.703+0000 7feac008d700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-03T21:37:16.971 INFO:journalctl@ceph.osd.0.smithi067.stdout:May 03 21:37:16 smithi067 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-0[41836]: 2024-05-03T21:37:16.703+0000 7feac008d700 -1 osd.0 87 *** Got signal Terminated *** 2024-05-03T21:37:16.971 INFO:journalctl@ceph.osd.0.smithi067.stdout:May 03 21:37:16 smithi067 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-0[41836]: 2024-05-03T21:37:16.703+0000 7feac008d700 -1 osd.0 87 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-05-03T21:37:22.342 INFO:journalctl@ceph.osd.0.smithi067.stdout:May 03 21:37:22 smithi067 podman[87551]: 2024-05-03 21:37:22.070373071 +0000 UTC m=+5.478163946 container died 29510a6802ef39eb214325f1c8f860a4b1063da62ebfc8281ecc7130de514018 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:68e15ae152ef5c805c733e76da8fc1d3dd4ee669d5ba025f457e05725baa24dd, name=ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-0, ceph=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.vendor=CentOS, RELEASE=wip-rishabh-testing-20240501.193033-9a047fe, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240430, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_CLEAN=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=) 2024-05-03T21:37:22.721 INFO:journalctl@ceph.osd.0.smithi067.stdout:May 03 21:37:22 smithi067 podman[87551]: 2024-05-03 21:37:22.437055497 +0000 UTC m=+5.844846383 container cleanup 29510a6802ef39eb214325f1c8f860a4b1063da62ebfc8281ecc7130de514018 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:68e15ae152ef5c805c733e76da8fc1d3dd4ee669d5ba025f457e05725baa24dd, name=ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-0, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-rishabh-testing-20240501.193033-9a047fe, ceph=True, CEPH_POINT_RELEASE=, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_CLEAN=True, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.build-date=20240430, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD) 2024-05-03T21:37:22.721 INFO:journalctl@ceph.osd.0.smithi067.stdout:May 03 21:37:22 smithi067 bash[87551]: ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-0 2024-05-03T21:37:23.062 INFO:journalctl@ceph.osd.0.smithi067.stdout:May 03 21:37:22 smithi067 podman[87566]: 2024-05-03 21:37:22.730104701 +0000 UTC m=+0.654231817 container remove 29510a6802ef39eb214325f1c8f860a4b1063da62ebfc8281ecc7130de514018 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:68e15ae152ef5c805c733e76da8fc1d3dd4ee669d5ba025f457e05725baa24dd, name=ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-0, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, ceph=True, org.label-schema.build-date=20240430, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, RELEASE=wip-rishabh-testing-20240501.193033-9a047fe, maintainer=Guillaume Abrioux ) 2024-05-03T21:37:23.360 INFO:journalctl@ceph.osd.0.smithi067.stdout:May 03 21:37:23 smithi067 podman[87661]: 2024-05-03 21:37:22.964337569 +0000 UTC m=+0.018207403 image pull ee4d846202496ba6096593e8e8becc0700a5eb7ebd95536031187f5f337d4012 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:68e15ae152ef5c805c733e76da8fc1d3dd4ee669d5ba025f457e05725baa24dd 2024-05-03T21:37:23.360 INFO:journalctl@ceph.osd.0.smithi067.stdout:May 03 21:37:23 smithi067 podman[87661]: 2024-05-03 21:37:23.064479524 +0000 UTC m=+0.118349395 container create 939a1a19064730775b784e5114b67acc799306093ddcaf791466c2d4947fc5f2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:68e15ae152ef5c805c733e76da8fc1d3dd4ee669d5ba025f457e05725baa24dd, name=ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-0-deactivate, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_CLEAN=True, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_BRANCH=HEAD, RELEASE=wip-rishabh-testing-20240501.193033-9a047fe, org.label-schema.build-date=20240430, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2) 2024-05-03T21:37:23.360 INFO:journalctl@ceph.osd.0.smithi067.stdout:May 03 21:37:23 smithi067 podman[87661]: 2024-05-03 21:37:23.275401292 +0000 UTC m=+0.329271132 container init 939a1a19064730775b784e5114b67acc799306093ddcaf791466c2d4947fc5f2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:68e15ae152ef5c805c733e76da8fc1d3dd4ee669d5ba025f457e05725baa24dd, name=ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-0-deactivate, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-rishabh-testing-20240501.193033-9a047fe, org.label-schema.schema-version=1.0, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.build-date=20240430, GIT_CLEAN=True, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, ceph=True, maintainer=Guillaume Abrioux ) 2024-05-03T21:37:23.360 INFO:journalctl@ceph.osd.0.smithi067.stdout:May 03 21:37:23 smithi067 podman[87661]: 2024-05-03 21:37:23.278742106 +0000 UTC m=+0.332611931 container start 939a1a19064730775b784e5114b67acc799306093ddcaf791466c2d4947fc5f2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:68e15ae152ef5c805c733e76da8fc1d3dd4ee669d5ba025f457e05725baa24dd, name=ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-0-deactivate, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240430, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-rishabh-testing-20240501.193033-9a047fe, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, ceph=True, maintainer=Guillaume Abrioux ) 2024-05-03T21:37:23.360 INFO:journalctl@ceph.osd.0.smithi067.stdout:May 03 21:37:23 smithi067 podman[87661]: 2024-05-03 21:37:23.358441932 +0000 UTC m=+0.412311772 container attach 939a1a19064730775b784e5114b67acc799306093ddcaf791466c2d4947fc5f2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:68e15ae152ef5c805c733e76da8fc1d3dd4ee669d5ba025f457e05725baa24dd, name=ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-0-deactivate, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240430, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, RELEASE=wip-rishabh-testing-20240501.193033-9a047fe, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, ceph=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.name=CentOS Stream 8 Base Image) 2024-05-03T21:37:23.946 INFO:journalctl@ceph.osd.0.smithi067.stdout:May 03 21:37:23 smithi067 podman[87688]: 2024-05-03 21:37:23.621310474 +0000 UTC m=+0.020496670 container died 939a1a19064730775b784e5114b67acc799306093ddcaf791466c2d4947fc5f2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:68e15ae152ef5c805c733e76da8fc1d3dd4ee669d5ba025f457e05725baa24dd, name=ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-0-deactivate, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240430, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, io.buildah.version=1.33.5, RELEASE=wip-rishabh-testing-20240501.193033-9a047fe) 2024-05-03T21:37:24.200 INFO:journalctl@ceph.osd.0.smithi067.stdout:May 03 21:37:24 smithi067 podman[87688]: 2024-05-03 21:37:24.187167672 +0000 UTC m=+0.586353920 container remove 939a1a19064730775b784e5114b67acc799306093ddcaf791466c2d4947fc5f2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:68e15ae152ef5c805c733e76da8fc1d3dd4ee669d5ba025f457e05725baa24dd, name=ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-0-deactivate, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, RELEASE=wip-rishabh-testing-20240501.193033-9a047fe, ceph=True, GIT_CLEAN=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240430) 2024-05-03T21:37:24.205 DEBUG:teuthology.orchestra.run.smithi067:> sudo pkill -f 'journalctl -f -n 0 -u ceph-713bc57a-0991-11ef-bc97-c7b262605968@osd.0.service' 2024-05-03T21:37:24.247 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-03T21:37:24.247 INFO:tasks.cephadm.osd.0:Stopped osd.0 2024-05-03T21:37:24.247 INFO:tasks.cephadm.osd.1:Stopping osd.1... 2024-05-03T21:37:24.248 DEBUG:teuthology.orchestra.run.smithi082:> sudo systemctl stop ceph-713bc57a-0991-11ef-bc97-c7b262605968@osd.1 2024-05-03T21:37:24.565 INFO:journalctl@ceph.osd.1.smithi082.stdout:May 03 21:37:24 smithi082 systemd[1]: Stopping Ceph osd.1 for 713bc57a-0991-11ef-bc97-c7b262605968... 2024-05-03T21:37:24.995 INFO:journalctl@ceph.osd.1.smithi082.stdout:May 03 21:37:24 smithi082 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-1[37530]: 2024-05-03T21:37:24.563+0000 7ff962d2f700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-03T21:37:24.995 INFO:journalctl@ceph.osd.1.smithi082.stdout:May 03 21:37:24 smithi082 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-1[37530]: 2024-05-03T21:37:24.563+0000 7ff962d2f700 -1 osd.1 87 *** Got signal Terminated *** 2024-05-03T21:37:24.995 INFO:journalctl@ceph.osd.1.smithi082.stdout:May 03 21:37:24 smithi082 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-1[37530]: 2024-05-03T21:37:24.563+0000 7ff962d2f700 -1 osd.1 87 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-05-03T21:37:30.133 INFO:journalctl@ceph.osd.1.smithi082.stdout:May 03 21:37:29 smithi082 podman[68238]: 2024-05-03 21:37:29.878147461 +0000 UTC m=+5.428178504 container died a35a94fe3d8441a47967cf31e324f527f62c24cb92a0ec20cd6cfc76b2f65c3e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:68e15ae152ef5c805c733e76da8fc1d3dd4ee669d5ba025f457e05725baa24dd, name=ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-1, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-rishabh-testing-20240501.193033-9a047fe, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.build-date=20240430, GIT_CLEAN=True, CEPH_POINT_RELEASE=) 2024-05-03T21:37:30.415 INFO:journalctl@ceph.osd.1.smithi082.stdout:May 03 21:37:30 smithi082 podman[68238]: 2024-05-03 21:37:30.204661886 +0000 UTC m=+5.754692933 container cleanup a35a94fe3d8441a47967cf31e324f527f62c24cb92a0ec20cd6cfc76b2f65c3e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:68e15ae152ef5c805c733e76da8fc1d3dd4ee669d5ba025f457e05725baa24dd, name=ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-1, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.build-date=20240430, GIT_BRANCH=HEAD, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, RELEASE=wip-rishabh-testing-20240501.193033-9a047fe, io.buildah.version=1.33.5, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-05-03T21:37:30.415 INFO:journalctl@ceph.osd.1.smithi082.stdout:May 03 21:37:30 smithi082 bash[68238]: ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-1 2024-05-03T21:37:30.745 INFO:journalctl@ceph.osd.1.smithi082.stdout:May 03 21:37:30 smithi082 podman[68255]: 2024-05-03 21:37:30.414296584 +0000 UTC m=+0.533747012 container remove a35a94fe3d8441a47967cf31e324f527f62c24cb92a0ec20cd6cfc76b2f65c3e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:68e15ae152ef5c805c733e76da8fc1d3dd4ee669d5ba025f457e05725baa24dd, name=ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-1, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240430, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, RELEASE=wip-rishabh-testing-20240501.193033-9a047fe, io.buildah.version=1.33.5, GIT_CLEAN=True, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image) 2024-05-03T21:37:31.127 INFO:journalctl@ceph.osd.1.smithi082.stdout:May 03 21:37:30 smithi082 podman[68351]: 2024-05-03 21:37:30.729978449 +0000 UTC m=+0.018592091 image pull ee4d846202496ba6096593e8e8becc0700a5eb7ebd95536031187f5f337d4012 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:68e15ae152ef5c805c733e76da8fc1d3dd4ee669d5ba025f457e05725baa24dd 2024-05-03T21:37:31.127 INFO:journalctl@ceph.osd.1.smithi082.stdout:May 03 21:37:30 smithi082 podman[68351]: 2024-05-03 21:37:30.848424919 +0000 UTC m=+0.137038533 container create 4a2f7a98ecb26a647c0d3d4a9358216fa9fc1674a73209010a22649a54e0cff1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:68e15ae152ef5c805c733e76da8fc1d3dd4ee669d5ba025f457e05725baa24dd, name=ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-1-deactivate, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, maintainer=Guillaume Abrioux , RELEASE=wip-rishabh-testing-20240501.193033-9a047fe, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.build-date=20240430, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-05-03T21:37:31.127 INFO:journalctl@ceph.osd.1.smithi082.stdout:May 03 21:37:31 smithi082 podman[68351]: 2024-05-03 21:37:31.042473644 +0000 UTC m=+0.331087262 container init 4a2f7a98ecb26a647c0d3d4a9358216fa9fc1674a73209010a22649a54e0cff1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:68e15ae152ef5c805c733e76da8fc1d3dd4ee669d5ba025f457e05725baa24dd, name=ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-1-deactivate, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_CLEAN=True, org.label-schema.license=GPLv2, ceph=True, RELEASE=wip-rishabh-testing-20240501.193033-9a047fe, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240430) 2024-05-03T21:37:31.127 INFO:journalctl@ceph.osd.1.smithi082.stdout:May 03 21:37:31 smithi082 podman[68351]: 2024-05-03 21:37:31.04569818 +0000 UTC m=+0.334311796 container start 4a2f7a98ecb26a647c0d3d4a9358216fa9fc1674a73209010a22649a54e0cff1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:68e15ae152ef5c805c733e76da8fc1d3dd4ee669d5ba025f457e05725baa24dd, name=ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-1-deactivate, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240430, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, RELEASE=wip-rishabh-testing-20240501.193033-9a047fe, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_BRANCH=HEAD, GIT_CLEAN=True, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5) 2024-05-03T21:37:31.495 INFO:journalctl@ceph.osd.1.smithi082.stdout:May 03 21:37:31 smithi082 podman[68351]: 2024-05-03 21:37:31.125888669 +0000 UTC m=+0.414502292 container attach 4a2f7a98ecb26a647c0d3d4a9358216fa9fc1674a73209010a22649a54e0cff1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:68e15ae152ef5c805c733e76da8fc1d3dd4ee669d5ba025f457e05725baa24dd, name=ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-1-deactivate, RELEASE=wip-rishabh-testing-20240501.193033-9a047fe, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.build-date=20240430, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS) 2024-05-03T21:37:31.495 INFO:journalctl@ceph.osd.1.smithi082.stdout:May 03 21:37:31 smithi082 podman[68351]: 2024-05-03 21:37:31.345293167 +0000 UTC m=+0.633906787 container died 4a2f7a98ecb26a647c0d3d4a9358216fa9fc1674a73209010a22649a54e0cff1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:68e15ae152ef5c805c733e76da8fc1d3dd4ee669d5ba025f457e05725baa24dd, name=ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-1-deactivate, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240430, ceph=True, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-rishabh-testing-20240501.193033-9a047fe, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b) 2024-05-03T21:37:31.911 DEBUG:teuthology.orchestra.run.smithi082:> sudo pkill -f 'journalctl -f -n 0 -u ceph-713bc57a-0991-11ef-bc97-c7b262605968@osd.1.service' 2024-05-03T21:37:31.950 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-03T21:37:31.950 INFO:tasks.cephadm.osd.1:Stopped osd.1 2024-05-03T21:37:31.951 INFO:tasks.cephadm.osd.2:Stopping osd.2... 2024-05-03T21:37:31.951 DEBUG:teuthology.orchestra.run.smithi138:> sudo systemctl stop ceph-713bc57a-0991-11ef-bc97-c7b262605968@osd.2 2024-05-03T21:37:32.324 INFO:journalctl@ceph.osd.2.smithi138.stdout:May 03 21:37:31 smithi138 systemd[1]: Stopping Ceph osd.2 for 713bc57a-0991-11ef-bc97-c7b262605968... 2024-05-03T21:37:32.324 INFO:journalctl@ceph.osd.2.smithi138.stdout:May 03 21:37:32 smithi138 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-2[38260]: 2024-05-03T21:37:32.220+0000 7f86bab04700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-03T21:37:32.324 INFO:journalctl@ceph.osd.2.smithi138.stdout:May 03 21:37:32 smithi138 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-2[38260]: 2024-05-03T21:37:32.220+0000 7f86bab04700 -1 osd.2 87 *** Got signal Terminated *** 2024-05-03T21:37:32.324 INFO:journalctl@ceph.osd.2.smithi138.stdout:May 03 21:37:32 smithi138 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-2[38260]: 2024-05-03T21:37:32.220+0000 7f86bab04700 -1 osd.2 87 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-05-03T21:37:37.914 INFO:journalctl@ceph.osd.2.smithi138.stdout:May 03 21:37:37 smithi138 podman[68657]: 2024-05-03 21:37:37.624879449 +0000 UTC m=+5.485983285 container died 05849c3316c4c7932a213be172b7493495ec4d4f9781aa03352661d1122bd18d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:68e15ae152ef5c805c733e76da8fc1d3dd4ee669d5ba025f457e05725baa24dd, name=ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, io.buildah.version=1.33.5, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.build-date=20240430, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, RELEASE=wip-rishabh-testing-20240501.193033-9a047fe, CEPH_POINT_RELEASE=, GIT_CLEAN=True) 2024-05-03T21:37:38.230 INFO:journalctl@ceph.osd.2.smithi138.stdout:May 03 21:37:37 smithi138 podman[68657]: 2024-05-03 21:37:37.987023156 +0000 UTC m=+5.848126992 container cleanup 05849c3316c4c7932a213be172b7493495ec4d4f9781aa03352661d1122bd18d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:68e15ae152ef5c805c733e76da8fc1d3dd4ee669d5ba025f457e05725baa24dd, name=ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-2, io.buildah.version=1.33.5, ceph=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.build-date=20240430, org.label-schema.schema-version=1.0, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.vendor=CentOS, RELEASE=wip-rishabh-testing-20240501.193033-9a047fe, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True) 2024-05-03T21:37:38.230 INFO:journalctl@ceph.osd.2.smithi138.stdout:May 03 21:37:37 smithi138 bash[68657]: ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-2 2024-05-03T21:37:38.548 INFO:journalctl@ceph.osd.2.smithi138.stdout:May 03 21:37:38 smithi138 podman[68674]: 2024-05-03 21:37:38.229669031 +0000 UTC m=+0.604467212 container remove 05849c3316c4c7932a213be172b7493495ec4d4f9781aa03352661d1122bd18d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:68e15ae152ef5c805c733e76da8fc1d3dd4ee669d5ba025f457e05725baa24dd, name=ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-2, GIT_BRANCH=HEAD, ceph=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240430, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.license=GPLv2, RELEASE=wip-rishabh-testing-20240501.193033-9a047fe, GIT_CLEAN=True, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5) 2024-05-03T21:37:38.824 INFO:journalctl@ceph.osd.2.smithi138.stdout:May 03 21:37:38 smithi138 podman[68762]: 2024-05-03 21:37:38.547243751 +0000 UTC m=+0.108207500 container create 7877ffa707c7a6849342f80be75e0c5999985196eec35219f2883fa40923affc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:68e15ae152ef5c805c733e76da8fc1d3dd4ee669d5ba025f457e05725baa24dd, name=ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-2-deactivate, org.label-schema.build-date=20240430, RELEASE=wip-rishabh-testing-20240501.193033-9a047fe, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_BRANCH=HEAD) 2024-05-03T21:37:38.824 INFO:journalctl@ceph.osd.2.smithi138.stdout:May 03 21:37:38 smithi138 podman[68762]: 2024-05-03 21:37:38.456920302 +0000 UTC m=+0.017884083 image pull ee4d846202496ba6096593e8e8becc0700a5eb7ebd95536031187f5f337d4012 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:68e15ae152ef5c805c733e76da8fc1d3dd4ee669d5ba025f457e05725baa24dd 2024-05-03T21:37:38.824 INFO:journalctl@ceph.osd.2.smithi138.stdout:May 03 21:37:38 smithi138 podman[68762]: 2024-05-03 21:37:38.792146327 +0000 UTC m=+0.353110078 container init 7877ffa707c7a6849342f80be75e0c5999985196eec35219f2883fa40923affc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:68e15ae152ef5c805c733e76da8fc1d3dd4ee669d5ba025f457e05725baa24dd, name=ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-2-deactivate, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_CLEAN=True, CEPH_POINT_RELEASE=, ceph=True, RELEASE=wip-rishabh-testing-20240501.193033-9a047fe, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240430, org.label-schema.vendor=CentOS) 2024-05-03T21:37:38.824 INFO:journalctl@ceph.osd.2.smithi138.stdout:May 03 21:37:38 smithi138 podman[68762]: 2024-05-03 21:37:38.79582605 +0000 UTC m=+0.356789799 container start 7877ffa707c7a6849342f80be75e0c5999985196eec35219f2883fa40923affc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:68e15ae152ef5c805c733e76da8fc1d3dd4ee669d5ba025f457e05725baa24dd, name=ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-2-deactivate, org.label-schema.build-date=20240430, GIT_CLEAN=True, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, ceph=True, RELEASE=wip-rishabh-testing-20240501.193033-9a047fe) 2024-05-03T21:37:39.092 INFO:journalctl@ceph.osd.2.smithi138.stdout:May 03 21:37:38 smithi138 podman[68762]: 2024-05-03 21:37:38.87525181 +0000 UTC m=+0.436215560 container attach 7877ffa707c7a6849342f80be75e0c5999985196eec35219f2883fa40923affc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:68e15ae152ef5c805c733e76da8fc1d3dd4ee669d5ba025f457e05725baa24dd, name=ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-2-deactivate, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, RELEASE=wip-rishabh-testing-20240501.193033-9a047fe, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240430, ceph=True, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD) 2024-05-03T21:37:39.376 INFO:journalctl@ceph.osd.2.smithi138.stdout:May 03 21:37:39 smithi138 podman[68787]: 2024-05-03 21:37:39.126450891 +0000 UTC m=+0.027632691 container died 7877ffa707c7a6849342f80be75e0c5999985196eec35219f2883fa40923affc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:68e15ae152ef5c805c733e76da8fc1d3dd4ee669d5ba025f457e05725baa24dd, name=ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-2-deactivate, ceph=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.schema-version=1.0, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240430, org.label-schema.license=GPLv2, RELEASE=wip-rishabh-testing-20240501.193033-9a047fe) 2024-05-03T21:37:39.667 DEBUG:teuthology.orchestra.run.smithi138:> sudo pkill -f 'journalctl -f -n 0 -u ceph-713bc57a-0991-11ef-bc97-c7b262605968@osd.2.service' 2024-05-03T21:37:39.706 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-03T21:37:39.706 INFO:tasks.cephadm.osd.2:Stopped osd.2 2024-05-03T21:37:39.706 INFO:tasks.cephadm.osd.3:Stopping osd.3... 2024-05-03T21:37:39.707 DEBUG:teuthology.orchestra.run.smithi067:> sudo systemctl stop ceph-713bc57a-0991-11ef-bc97-c7b262605968@osd.3 2024-05-03T21:37:40.050 INFO:journalctl@ceph.osd.3.smithi067.stdout:May 03 21:37:39 smithi067 systemd[1]: Stopping Ceph osd.3 for 713bc57a-0991-11ef-bc97-c7b262605968... 2024-05-03T21:37:40.470 INFO:journalctl@ceph.osd.3.smithi067.stdout:May 03 21:37:40 smithi067 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-3[48631]: 2024-05-03T21:37:40.047+0000 7fa4b912c700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-03T21:37:40.471 INFO:journalctl@ceph.osd.3.smithi067.stdout:May 03 21:37:40 smithi067 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-3[48631]: 2024-05-03T21:37:40.047+0000 7fa4b912c700 -1 osd.3 87 *** Got signal Terminated *** 2024-05-03T21:37:40.471 INFO:journalctl@ceph.osd.3.smithi067.stdout:May 03 21:37:40 smithi067 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-3[48631]: 2024-05-03T21:37:40.047+0000 7fa4b912c700 -1 osd.3 87 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-05-03T21:37:43.245 INFO:journalctl@ceph.osd.10.smithi082.stdout:May 03 21:37:42 smithi082 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-10[57676]: 2024-05-03T21:37:42.970+0000 7f790b86c700 -1 osd.10 87 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-03T21:37:18.101114+0000 front 2024-05-03T21:37:18.101101+0000 (oldest deadline 2024-05-03T21:37:42.800953+0000) 2024-05-03T21:37:44.245 INFO:journalctl@ceph.osd.10.smithi082.stdout:May 03 21:37:43 smithi082 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-10[57676]: 2024-05-03T21:37:43.954+0000 7f790b86c700 -1 osd.10 87 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-03T21:37:18.101114+0000 front 2024-05-03T21:37:18.101101+0000 (oldest deadline 2024-05-03T21:37:42.800953+0000) 2024-05-03T21:37:44.720 INFO:journalctl@ceph.osd.3.smithi067.stdout:May 03 21:37:44 smithi067 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-3[48631]: 2024-05-03T21:37:44.257+0000 7fa4b1718700 -1 osd.3 87 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-03T21:37:21.250221+0000 front 2024-05-03T21:37:21.250175+0000 (oldest deadline 2024-05-03T21:37:43.550289+0000) 2024-05-03T21:37:45.495 INFO:journalctl@ceph.osd.10.smithi082.stdout:May 03 21:37:44 smithi082 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-10[57676]: 2024-05-03T21:37:44.995+0000 7f790b86c700 -1 osd.10 87 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-03T21:37:18.101114+0000 front 2024-05-03T21:37:18.101101+0000 (oldest deadline 2024-05-03T21:37:42.800953+0000) 2024-05-03T21:37:45.721 INFO:journalctl@ceph.osd.3.smithi067.stdout:May 03 21:37:45 smithi067 podman[87792]: 2024-05-03 21:37:45.397925074 +0000 UTC m=+5.483216182 container died 290dc8a7488b3cd08335769d301c7bcc88a1dd9c9989dced522c552598f65da6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:68e15ae152ef5c805c733e76da8fc1d3dd4ee669d5ba025f457e05725baa24dd, name=ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-3, RELEASE=wip-rishabh-testing-20240501.193033-9a047fe, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240430, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , ceph=True) 2024-05-03T21:37:45.721 INFO:journalctl@ceph.osd.9.smithi067.stdout:May 03 21:37:45 smithi067 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-9[62235]: 2024-05-03T21:37:45.560+0000 7fcff92a2700 -1 osd.9 87 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-03T21:37:20.101372+0000 front 2024-05-03T21:37:20.101400+0000 (oldest deadline 2024-05-03T21:37:44.801519+0000) 2024-05-03T21:37:46.220 INFO:journalctl@ceph.osd.3.smithi067.stdout:May 03 21:37:45 smithi067 podman[87792]: 2024-05-03 21:37:45.72312158 +0000 UTC m=+5.808412694 container cleanup 290dc8a7488b3cd08335769d301c7bcc88a1dd9c9989dced522c552598f65da6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:68e15ae152ef5c805c733e76da8fc1d3dd4ee669d5ba025f457e05725baa24dd, name=ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-3, ceph=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, RELEASE=wip-rishabh-testing-20240501.193033-9a047fe, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.build-date=20240430, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_CLEAN=True, io.buildah.version=1.33.5) 2024-05-03T21:37:46.221 INFO:journalctl@ceph.osd.3.smithi067.stdout:May 03 21:37:45 smithi067 bash[87792]: ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-3 2024-05-03T21:37:46.221 INFO:journalctl@ceph.osd.3.smithi067.stdout:May 03 21:37:45 smithi067 podman[87806]: 2024-05-03 21:37:45.882407779 +0000 UTC m=+0.479817483 container remove 290dc8a7488b3cd08335769d301c7bcc88a1dd9c9989dced522c552598f65da6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:68e15ae152ef5c805c733e76da8fc1d3dd4ee669d5ba025f457e05725baa24dd, name=ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-3, GIT_BRANCH=HEAD, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.build-date=20240430, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, RELEASE=wip-rishabh-testing-20240501.193033-9a047fe, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.license=GPLv2) 2024-05-03T21:37:46.380 INFO:journalctl@ceph.osd.11.smithi138.stdout:May 03 21:37:46 smithi138 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-11[58547]: 2024-05-03T21:37:46.107+0000 7fdfdcb59700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-03T21:37:21.054826+0000 front 2024-05-03T21:37:21.054670+0000 (oldest deadline 2024-05-03T21:37:45.154864+0000) 2024-05-03T21:37:46.495 INFO:journalctl@ceph.osd.10.smithi082.stdout:May 03 21:37:46 smithi082 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-10[57676]: 2024-05-03T21:37:46.013+0000 7f790b86c700 -1 osd.10 87 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-03T21:37:18.101114+0000 front 2024-05-03T21:37:18.101101+0000 (oldest deadline 2024-05-03T21:37:42.800953+0000) 2024-05-03T21:37:46.503 INFO:journalctl@ceph.osd.3.smithi067.stdout:May 03 21:37:46 smithi067 podman[87899]: 2024-05-03 21:37:46.203110504 +0000 UTC m=+0.018627646 image pull ee4d846202496ba6096593e8e8becc0700a5eb7ebd95536031187f5f337d4012 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:68e15ae152ef5c805c733e76da8fc1d3dd4ee669d5ba025f457e05725baa24dd 2024-05-03T21:37:46.503 INFO:journalctl@ceph.osd.3.smithi067.stdout:May 03 21:37:46 smithi067 podman[87899]: 2024-05-03 21:37:46.316701449 +0000 UTC m=+0.132218571 container create e6f0f0dd8387a5b16dfe80d22d9aa8a8d9316f6dc043124383c728a63eb3cabe (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:68e15ae152ef5c805c733e76da8fc1d3dd4ee669d5ba025f457e05725baa24dd, name=ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-3-deactivate, RELEASE=wip-rishabh-testing-20240501.193033-9a047fe, org.label-schema.build-date=20240430, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, ceph=True, org.label-schema.license=GPLv2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-05-03T21:37:46.816 INFO:journalctl@ceph.osd.9.smithi067.stdout:May 03 21:37:46 smithi067 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-9[62235]: 2024-05-03T21:37:46.545+0000 7fcff92a2700 -1 osd.9 87 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-03T21:37:20.101372+0000 front 2024-05-03T21:37:20.101400+0000 (oldest deadline 2024-05-03T21:37:44.801519+0000) 2024-05-03T21:37:46.817 INFO:journalctl@ceph.osd.3.smithi067.stdout:May 03 21:37:46 smithi067 podman[87899]: 2024-05-03 21:37:46.502371621 +0000 UTC m=+0.317888744 container init e6f0f0dd8387a5b16dfe80d22d9aa8a8d9316f6dc043124383c728a63eb3cabe (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:68e15ae152ef5c805c733e76da8fc1d3dd4ee669d5ba025f457e05725baa24dd, name=ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-3-deactivate, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, io.buildah.version=1.33.5, org.label-schema.build-date=20240430, RELEASE=wip-rishabh-testing-20240501.193033-9a047fe, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, ceph=True, GIT_BRANCH=HEAD) 2024-05-03T21:37:46.817 INFO:journalctl@ceph.osd.3.smithi067.stdout:May 03 21:37:46 smithi067 podman[87899]: 2024-05-03 21:37:46.505837291 +0000 UTC m=+0.321354416 container start e6f0f0dd8387a5b16dfe80d22d9aa8a8d9316f6dc043124383c728a63eb3cabe (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:68e15ae152ef5c805c733e76da8fc1d3dd4ee669d5ba025f457e05725baa24dd, name=ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-3-deactivate, io.buildah.version=1.33.5, org.label-schema.build-date=20240430, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, ceph=True, GIT_CLEAN=True, RELEASE=wip-rishabh-testing-20240501.193033-9a047fe, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b) 2024-05-03T21:37:46.817 INFO:journalctl@ceph.osd.3.smithi067.stdout:May 03 21:37:46 smithi067 podman[87899]: 2024-05-03 21:37:46.627445726 +0000 UTC m=+0.442962865 container attach e6f0f0dd8387a5b16dfe80d22d9aa8a8d9316f6dc043124383c728a63eb3cabe (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:68e15ae152ef5c805c733e76da8fc1d3dd4ee669d5ba025f457e05725baa24dd, name=ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-3-deactivate, GIT_BRANCH=HEAD, RELEASE=wip-rishabh-testing-20240501.193033-9a047fe, org.label-schema.vendor=CentOS, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, ceph=True, GIT_CLEAN=True, org.label-schema.build-date=20240430, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2) 2024-05-03T21:37:46.824 INFO:journalctl@ceph.osd.5.smithi138.stdout:May 03 21:37:46 smithi138 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-5[45227]: 2024-05-03T21:37:46.379+0000 7fe1cfb9f700 -1 osd.5 87 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-03T21:37:20.947718+0000 front 2024-05-03T21:37:20.947686+0000 (oldest deadline 2024-05-03T21:37:45.548290+0000) 2024-05-03T21:37:47.181 INFO:journalctl@ceph.osd.3.smithi067.stdout:May 03 21:37:46 smithi067 podman[87899]: 2024-05-03 21:37:46.815988461 +0000 UTC m=+0.631505782 container died e6f0f0dd8387a5b16dfe80d22d9aa8a8d9316f6dc043124383c728a63eb3cabe (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:68e15ae152ef5c805c733e76da8fc1d3dd4ee669d5ba025f457e05725baa24dd, name=ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-3-deactivate, GIT_CLEAN=True, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, RELEASE=wip-rishabh-testing-20240501.193033-9a047fe, org.label-schema.build-date=20240430) 2024-05-03T21:37:47.432 INFO:journalctl@ceph.osd.3.smithi067.stdout:May 03 21:37:47 smithi067 podman[87925]: 2024-05-03 21:37:47.41425258 +0000 UTC m=+0.590503994 container remove e6f0f0dd8387a5b16dfe80d22d9aa8a8d9316f6dc043124383c728a63eb3cabe (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:68e15ae152ef5c805c733e76da8fc1d3dd4ee669d5ba025f457e05725baa24dd, name=ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-3-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, CEPH_POINT_RELEASE=, RELEASE=wip-rishabh-testing-20240501.193033-9a047fe, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240430, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-05-03T21:37:47.436 DEBUG:teuthology.orchestra.run.smithi067:> sudo pkill -f 'journalctl -f -n 0 -u ceph-713bc57a-0991-11ef-bc97-c7b262605968@osd.3.service' 2024-05-03T21:37:47.477 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-03T21:37:47.478 INFO:tasks.cephadm.osd.3:Stopped osd.3 2024-05-03T21:37:47.478 INFO:tasks.cephadm.osd.4:Stopping osd.4... 2024-05-03T21:37:47.478 DEBUG:teuthology.orchestra.run.smithi082:> sudo systemctl stop ceph-713bc57a-0991-11ef-bc97-c7b262605968@osd.4 2024-05-03T21:37:47.494 INFO:journalctl@ceph.osd.10.smithi082.stdout:May 03 21:37:46 smithi082 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-10[57676]: 2024-05-03T21:37:46.994+0000 7f790b86c700 -1 osd.10 87 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-03T21:37:18.101114+0000 front 2024-05-03T21:37:18.101101+0000 (oldest deadline 2024-05-03T21:37:42.800953+0000) 2024-05-03T21:37:47.574 INFO:journalctl@ceph.osd.11.smithi138.stdout:May 03 21:37:47 smithi138 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-11[58547]: 2024-05-03T21:37:47.136+0000 7fdfdcb59700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-03T21:37:21.054826+0000 front 2024-05-03T21:37:21.054670+0000 (oldest deadline 2024-05-03T21:37:45.154864+0000) 2024-05-03T21:37:47.574 INFO:journalctl@ceph.osd.5.smithi138.stdout:May 03 21:37:47 smithi138 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-5[45227]: 2024-05-03T21:37:47.382+0000 7fe1cfb9f700 -1 osd.5 87 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-03T21:37:20.947718+0000 front 2024-05-03T21:37:20.947686+0000 (oldest deadline 2024-05-03T21:37:45.548290+0000) 2024-05-03T21:37:47.720 INFO:journalctl@ceph.osd.9.smithi067.stdout:May 03 21:37:47 smithi067 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-9[62235]: 2024-05-03T21:37:47.554+0000 7fcff92a2700 -1 osd.9 87 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-03T21:37:20.101372+0000 front 2024-05-03T21:37:20.101400+0000 (oldest deadline 2024-05-03T21:37:44.801519+0000) 2024-05-03T21:37:47.721 INFO:journalctl@ceph.osd.6.smithi067.stdout:May 03 21:37:47 smithi067 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-6[55180]: 2024-05-03T21:37:47.508+0000 7f389239a700 -1 osd.6 87 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-03T21:37:19.073534+0000 front 2024-05-03T21:37:19.073793+0000 (oldest deadline 2024-05-03T21:37:46.673918+0000) 2024-05-03T21:37:47.822 INFO:journalctl@ceph.osd.4.smithi082.stdout:May 03 21:37:47 smithi082 systemd[1]: Stopping Ceph osd.4 for 713bc57a-0991-11ef-bc97-c7b262605968... 2024-05-03T21:37:48.245 INFO:journalctl@ceph.osd.10.smithi082.stdout:May 03 21:37:47 smithi082 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-10[57676]: 2024-05-03T21:37:47.972+0000 7f790b86c700 -1 osd.10 87 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-03T21:37:18.101114+0000 front 2024-05-03T21:37:18.101101+0000 (oldest deadline 2024-05-03T21:37:42.800953+0000) 2024-05-03T21:37:48.245 INFO:journalctl@ceph.osd.4.smithi082.stdout:May 03 21:37:47 smithi082 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-4[44044]: 2024-05-03T21:37:47.821+0000 7f16082f8700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-03T21:37:48.245 INFO:journalctl@ceph.osd.4.smithi082.stdout:May 03 21:37:47 smithi082 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-4[44044]: 2024-05-03T21:37:47.821+0000 7f16082f8700 -1 osd.4 87 *** Got signal Terminated *** 2024-05-03T21:37:48.246 INFO:journalctl@ceph.osd.4.smithi082.stdout:May 03 21:37:47 smithi082 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-4[44044]: 2024-05-03T21:37:47.821+0000 7f16082f8700 -1 osd.4 87 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-05-03T21:37:48.383 INFO:journalctl@ceph.osd.11.smithi138.stdout:May 03 21:37:48 smithi138 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-11[58547]: 2024-05-03T21:37:48.086+0000 7fdfdcb59700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-03T21:37:21.054826+0000 front 2024-05-03T21:37:21.054670+0000 (oldest deadline 2024-05-03T21:37:45.154864+0000) 2024-05-03T21:37:48.824 INFO:journalctl@ceph.osd.5.smithi138.stdout:May 03 21:37:48 smithi138 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-5[45227]: 2024-05-03T21:37:48.383+0000 7fe1cfb9f700 -1 osd.5 87 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-03T21:37:20.947718+0000 front 2024-05-03T21:37:20.947686+0000 (oldest deadline 2024-05-03T21:37:45.548290+0000) 2024-05-03T21:37:48.970 INFO:journalctl@ceph.osd.6.smithi067.stdout:May 03 21:37:48 smithi067 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-6[55180]: 2024-05-03T21:37:48.486+0000 7f389239a700 -1 osd.6 87 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-03T21:37:19.073534+0000 front 2024-05-03T21:37:19.073793+0000 (oldest deadline 2024-05-03T21:37:46.673918+0000) 2024-05-03T21:37:48.971 INFO:journalctl@ceph.osd.9.smithi067.stdout:May 03 21:37:48 smithi067 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-9[62235]: 2024-05-03T21:37:48.570+0000 7fcff92a2700 -1 osd.9 87 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-03T21:37:20.101372+0000 front 2024-05-03T21:37:20.101400+0000 (oldest deadline 2024-05-03T21:37:44.801519+0000) 2024-05-03T21:37:49.409 INFO:journalctl@ceph.osd.11.smithi138.stdout:May 03 21:37:49 smithi138 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-11[58547]: 2024-05-03T21:37:49.128+0000 7fdfdcb59700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-03T21:37:21.054826+0000 front 2024-05-03T21:37:21.054670+0000 (oldest deadline 2024-05-03T21:37:45.154864+0000) 2024-05-03T21:37:49.495 INFO:journalctl@ceph.osd.10.smithi082.stdout:May 03 21:37:49 smithi082 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-10[57676]: 2024-05-03T21:37:49.022+0000 7f790b86c700 -1 osd.10 87 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-03T21:37:18.101114+0000 front 2024-05-03T21:37:18.101101+0000 (oldest deadline 2024-05-03T21:37:42.800953+0000) 2024-05-03T21:37:49.824 INFO:journalctl@ceph.osd.5.smithi138.stdout:May 03 21:37:49 smithi138 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-5[45227]: 2024-05-03T21:37:49.408+0000 7fe1cfb9f700 -1 osd.5 87 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-03T21:37:20.947718+0000 front 2024-05-03T21:37:20.947686+0000 (oldest deadline 2024-05-03T21:37:45.548290+0000) 2024-05-03T21:37:49.970 INFO:journalctl@ceph.osd.6.smithi067.stdout:May 03 21:37:49 smithi067 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-6[55180]: 2024-05-03T21:37:49.518+0000 7f389239a700 -1 osd.6 87 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-03T21:37:19.073534+0000 front 2024-05-03T21:37:19.073793+0000 (oldest deadline 2024-05-03T21:37:46.673918+0000) 2024-05-03T21:37:49.971 INFO:journalctl@ceph.osd.9.smithi067.stdout:May 03 21:37:49 smithi067 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-9[62235]: 2024-05-03T21:37:49.590+0000 7fcff92a2700 -1 osd.9 87 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-03T21:37:20.101372+0000 front 2024-05-03T21:37:20.101400+0000 (oldest deadline 2024-05-03T21:37:44.801519+0000) 2024-05-03T21:37:50.392 INFO:journalctl@ceph.osd.11.smithi138.stdout:May 03 21:37:50 smithi138 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-11[58547]: 2024-05-03T21:37:50.138+0000 7fdfdcb59700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-03T21:37:21.054826+0000 front 2024-05-03T21:37:21.054670+0000 (oldest deadline 2024-05-03T21:37:45.154864+0000) 2024-05-03T21:37:50.494 INFO:journalctl@ceph.osd.10.smithi082.stdout:May 03 21:37:50 smithi082 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-10[57676]: 2024-05-03T21:37:50.072+0000 7f790b86c700 -1 osd.10 87 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-03T21:37:18.101114+0000 front 2024-05-03T21:37:18.101101+0000 (oldest deadline 2024-05-03T21:37:42.800953+0000) 2024-05-03T21:37:50.824 INFO:journalctl@ceph.osd.5.smithi138.stdout:May 03 21:37:50 smithi138 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-5[45227]: 2024-05-03T21:37:50.391+0000 7fe1cfb9f700 -1 osd.5 87 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-03T21:37:20.947718+0000 front 2024-05-03T21:37:20.947686+0000 (oldest deadline 2024-05-03T21:37:45.548290+0000) 2024-05-03T21:37:50.824 INFO:journalctl@ceph.osd.5.smithi138.stdout:May 03 21:37:50 smithi138 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-5[45227]: 2024-05-03T21:37:50.391+0000 7fe1cfb9f700 -1 osd.5 87 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-05-03T21:37:25.549038+0000 front 2024-05-03T21:37:25.549065+0000 (oldest deadline 2024-05-03T21:37:50.249031+0000) 2024-05-03T21:37:50.970 INFO:journalctl@ceph.osd.9.smithi067.stdout:May 03 21:37:50 smithi067 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-9[62235]: 2024-05-03T21:37:50.633+0000 7fcff92a2700 -1 osd.9 87 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-03T21:37:20.101372+0000 front 2024-05-03T21:37:20.101400+0000 (oldest deadline 2024-05-03T21:37:44.801519+0000) 2024-05-03T21:37:50.971 INFO:journalctl@ceph.osd.6.smithi067.stdout:May 03 21:37:50 smithi067 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-6[55180]: 2024-05-03T21:37:50.522+0000 7f389239a700 -1 osd.6 87 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-03T21:37:19.073534+0000 front 2024-05-03T21:37:19.073793+0000 (oldest deadline 2024-05-03T21:37:46.673918+0000) 2024-05-03T21:37:51.366 INFO:journalctl@ceph.osd.11.smithi138.stdout:May 03 21:37:51 smithi138 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-11[58547]: 2024-05-03T21:37:51.117+0000 7fdfdcb59700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-03T21:37:21.054826+0000 front 2024-05-03T21:37:21.054670+0000 (oldest deadline 2024-05-03T21:37:45.154864+0000) 2024-05-03T21:37:51.366 INFO:journalctl@ceph.osd.8.smithi138.stdout:May 03 21:37:51 smithi138 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-8[51920]: 2024-05-03T21:37:51.097+0000 7f4b828df700 -1 osd.8 87 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-05-03T21:37:29.505823+0000 front 2024-05-03T21:37:29.505840+0000 (oldest deadline 2024-05-03T21:37:50.605770+0000) 2024-05-03T21:37:51.495 INFO:journalctl@ceph.osd.10.smithi082.stdout:May 03 21:37:51 smithi082 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-10[57676]: 2024-05-03T21:37:51.060+0000 7f790b86c700 -1 osd.10 87 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-03T21:37:18.101114+0000 front 2024-05-03T21:37:18.101101+0000 (oldest deadline 2024-05-03T21:37:42.800953+0000) 2024-05-03T21:37:51.824 INFO:journalctl@ceph.osd.5.smithi138.stdout:May 03 21:37:51 smithi138 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-5[45227]: 2024-05-03T21:37:51.365+0000 7fe1cfb9f700 -1 osd.5 87 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-03T21:37:20.947718+0000 front 2024-05-03T21:37:20.947686+0000 (oldest deadline 2024-05-03T21:37:45.548290+0000) 2024-05-03T21:37:51.824 INFO:journalctl@ceph.osd.5.smithi138.stdout:May 03 21:37:51 smithi138 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-5[45227]: 2024-05-03T21:37:51.365+0000 7fe1cfb9f700 -1 osd.5 87 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-05-03T21:37:25.549038+0000 front 2024-05-03T21:37:25.549065+0000 (oldest deadline 2024-05-03T21:37:50.249031+0000) 2024-05-03T21:37:51.970 INFO:journalctl@ceph.osd.6.smithi067.stdout:May 03 21:37:51 smithi067 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-6[55180]: 2024-05-03T21:37:51.539+0000 7f389239a700 -1 osd.6 87 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-03T21:37:19.073534+0000 front 2024-05-03T21:37:19.073793+0000 (oldest deadline 2024-05-03T21:37:46.673918+0000) 2024-05-03T21:37:51.971 INFO:journalctl@ceph.osd.9.smithi067.stdout:May 03 21:37:51 smithi067 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-9[62235]: 2024-05-03T21:37:51.640+0000 7fcff92a2700 -1 osd.9 87 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-03T21:37:20.101372+0000 front 2024-05-03T21:37:20.101400+0000 (oldest deadline 2024-05-03T21:37:44.801519+0000) 2024-05-03T21:37:52.317 INFO:journalctl@ceph.osd.11.smithi138.stdout:May 03 21:37:52 smithi138 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-11[58547]: 2024-05-03T21:37:52.155+0000 7fdfdcb59700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-03T21:37:21.054826+0000 front 2024-05-03T21:37:21.054670+0000 (oldest deadline 2024-05-03T21:37:45.154864+0000) 2024-05-03T21:37:52.317 INFO:journalctl@ceph.osd.8.smithi138.stdout:May 03 21:37:52 smithi138 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-8[51920]: 2024-05-03T21:37:52.053+0000 7f4b828df700 -1 osd.8 87 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-05-03T21:37:29.505823+0000 front 2024-05-03T21:37:29.505840+0000 (oldest deadline 2024-05-03T21:37:50.605770+0000) 2024-05-03T21:37:52.495 INFO:journalctl@ceph.osd.10.smithi082.stdout:May 03 21:37:52 smithi082 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-10[57676]: 2024-05-03T21:37:52.069+0000 7f790b86c700 -1 osd.10 87 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-03T21:37:18.101114+0000 front 2024-05-03T21:37:18.101101+0000 (oldest deadline 2024-05-03T21:37:42.800953+0000) 2024-05-03T21:37:52.495 INFO:journalctl@ceph.osd.10.smithi082.stdout:May 03 21:37:52 smithi082 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-10[57676]: 2024-05-03T21:37:52.069+0000 7f790b86c700 -1 osd.10 87 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-05-03T21:37:28.303836+0000 front 2024-05-03T21:37:28.303941+0000 (oldest deadline 2024-05-03T21:37:51.804075+0000) 2024-05-03T21:37:52.574 INFO:journalctl@ceph.osd.5.smithi138.stdout:May 03 21:37:52 smithi138 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-5[45227]: 2024-05-03T21:37:52.316+0000 7fe1cfb9f700 -1 osd.5 87 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-03T21:37:20.947718+0000 front 2024-05-03T21:37:20.947686+0000 (oldest deadline 2024-05-03T21:37:45.548290+0000) 2024-05-03T21:37:52.574 INFO:journalctl@ceph.osd.5.smithi138.stdout:May 03 21:37:52 smithi138 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-5[45227]: 2024-05-03T21:37:52.316+0000 7fe1cfb9f700 -1 osd.5 87 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-05-03T21:37:25.549038+0000 front 2024-05-03T21:37:25.549065+0000 (oldest deadline 2024-05-03T21:37:50.249031+0000) 2024-05-03T21:37:52.970 INFO:journalctl@ceph.osd.6.smithi067.stdout:May 03 21:37:52 smithi067 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-6[55180]: 2024-05-03T21:37:52.534+0000 7f389239a700 -1 osd.6 87 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-03T21:37:19.073534+0000 front 2024-05-03T21:37:19.073793+0000 (oldest deadline 2024-05-03T21:37:46.673918+0000) 2024-05-03T21:37:52.971 INFO:journalctl@ceph.osd.9.smithi067.stdout:May 03 21:37:52 smithi067 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-9[62235]: 2024-05-03T21:37:52.654+0000 7fcff92a2700 -1 osd.9 87 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-03T21:37:20.101372+0000 front 2024-05-03T21:37:20.101400+0000 (oldest deadline 2024-05-03T21:37:44.801519+0000) 2024-05-03T21:37:52.971 INFO:journalctl@ceph.osd.9.smithi067.stdout:May 03 21:37:52 smithi067 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-9[62235]: 2024-05-03T21:37:52.654+0000 7fcff92a2700 -1 osd.9 87 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-05-03T21:37:27.702901+0000 front 2024-05-03T21:37:27.702918+0000 (oldest deadline 2024-05-03T21:37:52.402882+0000) 2024-05-03T21:37:53.324 INFO:journalctl@ceph.osd.11.smithi138.stdout:May 03 21:37:53 smithi138 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-11[58547]: 2024-05-03T21:37:53.183+0000 7fdfdcb59700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-03T21:37:21.054826+0000 front 2024-05-03T21:37:21.054670+0000 (oldest deadline 2024-05-03T21:37:45.154864+0000) 2024-05-03T21:37:53.324 INFO:journalctl@ceph.osd.8.smithi138.stdout:May 03 21:37:53 smithi138 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-8[51920]: 2024-05-03T21:37:53.053+0000 7f4b828df700 -1 osd.8 87 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-05-03T21:37:29.505823+0000 front 2024-05-03T21:37:29.505840+0000 (oldest deadline 2024-05-03T21:37:50.605770+0000) 2024-05-03T21:37:53.495 INFO:journalctl@ceph.osd.4.smithi082.stdout:May 03 21:37:53 smithi082 podman[68485]: 2024-05-03 21:37:53.228229435 +0000 UTC m=+5.551973277 container died 28ed3b7118a04d14f85b51d4ea4e2801e8e5943ed08b52a1cc1e1f7c78608468 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:68e15ae152ef5c805c733e76da8fc1d3dd4ee669d5ba025f457e05725baa24dd, name=ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-4, ceph=True, GIT_BRANCH=HEAD, RELEASE=wip-rishabh-testing-20240501.193033-9a047fe, org.label-schema.schema-version=1.0, GIT_CLEAN=True, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240430, org.label-schema.name=CentOS Stream 8 Base Image) 2024-05-03T21:37:53.495 INFO:journalctl@ceph.osd.10.smithi082.stdout:May 03 21:37:53 smithi082 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-10[57676]: 2024-05-03T21:37:53.081+0000 7f790b86c700 -1 osd.10 87 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-03T21:37:18.101114+0000 front 2024-05-03T21:37:18.101101+0000 (oldest deadline 2024-05-03T21:37:42.800953+0000) 2024-05-03T21:37:53.495 INFO:journalctl@ceph.osd.10.smithi082.stdout:May 03 21:37:53 smithi082 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-10[57676]: 2024-05-03T21:37:53.081+0000 7f790b86c700 -1 osd.10 87 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-05-03T21:37:28.303836+0000 front 2024-05-03T21:37:28.303941+0000 (oldest deadline 2024-05-03T21:37:51.804075+0000) 2024-05-03T21:37:53.824 INFO:journalctl@ceph.osd.5.smithi138.stdout:May 03 21:37:53 smithi138 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-5[45227]: 2024-05-03T21:37:53.366+0000 7fe1cfb9f700 -1 osd.5 87 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-03T21:37:20.947718+0000 front 2024-05-03T21:37:20.947686+0000 (oldest deadline 2024-05-03T21:37:45.548290+0000) 2024-05-03T21:37:53.824 INFO:journalctl@ceph.osd.5.smithi138.stdout:May 03 21:37:53 smithi138 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-5[45227]: 2024-05-03T21:37:53.366+0000 7fe1cfb9f700 -1 osd.5 87 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-05-03T21:37:25.549038+0000 front 2024-05-03T21:37:25.549065+0000 (oldest deadline 2024-05-03T21:37:50.249031+0000) 2024-05-03T21:37:53.839 INFO:journalctl@ceph.osd.4.smithi082.stdout:May 03 21:37:53 smithi082 podman[68485]: 2024-05-03 21:37:53.579674307 +0000 UTC m=+5.903418156 container cleanup 28ed3b7118a04d14f85b51d4ea4e2801e8e5943ed08b52a1cc1e1f7c78608468 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:68e15ae152ef5c805c733e76da8fc1d3dd4ee669d5ba025f457e05725baa24dd, name=ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-4, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, RELEASE=wip-rishabh-testing-20240501.193033-9a047fe, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240430, GIT_BRANCH=HEAD) 2024-05-03T21:37:53.840 INFO:journalctl@ceph.osd.4.smithi082.stdout:May 03 21:37:53 smithi082 bash[68485]: ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-4 2024-05-03T21:37:53.970 INFO:journalctl@ceph.osd.6.smithi067.stdout:May 03 21:37:53 smithi067 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-6[55180]: 2024-05-03T21:37:53.571+0000 7f389239a700 -1 osd.6 87 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-03T21:37:19.073534+0000 front 2024-05-03T21:37:19.073793+0000 (oldest deadline 2024-05-03T21:37:46.673918+0000) 2024-05-03T21:37:53.971 INFO:journalctl@ceph.osd.6.smithi067.stdout:May 03 21:37:53 smithi067 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-6[55180]: 2024-05-03T21:37:53.571+0000 7f389239a700 -1 osd.6 87 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-05-03T21:37:26.674547+0000 front 2024-05-03T21:37:26.674628+0000 (oldest deadline 2024-05-03T21:37:52.574465+0000) 2024-05-03T21:37:53.971 INFO:journalctl@ceph.osd.9.smithi067.stdout:May 03 21:37:53 smithi067 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-9[62235]: 2024-05-03T21:37:53.697+0000 7fcff92a2700 -1 osd.9 87 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-03T21:37:20.101372+0000 front 2024-05-03T21:37:20.101400+0000 (oldest deadline 2024-05-03T21:37:44.801519+0000) 2024-05-03T21:37:53.971 INFO:journalctl@ceph.osd.9.smithi067.stdout:May 03 21:37:53 smithi067 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-9[62235]: 2024-05-03T21:37:53.697+0000 7fcff92a2700 -1 osd.9 87 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-05-03T21:37:27.702901+0000 front 2024-05-03T21:37:27.702918+0000 (oldest deadline 2024-05-03T21:37:52.402882+0000) 2024-05-03T21:37:54.122 INFO:journalctl@ceph.osd.4.smithi082.stdout:May 03 21:37:53 smithi082 podman[68502]: 2024-05-03 21:37:53.839192925 +0000 UTC m=+0.611949690 container remove 28ed3b7118a04d14f85b51d4ea4e2801e8e5943ed08b52a1cc1e1f7c78608468 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:68e15ae152ef5c805c733e76da8fc1d3dd4ee669d5ba025f457e05725baa24dd, name=ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-4, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240430, org.label-schema.license=GPLv2, GIT_CLEAN=True, RELEASE=wip-rishabh-testing-20240501.193033-9a047fe, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, CEPH_POINT_RELEASE=, ceph=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS) 2024-05-03T21:37:54.324 INFO:journalctl@ceph.osd.8.smithi138.stdout:May 03 21:37:54 smithi138 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-8[51920]: 2024-05-03T21:37:54.046+0000 7f4b828df700 -1 osd.8 87 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-05-03T21:37:29.505823+0000 front 2024-05-03T21:37:29.505840+0000 (oldest deadline 2024-05-03T21:37:50.605770+0000) 2024-05-03T21:37:54.324 INFO:journalctl@ceph.osd.11.smithi138.stdout:May 03 21:37:54 smithi138 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-11[58547]: 2024-05-03T21:37:54.198+0000 7fdfdcb59700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-03T21:37:21.054826+0000 front 2024-05-03T21:37:21.054670+0000 (oldest deadline 2024-05-03T21:37:45.154864+0000) 2024-05-03T21:37:54.324 INFO:journalctl@ceph.osd.11.smithi138.stdout:May 03 21:37:54 smithi138 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-11[58547]: 2024-05-03T21:37:54.198+0000 7fdfdcb59700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-05-03T21:37:29.256054+0000 front 2024-05-03T21:37:29.256037+0000 (oldest deadline 2024-05-03T21:37:53.956115+0000) 2024-05-03T21:37:54.426 INFO:journalctl@ceph.osd.10.smithi082.stdout:May 03 21:37:54 smithi082 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-10[57676]: 2024-05-03T21:37:54.121+0000 7f790b86c700 -1 osd.10 87 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-03T21:37:18.101114+0000 front 2024-05-03T21:37:18.101101+0000 (oldest deadline 2024-05-03T21:37:42.800953+0000) 2024-05-03T21:37:54.426 INFO:journalctl@ceph.osd.10.smithi082.stdout:May 03 21:37:54 smithi082 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-10[57676]: 2024-05-03T21:37:54.121+0000 7f790b86c700 -1 osd.10 87 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-05-03T21:37:28.303836+0000 front 2024-05-03T21:37:28.303941+0000 (oldest deadline 2024-05-03T21:37:51.804075+0000) 2024-05-03T21:37:54.426 INFO:journalctl@ceph.osd.4.smithi082.stdout:May 03 21:37:54 smithi082 podman[68594]: 2024-05-03 21:37:54.148206929 +0000 UTC m=+0.106635677 container create d7312277f7d3d2089adc7f2a43f72973388a542b701cb00a982a82e4fab044e9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:68e15ae152ef5c805c733e76da8fc1d3dd4ee669d5ba025f457e05725baa24dd, name=ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-4-deactivate, ceph=True, io.buildah.version=1.33.5, RELEASE=wip-rishabh-testing-20240501.193033-9a047fe, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.build-date=20240430, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.schema-version=1.0) 2024-05-03T21:37:54.426 INFO:journalctl@ceph.osd.4.smithi082.stdout:May 03 21:37:54 smithi082 podman[68594]: 2024-05-03 21:37:54.059552992 +0000 UTC m=+0.017981769 image pull ee4d846202496ba6096593e8e8becc0700a5eb7ebd95536031187f5f337d4012 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:68e15ae152ef5c805c733e76da8fc1d3dd4ee669d5ba025f457e05725baa24dd 2024-05-03T21:37:54.427 INFO:journalctl@ceph.osd.4.smithi082.stdout:May 03 21:37:54 smithi082 podman[68594]: 2024-05-03 21:37:54.35954384 +0000 UTC m=+0.317972596 container init d7312277f7d3d2089adc7f2a43f72973388a542b701cb00a982a82e4fab044e9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:68e15ae152ef5c805c733e76da8fc1d3dd4ee669d5ba025f457e05725baa24dd, name=ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-4-deactivate, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, RELEASE=wip-rishabh-testing-20240501.193033-9a047fe, CEPH_POINT_RELEASE=, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240430) 2024-05-03T21:37:54.427 INFO:journalctl@ceph.osd.4.smithi082.stdout:May 03 21:37:54 smithi082 podman[68594]: 2024-05-03 21:37:54.362717574 +0000 UTC m=+0.321146326 container start d7312277f7d3d2089adc7f2a43f72973388a542b701cb00a982a82e4fab044e9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:68e15ae152ef5c805c733e76da8fc1d3dd4ee669d5ba025f457e05725baa24dd, name=ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-4-deactivate, org.label-schema.license=GPLv2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240430, RELEASE=wip-rishabh-testing-20240501.193033-9a047fe, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_CLEAN=True, ceph=True) 2024-05-03T21:37:54.745 INFO:journalctl@ceph.osd.4.smithi082.stdout:May 03 21:37:54 smithi082 podman[68594]: 2024-05-03 21:37:54.500563425 +0000 UTC m=+0.458992178 container attach d7312277f7d3d2089adc7f2a43f72973388a542b701cb00a982a82e4fab044e9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:68e15ae152ef5c805c733e76da8fc1d3dd4ee669d5ba025f457e05725baa24dd, name=ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-4-deactivate, org.label-schema.vendor=CentOS, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240430, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.schema-version=1.0, RELEASE=wip-rishabh-testing-20240501.193033-9a047fe, org.label-schema.license=GPLv2) 2024-05-03T21:37:54.745 INFO:journalctl@ceph.osd.4.smithi082.stdout:May 03 21:37:54 smithi082 podman[68594]: 2024-05-03 21:37:54.659048546 +0000 UTC m=+0.617477306 container died d7312277f7d3d2089adc7f2a43f72973388a542b701cb00a982a82e4fab044e9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:68e15ae152ef5c805c733e76da8fc1d3dd4ee669d5ba025f457e05725baa24dd, name=ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-4-deactivate, RELEASE=wip-rishabh-testing-20240501.193033-9a047fe, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240430, maintainer=Guillaume Abrioux , GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_CLEAN=True) 2024-05-03T21:37:54.746 INFO:journalctl@ceph.osd.7.smithi082.stdout:May 03 21:37:54 smithi082 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-7[50761]: 2024-05-03T21:37:54.424+0000 7f44b54b5700 -1 osd.7 87 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-05-03T21:37:26.771556+0000 front 2024-05-03T21:37:26.771586+0000 (oldest deadline 2024-05-03T21:37:54.372236+0000) 2024-05-03T21:37:54.824 INFO:journalctl@ceph.osd.5.smithi138.stdout:May 03 21:37:54 smithi138 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-5[45227]: 2024-05-03T21:37:54.325+0000 7fe1cfb9f700 -1 osd.5 87 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-03T21:37:20.947718+0000 front 2024-05-03T21:37:20.947686+0000 (oldest deadline 2024-05-03T21:37:45.548290+0000) 2024-05-03T21:37:54.824 INFO:journalctl@ceph.osd.5.smithi138.stdout:May 03 21:37:54 smithi138 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-5[45227]: 2024-05-03T21:37:54.325+0000 7fe1cfb9f700 -1 osd.5 87 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-05-03T21:37:25.549038+0000 front 2024-05-03T21:37:25.549065+0000 (oldest deadline 2024-05-03T21:37:50.249031+0000) 2024-05-03T21:37:54.970 INFO:journalctl@ceph.osd.6.smithi067.stdout:May 03 21:37:54 smithi067 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-6[55180]: 2024-05-03T21:37:54.609+0000 7f389239a700 -1 osd.6 87 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-03T21:37:19.073534+0000 front 2024-05-03T21:37:19.073793+0000 (oldest deadline 2024-05-03T21:37:46.673918+0000) 2024-05-03T21:37:54.971 INFO:journalctl@ceph.osd.6.smithi067.stdout:May 03 21:37:54 smithi067 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-6[55180]: 2024-05-03T21:37:54.609+0000 7f389239a700 -1 osd.6 87 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-05-03T21:37:26.674547+0000 front 2024-05-03T21:37:26.674628+0000 (oldest deadline 2024-05-03T21:37:52.574465+0000) 2024-05-03T21:37:54.971 INFO:journalctl@ceph.osd.9.smithi067.stdout:May 03 21:37:54 smithi067 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-9[62235]: 2024-05-03T21:37:54.686+0000 7fcff92a2700 -1 osd.9 87 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-03T21:37:20.101372+0000 front 2024-05-03T21:37:20.101400+0000 (oldest deadline 2024-05-03T21:37:44.801519+0000) 2024-05-03T21:37:54.971 INFO:journalctl@ceph.osd.9.smithi067.stdout:May 03 21:37:54 smithi067 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-9[62235]: 2024-05-03T21:37:54.686+0000 7fcff92a2700 -1 osd.9 87 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-05-03T21:37:27.702901+0000 front 2024-05-03T21:37:27.702918+0000 (oldest deadline 2024-05-03T21:37:52.402882+0000) 2024-05-03T21:37:55.204 INFO:journalctl@ceph.osd.10.smithi082.stdout:May 03 21:37:55 smithi082 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-10[57676]: 2024-05-03T21:37:55.099+0000 7f790b86c700 -1 osd.10 87 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-03T21:37:18.101114+0000 front 2024-05-03T21:37:18.101101+0000 (oldest deadline 2024-05-03T21:37:42.800953+0000) 2024-05-03T21:37:55.205 INFO:journalctl@ceph.osd.10.smithi082.stdout:May 03 21:37:55 smithi082 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-10[57676]: 2024-05-03T21:37:55.099+0000 7f790b86c700 -1 osd.10 87 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-05-03T21:37:28.303836+0000 front 2024-05-03T21:37:28.303941+0000 (oldest deadline 2024-05-03T21:37:51.804075+0000) 2024-05-03T21:37:55.225 DEBUG:teuthology.orchestra.run.smithi082:> sudo pkill -f 'journalctl -f -n 0 -u ceph-713bc57a-0991-11ef-bc97-c7b262605968@osd.4.service' 2024-05-03T21:37:55.302 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-03T21:37:55.305 INFO:tasks.cephadm.osd.4:Stopped osd.4 2024-05-03T21:37:55.305 INFO:tasks.cephadm.osd.5:Stopping osd.5... 2024-05-03T21:37:55.306 DEBUG:teuthology.orchestra.run.smithi138:> sudo systemctl stop ceph-713bc57a-0991-11ef-bc97-c7b262605968@osd.5 2024-05-03T21:37:55.326 INFO:journalctl@ceph.osd.8.smithi138.stdout:May 03 21:37:55 smithi138 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-8[51920]: 2024-05-03T21:37:55.073+0000 7f4b828df700 -1 osd.8 87 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-05-03T21:37:29.505823+0000 front 2024-05-03T21:37:29.505840+0000 (oldest deadline 2024-05-03T21:37:50.605770+0000) 2024-05-03T21:37:55.326 INFO:journalctl@ceph.osd.11.smithi138.stdout:May 03 21:37:55 smithi138 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-11[58547]: 2024-05-03T21:37:55.164+0000 7fdfdcb59700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-03T21:37:21.054826+0000 front 2024-05-03T21:37:21.054670+0000 (oldest deadline 2024-05-03T21:37:45.154864+0000) 2024-05-03T21:37:55.327 INFO:journalctl@ceph.osd.11.smithi138.stdout:May 03 21:37:55 smithi138 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-11[58547]: 2024-05-03T21:37:55.164+0000 7fdfdcb59700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-05-03T21:37:29.256054+0000 front 2024-05-03T21:37:29.256037+0000 (oldest deadline 2024-05-03T21:37:53.956115+0000) 2024-05-03T21:37:55.494 INFO:journalctl@ceph.osd.7.smithi082.stdout:May 03 21:37:55 smithi082 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-7[50761]: 2024-05-03T21:37:55.380+0000 7f44b54b5700 -1 osd.7 87 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-05-03T21:37:26.771556+0000 front 2024-05-03T21:37:26.771586+0000 (oldest deadline 2024-05-03T21:37:54.372236+0000) 2024-05-03T21:37:55.616 INFO:journalctl@ceph.osd.5.smithi138.stdout:May 03 21:37:55 smithi138 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-5[45227]: 2024-05-03T21:37:55.324+0000 7fe1cfb9f700 -1 osd.5 87 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-03T21:37:20.947718+0000 front 2024-05-03T21:37:20.947686+0000 (oldest deadline 2024-05-03T21:37:45.548290+0000) 2024-05-03T21:37:55.616 INFO:journalctl@ceph.osd.5.smithi138.stdout:May 03 21:37:55 smithi138 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-5[45227]: 2024-05-03T21:37:55.324+0000 7fe1cfb9f700 -1 osd.5 87 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-05-03T21:37:25.549038+0000 front 2024-05-03T21:37:25.549065+0000 (oldest deadline 2024-05-03T21:37:50.249031+0000) 2024-05-03T21:37:55.616 INFO:journalctl@ceph.osd.5.smithi138.stdout:May 03 21:37:55 smithi138 systemd[1]: Stopping Ceph osd.5 for 713bc57a-0991-11ef-bc97-c7b262605968... 2024-05-03T21:37:55.616 INFO:journalctl@ceph.osd.5.smithi138.stdout:May 03 21:37:55 smithi138 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-5[45227]: 2024-05-03T21:37:55.614+0000 7fe1d75b3700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-03T21:37:55.616 INFO:journalctl@ceph.osd.5.smithi138.stdout:May 03 21:37:55 smithi138 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-5[45227]: 2024-05-03T21:37:55.614+0000 7fe1d75b3700 -1 osd.5 87 *** Got signal Terminated *** 2024-05-03T21:37:55.616 INFO:journalctl@ceph.osd.5.smithi138.stdout:May 03 21:37:55 smithi138 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-5[45227]: 2024-05-03T21:37:55.614+0000 7fe1d75b3700 -1 osd.5 87 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-05-03T21:37:55.971 INFO:journalctl@ceph.osd.6.smithi067.stdout:May 03 21:37:55 smithi067 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-6[55180]: 2024-05-03T21:37:55.608+0000 7f389239a700 -1 osd.6 87 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-03T21:37:19.073534+0000 front 2024-05-03T21:37:19.073793+0000 (oldest deadline 2024-05-03T21:37:46.673918+0000) 2024-05-03T21:37:55.971 INFO:journalctl@ceph.osd.6.smithi067.stdout:May 03 21:37:55 smithi067 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-6[55180]: 2024-05-03T21:37:55.608+0000 7f389239a700 -1 osd.6 87 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-05-03T21:37:26.674547+0000 front 2024-05-03T21:37:26.674628+0000 (oldest deadline 2024-05-03T21:37:52.574465+0000) 2024-05-03T21:37:55.971 INFO:journalctl@ceph.osd.9.smithi067.stdout:May 03 21:37:55 smithi067 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-9[62235]: 2024-05-03T21:37:55.655+0000 7fcff92a2700 -1 osd.9 87 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-03T21:37:20.101372+0000 front 2024-05-03T21:37:20.101400+0000 (oldest deadline 2024-05-03T21:37:44.801519+0000) 2024-05-03T21:37:55.971 INFO:journalctl@ceph.osd.9.smithi067.stdout:May 03 21:37:55 smithi067 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-9[62235]: 2024-05-03T21:37:55.655+0000 7fcff92a2700 -1 osd.9 87 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-05-03T21:37:27.702901+0000 front 2024-05-03T21:37:27.702918+0000 (oldest deadline 2024-05-03T21:37:52.402882+0000) 2024-05-03T21:37:56.394 INFO:journalctl@ceph.osd.10.smithi082.stdout:May 03 21:37:56 smithi082 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-10[57676]: 2024-05-03T21:37:56.107+0000 7f790b86c700 -1 osd.10 87 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-03T21:37:18.101114+0000 front 2024-05-03T21:37:18.101101+0000 (oldest deadline 2024-05-03T21:37:42.800953+0000) 2024-05-03T21:37:56.394 INFO:journalctl@ceph.osd.10.smithi082.stdout:May 03 21:37:56 smithi082 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-10[57676]: 2024-05-03T21:37:56.107+0000 7f790b86c700 -1 osd.10 87 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-05-03T21:37:28.303836+0000 front 2024-05-03T21:37:28.303941+0000 (oldest deadline 2024-05-03T21:37:51.804075+0000) 2024-05-03T21:37:56.432 INFO:journalctl@ceph.osd.5.smithi138.stdout:May 03 21:37:56 smithi138 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-5[45227]: 2024-05-03T21:37:56.291+0000 7fe1cfb9f700 -1 osd.5 87 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-03T21:37:20.947718+0000 front 2024-05-03T21:37:20.947686+0000 (oldest deadline 2024-05-03T21:37:45.548290+0000) 2024-05-03T21:37:56.432 INFO:journalctl@ceph.osd.5.smithi138.stdout:May 03 21:37:56 smithi138 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-5[45227]: 2024-05-03T21:37:56.291+0000 7fe1cfb9f700 -1 osd.5 87 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-05-03T21:37:25.549038+0000 front 2024-05-03T21:37:25.549065+0000 (oldest deadline 2024-05-03T21:37:50.249031+0000) 2024-05-03T21:37:56.433 INFO:journalctl@ceph.osd.11.smithi138.stdout:May 03 21:37:56 smithi138 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-11[58547]: 2024-05-03T21:37:56.123+0000 7fdfdcb59700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-03T21:37:21.054826+0000 front 2024-05-03T21:37:21.054670+0000 (oldest deadline 2024-05-03T21:37:45.154864+0000) 2024-05-03T21:37:56.433 INFO:journalctl@ceph.osd.11.smithi138.stdout:May 03 21:37:56 smithi138 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-11[58547]: 2024-05-03T21:37:56.123+0000 7fdfdcb59700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-05-03T21:37:29.256054+0000 front 2024-05-03T21:37:29.256037+0000 (oldest deadline 2024-05-03T21:37:53.956115+0000) 2024-05-03T21:37:56.433 INFO:journalctl@ceph.osd.8.smithi138.stdout:May 03 21:37:56 smithi138 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-8[51920]: 2024-05-03T21:37:56.109+0000 7f4b828df700 -1 osd.8 87 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-05-03T21:37:29.505823+0000 front 2024-05-03T21:37:29.505840+0000 (oldest deadline 2024-05-03T21:37:50.605770+0000) 2024-05-03T21:37:56.745 INFO:journalctl@ceph.osd.7.smithi082.stdout:May 03 21:37:56 smithi082 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-7[50761]: 2024-05-03T21:37:56.393+0000 7f44b54b5700 -1 osd.7 87 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-05-03T21:37:26.771556+0000 front 2024-05-03T21:37:26.771586+0000 (oldest deadline 2024-05-03T21:37:54.372236+0000) 2024-05-03T21:37:56.970 INFO:journalctl@ceph.osd.6.smithi067.stdout:May 03 21:37:56 smithi067 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-6[55180]: 2024-05-03T21:37:56.567+0000 7f389239a700 -1 osd.6 87 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-03T21:37:19.073534+0000 front 2024-05-03T21:37:19.073793+0000 (oldest deadline 2024-05-03T21:37:46.673918+0000) 2024-05-03T21:37:56.971 INFO:journalctl@ceph.osd.6.smithi067.stdout:May 03 21:37:56 smithi067 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-6[55180]: 2024-05-03T21:37:56.567+0000 7f389239a700 -1 osd.6 87 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-05-03T21:37:26.674547+0000 front 2024-05-03T21:37:26.674628+0000 (oldest deadline 2024-05-03T21:37:52.574465+0000) 2024-05-03T21:37:56.971 INFO:journalctl@ceph.osd.9.smithi067.stdout:May 03 21:37:56 smithi067 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-9[62235]: 2024-05-03T21:37:56.679+0000 7fcff92a2700 -1 osd.9 87 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-03T21:37:20.101372+0000 front 2024-05-03T21:37:20.101400+0000 (oldest deadline 2024-05-03T21:37:44.801519+0000) 2024-05-03T21:37:56.971 INFO:journalctl@ceph.osd.9.smithi067.stdout:May 03 21:37:56 smithi067 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-9[62235]: 2024-05-03T21:37:56.679+0000 7fcff92a2700 -1 osd.9 87 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-05-03T21:37:27.702901+0000 front 2024-05-03T21:37:27.702918+0000 (oldest deadline 2024-05-03T21:37:52.402882+0000) 2024-05-03T21:37:57.495 INFO:journalctl@ceph.osd.7.smithi082.stdout:May 03 21:37:57 smithi082 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-7[50761]: 2024-05-03T21:37:57.352+0000 7f44b54b5700 -1 osd.7 87 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-05-03T21:37:26.771556+0000 front 2024-05-03T21:37:26.771586+0000 (oldest deadline 2024-05-03T21:37:54.372236+0000) 2024-05-03T21:37:57.495 INFO:journalctl@ceph.osd.10.smithi082.stdout:May 03 21:37:57 smithi082 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-10[57676]: 2024-05-03T21:37:57.140+0000 7f790b86c700 -1 osd.10 87 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-03T21:37:18.101114+0000 front 2024-05-03T21:37:18.101101+0000 (oldest deadline 2024-05-03T21:37:42.800953+0000) 2024-05-03T21:37:57.495 INFO:journalctl@ceph.osd.10.smithi082.stdout:May 03 21:37:57 smithi082 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-10[57676]: 2024-05-03T21:37:57.140+0000 7f790b86c700 -1 osd.10 87 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-05-03T21:37:28.303836+0000 front 2024-05-03T21:37:28.303941+0000 (oldest deadline 2024-05-03T21:37:51.804075+0000) 2024-05-03T21:37:57.574 INFO:journalctl@ceph.osd.8.smithi138.stdout:May 03 21:37:57 smithi138 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-8[51920]: 2024-05-03T21:37:57.106+0000 7f4b828df700 -1 osd.8 87 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-05-03T21:37:29.505823+0000 front 2024-05-03T21:37:29.505840+0000 (oldest deadline 2024-05-03T21:37:50.605770+0000) 2024-05-03T21:37:57.574 INFO:journalctl@ceph.osd.5.smithi138.stdout:May 03 21:37:57 smithi138 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-5[45227]: 2024-05-03T21:37:57.262+0000 7fe1cfb9f700 -1 osd.5 87 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-03T21:37:20.947718+0000 front 2024-05-03T21:37:20.947686+0000 (oldest deadline 2024-05-03T21:37:45.548290+0000) 2024-05-03T21:37:57.575 INFO:journalctl@ceph.osd.5.smithi138.stdout:May 03 21:37:57 smithi138 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-5[45227]: 2024-05-03T21:37:57.262+0000 7fe1cfb9f700 -1 osd.5 87 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-05-03T21:37:25.549038+0000 front 2024-05-03T21:37:25.549065+0000 (oldest deadline 2024-05-03T21:37:50.249031+0000) 2024-05-03T21:37:57.575 INFO:journalctl@ceph.osd.11.smithi138.stdout:May 03 21:37:57 smithi138 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-11[58547]: 2024-05-03T21:37:57.125+0000 7fdfdcb59700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-03T21:37:21.054826+0000 front 2024-05-03T21:37:21.054670+0000 (oldest deadline 2024-05-03T21:37:45.154864+0000) 2024-05-03T21:37:57.575 INFO:journalctl@ceph.osd.11.smithi138.stdout:May 03 21:37:57 smithi138 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-11[58547]: 2024-05-03T21:37:57.125+0000 7fdfdcb59700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-05-03T21:37:29.256054+0000 front 2024-05-03T21:37:29.256037+0000 (oldest deadline 2024-05-03T21:37:53.956115+0000) 2024-05-03T21:37:57.971 INFO:journalctl@ceph.osd.6.smithi067.stdout:May 03 21:37:57 smithi067 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-6[55180]: 2024-05-03T21:37:57.601+0000 7f389239a700 -1 osd.6 87 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-03T21:37:19.073534+0000 front 2024-05-03T21:37:19.073793+0000 (oldest deadline 2024-05-03T21:37:46.673918+0000) 2024-05-03T21:37:57.971 INFO:journalctl@ceph.osd.6.smithi067.stdout:May 03 21:37:57 smithi067 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-6[55180]: 2024-05-03T21:37:57.601+0000 7f389239a700 -1 osd.6 87 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-05-03T21:37:26.674547+0000 front 2024-05-03T21:37:26.674628+0000 (oldest deadline 2024-05-03T21:37:52.574465+0000) 2024-05-03T21:37:57.971 INFO:journalctl@ceph.osd.9.smithi067.stdout:May 03 21:37:57 smithi067 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-9[62235]: 2024-05-03T21:37:57.684+0000 7fcff92a2700 -1 osd.9 87 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-03T21:37:20.101372+0000 front 2024-05-03T21:37:20.101400+0000 (oldest deadline 2024-05-03T21:37:44.801519+0000) 2024-05-03T21:37:57.971 INFO:journalctl@ceph.osd.9.smithi067.stdout:May 03 21:37:57 smithi067 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-9[62235]: 2024-05-03T21:37:57.684+0000 7fcff92a2700 -1 osd.9 87 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-05-03T21:37:27.702901+0000 front 2024-05-03T21:37:27.702918+0000 (oldest deadline 2024-05-03T21:37:52.402882+0000) 2024-05-03T21:37:58.495 INFO:journalctl@ceph.osd.7.smithi082.stdout:May 03 21:37:58 smithi082 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-7[50761]: 2024-05-03T21:37:58.364+0000 7f44b54b5700 -1 osd.7 87 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-05-03T21:37:26.771556+0000 front 2024-05-03T21:37:26.771586+0000 (oldest deadline 2024-05-03T21:37:54.372236+0000) 2024-05-03T21:37:58.495 INFO:journalctl@ceph.osd.10.smithi082.stdout:May 03 21:37:58 smithi082 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-10[57676]: 2024-05-03T21:37:58.132+0000 7f790b86c700 -1 osd.10 87 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-03T21:37:18.101114+0000 front 2024-05-03T21:37:18.101101+0000 (oldest deadline 2024-05-03T21:37:42.800953+0000) 2024-05-03T21:37:58.495 INFO:journalctl@ceph.osd.10.smithi082.stdout:May 03 21:37:58 smithi082 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-10[57676]: 2024-05-03T21:37:58.132+0000 7f790b86c700 -1 osd.10 87 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-05-03T21:37:28.303836+0000 front 2024-05-03T21:37:28.303941+0000 (oldest deadline 2024-05-03T21:37:51.804075+0000) 2024-05-03T21:37:58.574 INFO:journalctl@ceph.osd.8.smithi138.stdout:May 03 21:37:58 smithi138 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-8[51920]: 2024-05-03T21:37:58.075+0000 7f4b828df700 -1 osd.8 87 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-05-03T21:37:29.505823+0000 front 2024-05-03T21:37:29.505840+0000 (oldest deadline 2024-05-03T21:37:50.605770+0000) 2024-05-03T21:37:58.574 INFO:journalctl@ceph.osd.5.smithi138.stdout:May 03 21:37:58 smithi138 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-5[45227]: 2024-05-03T21:37:58.270+0000 7fe1cfb9f700 -1 osd.5 87 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-03T21:37:20.947718+0000 front 2024-05-03T21:37:20.947686+0000 (oldest deadline 2024-05-03T21:37:45.548290+0000) 2024-05-03T21:37:58.574 INFO:journalctl@ceph.osd.5.smithi138.stdout:May 03 21:37:58 smithi138 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-5[45227]: 2024-05-03T21:37:58.270+0000 7fe1cfb9f700 -1 osd.5 87 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-05-03T21:37:25.549038+0000 front 2024-05-03T21:37:25.549065+0000 (oldest deadline 2024-05-03T21:37:50.249031+0000) 2024-05-03T21:37:58.575 INFO:journalctl@ceph.osd.11.smithi138.stdout:May 03 21:37:58 smithi138 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-11[58547]: 2024-05-03T21:37:58.137+0000 7fdfdcb59700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-03T21:37:21.054826+0000 front 2024-05-03T21:37:21.054670+0000 (oldest deadline 2024-05-03T21:37:45.154864+0000) 2024-05-03T21:37:58.575 INFO:journalctl@ceph.osd.11.smithi138.stdout:May 03 21:37:58 smithi138 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-11[58547]: 2024-05-03T21:37:58.137+0000 7fdfdcb59700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-05-03T21:37:29.256054+0000 front 2024-05-03T21:37:29.256037+0000 (oldest deadline 2024-05-03T21:37:53.956115+0000) 2024-05-03T21:37:58.971 INFO:journalctl@ceph.osd.6.smithi067.stdout:May 03 21:37:58 smithi067 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-6[55180]: 2024-05-03T21:37:58.576+0000 7f389239a700 -1 osd.6 87 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-03T21:37:19.073534+0000 front 2024-05-03T21:37:19.073793+0000 (oldest deadline 2024-05-03T21:37:46.673918+0000) 2024-05-03T21:37:58.971 INFO:journalctl@ceph.osd.6.smithi067.stdout:May 03 21:37:58 smithi067 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-6[55180]: 2024-05-03T21:37:58.576+0000 7f389239a700 -1 osd.6 87 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-05-03T21:37:26.674547+0000 front 2024-05-03T21:37:26.674628+0000 (oldest deadline 2024-05-03T21:37:52.574465+0000) 2024-05-03T21:37:58.971 INFO:journalctl@ceph.osd.9.smithi067.stdout:May 03 21:37:58 smithi067 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-9[62235]: 2024-05-03T21:37:58.689+0000 7fcff92a2700 -1 osd.9 87 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-03T21:37:20.101372+0000 front 2024-05-03T21:37:20.101400+0000 (oldest deadline 2024-05-03T21:37:44.801519+0000) 2024-05-03T21:37:58.971 INFO:journalctl@ceph.osd.9.smithi067.stdout:May 03 21:37:58 smithi067 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-9[62235]: 2024-05-03T21:37:58.689+0000 7fcff92a2700 -1 osd.9 87 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-05-03T21:37:27.702901+0000 front 2024-05-03T21:37:27.702918+0000 (oldest deadline 2024-05-03T21:37:52.402882+0000) 2024-05-03T21:37:59.308 INFO:journalctl@ceph.osd.11.smithi138.stdout:May 03 21:37:59 smithi138 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-11[58547]: 2024-05-03T21:37:59.135+0000 7fdfdcb59700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-03T21:37:21.054826+0000 front 2024-05-03T21:37:21.054670+0000 (oldest deadline 2024-05-03T21:37:45.154864+0000) 2024-05-03T21:37:59.308 INFO:journalctl@ceph.osd.11.smithi138.stdout:May 03 21:37:59 smithi138 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-11[58547]: 2024-05-03T21:37:59.135+0000 7fdfdcb59700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-05-03T21:37:29.256054+0000 front 2024-05-03T21:37:29.256037+0000 (oldest deadline 2024-05-03T21:37:53.956115+0000) 2024-05-03T21:37:59.308 INFO:journalctl@ceph.osd.8.smithi138.stdout:May 03 21:37:59 smithi138 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-8[51920]: 2024-05-03T21:37:59.046+0000 7f4b828df700 -1 osd.8 87 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-05-03T21:37:29.505823+0000 front 2024-05-03T21:37:29.505840+0000 (oldest deadline 2024-05-03T21:37:50.605770+0000) 2024-05-03T21:37:59.308 INFO:journalctl@ceph.osd.8.smithi138.stdout:May 03 21:37:59 smithi138 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-8[51920]: 2024-05-03T21:37:59.046+0000 7f4b828df700 -1 osd.8 87 heartbeat_check: no reply from 172.21.15.138:6804 osd.2 since back 2024-05-03T21:37:34.706687+0000 front 2024-05-03T21:37:34.706869+0000 (oldest deadline 2024-05-03T21:37:58.206880+0000) 2024-05-03T21:37:59.495 INFO:journalctl@ceph.osd.10.smithi082.stdout:May 03 21:37:59 smithi082 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-10[57676]: 2024-05-03T21:37:59.104+0000 7f790b86c700 -1 osd.10 87 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-03T21:37:18.101114+0000 front 2024-05-03T21:37:18.101101+0000 (oldest deadline 2024-05-03T21:37:42.800953+0000) 2024-05-03T21:37:59.495 INFO:journalctl@ceph.osd.10.smithi082.stdout:May 03 21:37:59 smithi082 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-10[57676]: 2024-05-03T21:37:59.104+0000 7f790b86c700 -1 osd.10 87 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-05-03T21:37:28.303836+0000 front 2024-05-03T21:37:28.303941+0000 (oldest deadline 2024-05-03T21:37:51.804075+0000) 2024-05-03T21:37:59.495 INFO:journalctl@ceph.osd.7.smithi082.stdout:May 03 21:37:59 smithi082 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-7[50761]: 2024-05-03T21:37:59.337+0000 7f44b54b5700 -1 osd.7 87 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-05-03T21:37:26.771556+0000 front 2024-05-03T21:37:26.771586+0000 (oldest deadline 2024-05-03T21:37:54.372236+0000) 2024-05-03T21:37:59.495 INFO:journalctl@ceph.osd.7.smithi082.stdout:May 03 21:37:59 smithi082 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-7[50761]: 2024-05-03T21:37:59.337+0000 7f44b54b5700 -1 osd.7 87 heartbeat_check: no reply from 172.21.15.138:6804 osd.2 since back 2024-05-03T21:37:34.372911+0000 front 2024-05-03T21:37:34.372806+0000 (oldest deadline 2024-05-03T21:37:59.072727+0000) 2024-05-03T21:37:59.574 INFO:journalctl@ceph.osd.5.smithi138.stdout:May 03 21:37:59 smithi138 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-5[45227]: 2024-05-03T21:37:59.305+0000 7fe1cfb9f700 -1 osd.5 87 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-03T21:37:20.947718+0000 front 2024-05-03T21:37:20.947686+0000 (oldest deadline 2024-05-03T21:37:45.548290+0000) 2024-05-03T21:37:59.574 INFO:journalctl@ceph.osd.5.smithi138.stdout:May 03 21:37:59 smithi138 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-5[45227]: 2024-05-03T21:37:59.305+0000 7fe1cfb9f700 -1 osd.5 87 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-05-03T21:37:25.549038+0000 front 2024-05-03T21:37:25.549065+0000 (oldest deadline 2024-05-03T21:37:50.249031+0000) 2024-05-03T21:37:59.970 INFO:journalctl@ceph.osd.6.smithi067.stdout:May 03 21:37:59 smithi067 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-6[55180]: 2024-05-03T21:37:59.594+0000 7f389239a700 -1 osd.6 87 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-03T21:37:19.073534+0000 front 2024-05-03T21:37:19.073793+0000 (oldest deadline 2024-05-03T21:37:46.673918+0000) 2024-05-03T21:37:59.971 INFO:journalctl@ceph.osd.6.smithi067.stdout:May 03 21:37:59 smithi067 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-6[55180]: 2024-05-03T21:37:59.594+0000 7f389239a700 -1 osd.6 87 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-05-03T21:37:26.674547+0000 front 2024-05-03T21:37:26.674628+0000 (oldest deadline 2024-05-03T21:37:52.574465+0000) 2024-05-03T21:37:59.971 INFO:journalctl@ceph.osd.9.smithi067.stdout:May 03 21:37:59 smithi067 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-9[62235]: 2024-05-03T21:37:59.675+0000 7fcff92a2700 -1 osd.9 87 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-03T21:37:20.101372+0000 front 2024-05-03T21:37:20.101400+0000 (oldest deadline 2024-05-03T21:37:44.801519+0000) 2024-05-03T21:37:59.971 INFO:journalctl@ceph.osd.9.smithi067.stdout:May 03 21:37:59 smithi067 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-9[62235]: 2024-05-03T21:37:59.675+0000 7fcff92a2700 -1 osd.9 87 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-05-03T21:37:27.702901+0000 front 2024-05-03T21:37:27.702918+0000 (oldest deadline 2024-05-03T21:37:52.402882+0000) 2024-05-03T21:38:00.324 INFO:journalctl@ceph.osd.8.smithi138.stdout:May 03 21:38:00 smithi138 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-8[51920]: 2024-05-03T21:38:00.040+0000 7f4b828df700 -1 osd.8 87 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-05-03T21:37:29.505823+0000 front 2024-05-03T21:37:29.505840+0000 (oldest deadline 2024-05-03T21:37:50.605770+0000) 2024-05-03T21:38:00.324 INFO:journalctl@ceph.osd.8.smithi138.stdout:May 03 21:38:00 smithi138 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-8[51920]: 2024-05-03T21:38:00.040+0000 7f4b828df700 -1 osd.8 87 heartbeat_check: no reply from 172.21.15.138:6804 osd.2 since back 2024-05-03T21:37:34.706687+0000 front 2024-05-03T21:37:34.706869+0000 (oldest deadline 2024-05-03T21:37:58.206880+0000) 2024-05-03T21:38:00.325 INFO:journalctl@ceph.osd.5.smithi138.stdout:May 03 21:38:00 smithi138 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-5[45227]: 2024-05-03T21:38:00.260+0000 7fe1cfb9f700 -1 osd.5 87 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-03T21:37:20.947718+0000 front 2024-05-03T21:37:20.947686+0000 (oldest deadline 2024-05-03T21:37:45.548290+0000) 2024-05-03T21:38:00.325 INFO:journalctl@ceph.osd.5.smithi138.stdout:May 03 21:38:00 smithi138 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-5[45227]: 2024-05-03T21:38:00.260+0000 7fe1cfb9f700 -1 osd.5 87 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-05-03T21:37:25.549038+0000 front 2024-05-03T21:37:25.549065+0000 (oldest deadline 2024-05-03T21:37:50.249031+0000) 2024-05-03T21:38:00.325 INFO:journalctl@ceph.osd.11.smithi138.stdout:May 03 21:38:00 smithi138 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-11[58547]: 2024-05-03T21:38:00.089+0000 7fdfdcb59700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-03T21:37:21.054826+0000 front 2024-05-03T21:37:21.054670+0000 (oldest deadline 2024-05-03T21:37:45.154864+0000) 2024-05-03T21:38:00.325 INFO:journalctl@ceph.osd.11.smithi138.stdout:May 03 21:38:00 smithi138 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-11[58547]: 2024-05-03T21:38:00.089+0000 7fdfdcb59700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-05-03T21:37:29.256054+0000 front 2024-05-03T21:37:29.256037+0000 (oldest deadline 2024-05-03T21:37:53.956115+0000) 2024-05-03T21:38:00.325 INFO:journalctl@ceph.osd.11.smithi138.stdout:May 03 21:38:00 smithi138 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-11[58547]: 2024-05-03T21:38:00.089+0000 7fdfdcb59700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.138:6804 osd.2 since back 2024-05-03T21:37:33.956668+0000 front 2024-05-03T21:37:33.956702+0000 (oldest deadline 2024-05-03T21:37:59.856824+0000) 2024-05-03T21:38:00.495 INFO:journalctl@ceph.osd.7.smithi082.stdout:May 03 21:38:00 smithi082 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-7[50761]: 2024-05-03T21:38:00.309+0000 7f44b54b5700 -1 osd.7 87 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-05-03T21:37:26.771556+0000 front 2024-05-03T21:37:26.771586+0000 (oldest deadline 2024-05-03T21:37:54.372236+0000) 2024-05-03T21:38:00.495 INFO:journalctl@ceph.osd.7.smithi082.stdout:May 03 21:38:00 smithi082 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-7[50761]: 2024-05-03T21:38:00.309+0000 7f44b54b5700 -1 osd.7 87 heartbeat_check: no reply from 172.21.15.138:6804 osd.2 since back 2024-05-03T21:37:34.372911+0000 front 2024-05-03T21:37:34.372806+0000 (oldest deadline 2024-05-03T21:37:59.072727+0000) 2024-05-03T21:38:00.495 INFO:journalctl@ceph.osd.10.smithi082.stdout:May 03 21:38:00 smithi082 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-10[57676]: 2024-05-03T21:38:00.103+0000 7f790b86c700 -1 osd.10 87 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-03T21:37:18.101114+0000 front 2024-05-03T21:37:18.101101+0000 (oldest deadline 2024-05-03T21:37:42.800953+0000) 2024-05-03T21:38:00.495 INFO:journalctl@ceph.osd.10.smithi082.stdout:May 03 21:38:00 smithi082 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-10[57676]: 2024-05-03T21:38:00.103+0000 7f790b86c700 -1 osd.10 87 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-05-03T21:37:28.303836+0000 front 2024-05-03T21:37:28.303941+0000 (oldest deadline 2024-05-03T21:37:51.804075+0000) 2024-05-03T21:38:00.971 INFO:journalctl@ceph.osd.6.smithi067.stdout:May 03 21:38:00 smithi067 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-6[55180]: 2024-05-03T21:38:00.627+0000 7f389239a700 -1 osd.6 87 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-03T21:37:19.073534+0000 front 2024-05-03T21:37:19.073793+0000 (oldest deadline 2024-05-03T21:37:46.673918+0000) 2024-05-03T21:38:00.971 INFO:journalctl@ceph.osd.6.smithi067.stdout:May 03 21:38:00 smithi067 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-6[55180]: 2024-05-03T21:38:00.627+0000 7f389239a700 -1 osd.6 87 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-05-03T21:37:26.674547+0000 front 2024-05-03T21:37:26.674628+0000 (oldest deadline 2024-05-03T21:37:52.574465+0000) 2024-05-03T21:38:00.971 INFO:journalctl@ceph.osd.9.smithi067.stdout:May 03 21:38:00 smithi067 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-9[62235]: 2024-05-03T21:38:00.677+0000 7fcff92a2700 -1 osd.9 87 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-03T21:37:20.101372+0000 front 2024-05-03T21:37:20.101400+0000 (oldest deadline 2024-05-03T21:37:44.801519+0000) 2024-05-03T21:38:00.971 INFO:journalctl@ceph.osd.9.smithi067.stdout:May 03 21:38:00 smithi067 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-9[62235]: 2024-05-03T21:38:00.677+0000 7fcff92a2700 -1 osd.9 87 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-05-03T21:37:27.702901+0000 front 2024-05-03T21:37:27.702918+0000 (oldest deadline 2024-05-03T21:37:52.402882+0000) 2024-05-03T21:38:00.971 INFO:journalctl@ceph.osd.9.smithi067.stdout:May 03 21:38:00 smithi067 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-9[62235]: 2024-05-03T21:38:00.677+0000 7fcff92a2700 -1 osd.9 87 heartbeat_check: no reply from 172.21.15.138:6804 osd.2 since back 2024-05-03T21:37:34.605024+0000 front 2024-05-03T21:37:34.604986+0000 (oldest deadline 2024-05-03T21:37:59.904623+0000) 2024-05-03T21:38:01.324 INFO:journalctl@ceph.osd.5.smithi138.stdout:May 03 21:38:01 smithi138 podman[68878]: 2024-05-03 21:38:01.054877184 +0000 UTC m=+5.558720838 container died a25605f9bc965d9904b2aa051709dc575481ffee1a6cb60124439c53bb73a943 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:68e15ae152ef5c805c733e76da8fc1d3dd4ee669d5ba025f457e05725baa24dd, name=ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-5, org.label-schema.schema-version=1.0, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, RELEASE=wip-rishabh-testing-20240501.193033-9a047fe, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240430, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.vendor=CentOS) 2024-05-03T21:38:01.324 INFO:journalctl@ceph.osd.8.smithi138.stdout:May 03 21:38:01 smithi138 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-8[51920]: 2024-05-03T21:38:01.069+0000 7f4b828df700 -1 osd.8 87 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-05-03T21:37:29.505823+0000 front 2024-05-03T21:37:29.505840+0000 (oldest deadline 2024-05-03T21:37:50.605770+0000) 2024-05-03T21:38:01.325 INFO:journalctl@ceph.osd.8.smithi138.stdout:May 03 21:38:01 smithi138 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-8[51920]: 2024-05-03T21:38:01.069+0000 7f4b828df700 -1 osd.8 87 heartbeat_check: no reply from 172.21.15.138:6804 osd.2 since back 2024-05-03T21:37:34.706687+0000 front 2024-05-03T21:37:34.706869+0000 (oldest deadline 2024-05-03T21:37:58.206880+0000) 2024-05-03T21:38:01.325 INFO:journalctl@ceph.osd.11.smithi138.stdout:May 03 21:38:01 smithi138 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-11[58547]: 2024-05-03T21:38:01.053+0000 7fdfdcb59700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-03T21:37:21.054826+0000 front 2024-05-03T21:37:21.054670+0000 (oldest deadline 2024-05-03T21:37:45.154864+0000) 2024-05-03T21:38:01.325 INFO:journalctl@ceph.osd.11.smithi138.stdout:May 03 21:38:01 smithi138 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-11[58547]: 2024-05-03T21:38:01.053+0000 7fdfdcb59700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-05-03T21:37:29.256054+0000 front 2024-05-03T21:37:29.256037+0000 (oldest deadline 2024-05-03T21:37:53.956115+0000) 2024-05-03T21:38:01.325 INFO:journalctl@ceph.osd.11.smithi138.stdout:May 03 21:38:01 smithi138 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-11[58547]: 2024-05-03T21:38:01.053+0000 7fdfdcb59700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.138:6804 osd.2 since back 2024-05-03T21:37:33.956668+0000 front 2024-05-03T21:37:33.956702+0000 (oldest deadline 2024-05-03T21:37:59.856824+0000) 2024-05-03T21:38:01.495 INFO:journalctl@ceph.osd.7.smithi082.stdout:May 03 21:38:01 smithi082 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-7[50761]: 2024-05-03T21:38:01.323+0000 7f44b54b5700 -1 osd.7 87 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-05-03T21:37:26.771556+0000 front 2024-05-03T21:37:26.771586+0000 (oldest deadline 2024-05-03T21:37:54.372236+0000) 2024-05-03T21:38:01.495 INFO:journalctl@ceph.osd.7.smithi082.stdout:May 03 21:38:01 smithi082 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-7[50761]: 2024-05-03T21:38:01.323+0000 7f44b54b5700 -1 osd.7 87 heartbeat_check: no reply from 172.21.15.138:6804 osd.2 since back 2024-05-03T21:37:34.372911+0000 front 2024-05-03T21:37:34.372806+0000 (oldest deadline 2024-05-03T21:37:59.072727+0000) 2024-05-03T21:38:01.495 INFO:journalctl@ceph.osd.10.smithi082.stdout:May 03 21:38:01 smithi082 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-10[57676]: 2024-05-03T21:38:01.084+0000 7f790b86c700 -1 osd.10 87 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-03T21:37:18.101114+0000 front 2024-05-03T21:37:18.101101+0000 (oldest deadline 2024-05-03T21:37:42.800953+0000) 2024-05-03T21:38:01.496 INFO:journalctl@ceph.osd.10.smithi082.stdout:May 03 21:38:01 smithi082 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-10[57676]: 2024-05-03T21:38:01.084+0000 7f790b86c700 -1 osd.10 87 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-05-03T21:37:28.303836+0000 front 2024-05-03T21:37:28.303941+0000 (oldest deadline 2024-05-03T21:37:51.804075+0000) 2024-05-03T21:38:01.683 INFO:journalctl@ceph.osd.5.smithi138.stdout:May 03 21:38:01 smithi138 podman[68878]: 2024-05-03 21:38:01.414429732 +0000 UTC m=+5.918273388 container cleanup a25605f9bc965d9904b2aa051709dc575481ffee1a6cb60124439c53bb73a943 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:68e15ae152ef5c805c733e76da8fc1d3dd4ee669d5ba025f457e05725baa24dd, name=ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-5, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_CLEAN=True, RELEASE=wip-rishabh-testing-20240501.193033-9a047fe, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240430, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b) 2024-05-03T21:38:01.683 INFO:journalctl@ceph.osd.5.smithi138.stdout:May 03 21:38:01 smithi138 bash[68878]: ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-5 2024-05-03T21:38:01.971 INFO:journalctl@ceph.osd.6.smithi067.stdout:May 03 21:38:01 smithi067 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-6[55180]: 2024-05-03T21:38:01.577+0000 7f389239a700 -1 osd.6 87 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-03T21:37:19.073534+0000 front 2024-05-03T21:37:19.073793+0000 (oldest deadline 2024-05-03T21:37:46.673918+0000) 2024-05-03T21:38:01.971 INFO:journalctl@ceph.osd.6.smithi067.stdout:May 03 21:38:01 smithi067 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-6[55180]: 2024-05-03T21:38:01.577+0000 7f389239a700 -1 osd.6 87 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-05-03T21:37:26.674547+0000 front 2024-05-03T21:37:26.674628+0000 (oldest deadline 2024-05-03T21:37:52.574465+0000) 2024-05-03T21:38:01.971 INFO:journalctl@ceph.osd.6.smithi067.stdout:May 03 21:38:01 smithi067 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-6[55180]: 2024-05-03T21:38:01.577+0000 7f389239a700 -1 osd.6 87 heartbeat_check: no reply from 172.21.15.138:6804 osd.2 since back 2024-05-03T21:37:35.375943+0000 front 2024-05-03T21:37:35.375805+0000 (oldest deadline 2024-05-03T21:38:01.275874+0000) 2024-05-03T21:38:01.971 INFO:journalctl@ceph.osd.9.smithi067.stdout:May 03 21:38:01 smithi067 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-9[62235]: 2024-05-03T21:38:01.650+0000 7fcff92a2700 -1 osd.9 87 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-03T21:37:20.101372+0000 front 2024-05-03T21:37:20.101400+0000 (oldest deadline 2024-05-03T21:37:44.801519+0000) 2024-05-03T21:38:01.971 INFO:journalctl@ceph.osd.9.smithi067.stdout:May 03 21:38:01 smithi067 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-9[62235]: 2024-05-03T21:38:01.650+0000 7fcff92a2700 -1 osd.9 87 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-05-03T21:37:27.702901+0000 front 2024-05-03T21:37:27.702918+0000 (oldest deadline 2024-05-03T21:37:52.402882+0000) 2024-05-03T21:38:01.972 INFO:journalctl@ceph.osd.9.smithi067.stdout:May 03 21:38:01 smithi067 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-9[62235]: 2024-05-03T21:38:01.650+0000 7fcff92a2700 -1 osd.9 87 heartbeat_check: no reply from 172.21.15.138:6804 osd.2 since back 2024-05-03T21:37:34.605024+0000 front 2024-05-03T21:37:34.604986+0000 (oldest deadline 2024-05-03T21:37:59.904623+0000) 2024-05-03T21:38:02.007 INFO:journalctl@ceph.osd.5.smithi138.stdout:May 03 21:38:01 smithi138 podman[68895]: 2024-05-03 21:38:01.682250518 +0000 UTC m=+0.622123966 container remove a25605f9bc965d9904b2aa051709dc575481ffee1a6cb60124439c53bb73a943 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:68e15ae152ef5c805c733e76da8fc1d3dd4ee669d5ba025f457e05725baa24dd, name=ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-5, org.label-schema.build-date=20240430, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , RELEASE=wip-rishabh-testing-20240501.193033-9a047fe, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b) 2024-05-03T21:38:02.295 INFO:journalctl@ceph.osd.8.smithi138.stdout:May 03 21:38:02 smithi138 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-8[51920]: 2024-05-03T21:38:02.106+0000 7f4b828df700 -1 osd.8 87 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-05-03T21:37:29.505823+0000 front 2024-05-03T21:37:29.505840+0000 (oldest deadline 2024-05-03T21:37:50.605770+0000) 2024-05-03T21:38:02.295 INFO:journalctl@ceph.osd.8.smithi138.stdout:May 03 21:38:02 smithi138 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-8[51920]: 2024-05-03T21:38:02.106+0000 7f4b828df700 -1 osd.8 87 heartbeat_check: no reply from 172.21.15.138:6804 osd.2 since back 2024-05-03T21:37:34.706687+0000 front 2024-05-03T21:37:34.706869+0000 (oldest deadline 2024-05-03T21:37:58.206880+0000) 2024-05-03T21:38:02.295 INFO:journalctl@ceph.osd.11.smithi138.stdout:May 03 21:38:02 smithi138 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-11[58547]: 2024-05-03T21:38:02.004+0000 7fdfdcb59700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-03T21:37:21.054826+0000 front 2024-05-03T21:37:21.054670+0000 (oldest deadline 2024-05-03T21:37:45.154864+0000) 2024-05-03T21:38:02.296 INFO:journalctl@ceph.osd.11.smithi138.stdout:May 03 21:38:02 smithi138 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-11[58547]: 2024-05-03T21:38:02.004+0000 7fdfdcb59700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-05-03T21:37:29.256054+0000 front 2024-05-03T21:37:29.256037+0000 (oldest deadline 2024-05-03T21:37:53.956115+0000) 2024-05-03T21:38:02.296 INFO:journalctl@ceph.osd.11.smithi138.stdout:May 03 21:38:02 smithi138 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-11[58547]: 2024-05-03T21:38:02.004+0000 7fdfdcb59700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.138:6804 osd.2 since back 2024-05-03T21:37:33.956668+0000 front 2024-05-03T21:37:33.956702+0000 (oldest deadline 2024-05-03T21:37:59.856824+0000) 2024-05-03T21:38:02.296 INFO:journalctl@ceph.osd.5.smithi138.stdout:May 03 21:38:02 smithi138 podman[68983]: 2024-05-03 21:38:01.932843522 +0000 UTC m=+0.017883092 image pull ee4d846202496ba6096593e8e8becc0700a5eb7ebd95536031187f5f337d4012 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:68e15ae152ef5c805c733e76da8fc1d3dd4ee669d5ba025f457e05725baa24dd 2024-05-03T21:38:02.296 INFO:journalctl@ceph.osd.5.smithi138.stdout:May 03 21:38:02 smithi138 podman[68983]: 2024-05-03 21:38:02.05887459 +0000 UTC m=+0.143914150 container create 1c52db57a038882957f9f84e4d9200f8871dd84e05d6eb4b1805ecff9e3fec93 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:68e15ae152ef5c805c733e76da8fc1d3dd4ee669d5ba025f457e05725baa24dd, name=ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-5-deactivate, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.build-date=20240430, GIT_CLEAN=True, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-rishabh-testing-20240501.193033-9a047fe, io.buildah.version=1.33.5) 2024-05-03T21:38:02.296 INFO:journalctl@ceph.osd.5.smithi138.stdout:May 03 21:38:02 smithi138 podman[68983]: 2024-05-03 21:38:02.236242511 +0000 UTC m=+0.321282080 container init 1c52db57a038882957f9f84e4d9200f8871dd84e05d6eb4b1805ecff9e3fec93 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:68e15ae152ef5c805c733e76da8fc1d3dd4ee669d5ba025f457e05725baa24dd, name=ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-5-deactivate, org.label-schema.build-date=20240430, ceph=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_BRANCH=HEAD, RELEASE=wip-rishabh-testing-20240501.193033-9a047fe, org.label-schema.license=GPLv2) 2024-05-03T21:38:02.297 INFO:journalctl@ceph.osd.5.smithi138.stdout:May 03 21:38:02 smithi138 podman[68983]: 2024-05-03 21:38:02.239280113 +0000 UTC m=+0.324319681 container start 1c52db57a038882957f9f84e4d9200f8871dd84e05d6eb4b1805ecff9e3fec93 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:68e15ae152ef5c805c733e76da8fc1d3dd4ee669d5ba025f457e05725baa24dd, name=ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-5-deactivate, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_CLEAN=True, ceph=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240430, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, RELEASE=wip-rishabh-testing-20240501.193033-9a047fe, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b) 2024-05-03T21:38:02.297 INFO:journalctl@ceph.osd.5.smithi138.stdout:May 03 21:38:02 smithi138 podman[68983]: 2024-05-03 21:38:02.294288636 +0000 UTC m=+0.379328197 container attach 1c52db57a038882957f9f84e4d9200f8871dd84e05d6eb4b1805ecff9e3fec93 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:68e15ae152ef5c805c733e76da8fc1d3dd4ee669d5ba025f457e05725baa24dd, name=ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-5-deactivate, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, ceph=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240430, RELEASE=wip-rishabh-testing-20240501.193033-9a047fe, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b) 2024-05-03T21:38:02.495 INFO:journalctl@ceph.osd.7.smithi082.stdout:May 03 21:38:02 smithi082 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-7[50761]: 2024-05-03T21:38:02.292+0000 7f44b54b5700 -1 osd.7 87 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-05-03T21:37:26.771556+0000 front 2024-05-03T21:37:26.771586+0000 (oldest deadline 2024-05-03T21:37:54.372236+0000) 2024-05-03T21:38:02.495 INFO:journalctl@ceph.osd.7.smithi082.stdout:May 03 21:38:02 smithi082 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-7[50761]: 2024-05-03T21:38:02.292+0000 7f44b54b5700 -1 osd.7 87 heartbeat_check: no reply from 172.21.15.138:6804 osd.2 since back 2024-05-03T21:37:34.372911+0000 front 2024-05-03T21:37:34.372806+0000 (oldest deadline 2024-05-03T21:37:59.072727+0000) 2024-05-03T21:38:02.495 INFO:journalctl@ceph.osd.10.smithi082.stdout:May 03 21:38:02 smithi082 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-10[57676]: 2024-05-03T21:38:02.043+0000 7f790b86c700 -1 osd.10 87 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-03T21:37:18.101114+0000 front 2024-05-03T21:37:18.101101+0000 (oldest deadline 2024-05-03T21:37:42.800953+0000) 2024-05-03T21:38:02.495 INFO:journalctl@ceph.osd.10.smithi082.stdout:May 03 21:38:02 smithi082 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-10[57676]: 2024-05-03T21:38:02.043+0000 7f790b86c700 -1 osd.10 87 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-05-03T21:37:28.303836+0000 front 2024-05-03T21:37:28.303941+0000 (oldest deadline 2024-05-03T21:37:51.804075+0000) 2024-05-03T21:38:02.496 INFO:journalctl@ceph.osd.10.smithi082.stdout:May 03 21:38:02 smithi082 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-10[57676]: 2024-05-03T21:38:02.043+0000 7f790b86c700 -1 osd.10 87 heartbeat_check: no reply from 172.21.15.138:6804 osd.2 since back 2024-05-03T21:37:36.406076+0000 front 2024-05-03T21:37:36.406028+0000 (oldest deadline 2024-05-03T21:38:01.705914+0000) 2024-05-03T21:38:02.574 INFO:journalctl@ceph.osd.5.smithi138.stdout:May 03 21:38:02 smithi138 podman[68983]: 2024-05-03 21:38:02.531411142 +0000 UTC m=+0.616450695 container died 1c52db57a038882957f9f84e4d9200f8871dd84e05d6eb4b1805ecff9e3fec93 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:68e15ae152ef5c805c733e76da8fc1d3dd4ee669d5ba025f457e05725baa24dd, name=ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-5-deactivate, RELEASE=wip-rishabh-testing-20240501.193033-9a047fe, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.build-date=20240430, ceph=True) 2024-05-03T21:38:02.971 INFO:journalctl@ceph.osd.6.smithi067.stdout:May 03 21:38:02 smithi067 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-6[55180]: 2024-05-03T21:38:02.588+0000 7f389239a700 -1 osd.6 87 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-03T21:37:19.073534+0000 front 2024-05-03T21:37:19.073793+0000 (oldest deadline 2024-05-03T21:37:46.673918+0000) 2024-05-03T21:38:02.971 INFO:journalctl@ceph.osd.6.smithi067.stdout:May 03 21:38:02 smithi067 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-6[55180]: 2024-05-03T21:38:02.588+0000 7f389239a700 -1 osd.6 87 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-05-03T21:37:26.674547+0000 front 2024-05-03T21:37:26.674628+0000 (oldest deadline 2024-05-03T21:37:52.574465+0000) 2024-05-03T21:38:02.971 INFO:journalctl@ceph.osd.6.smithi067.stdout:May 03 21:38:02 smithi067 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-6[55180]: 2024-05-03T21:38:02.588+0000 7f389239a700 -1 osd.6 87 heartbeat_check: no reply from 172.21.15.138:6804 osd.2 since back 2024-05-03T21:37:35.375943+0000 front 2024-05-03T21:37:35.375805+0000 (oldest deadline 2024-05-03T21:38:01.275874+0000) 2024-05-03T21:38:02.971 INFO:journalctl@ceph.osd.9.smithi067.stdout:May 03 21:38:02 smithi067 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-9[62235]: 2024-05-03T21:38:02.660+0000 7fcff92a2700 -1 osd.9 87 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-03T21:37:20.101372+0000 front 2024-05-03T21:37:20.101400+0000 (oldest deadline 2024-05-03T21:37:44.801519+0000) 2024-05-03T21:38:02.971 INFO:journalctl@ceph.osd.9.smithi067.stdout:May 03 21:38:02 smithi067 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-9[62235]: 2024-05-03T21:38:02.660+0000 7fcff92a2700 -1 osd.9 87 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-05-03T21:37:27.702901+0000 front 2024-05-03T21:37:27.702918+0000 (oldest deadline 2024-05-03T21:37:52.402882+0000) 2024-05-03T21:38:02.972 INFO:journalctl@ceph.osd.9.smithi067.stdout:May 03 21:38:02 smithi067 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-9[62235]: 2024-05-03T21:38:02.660+0000 7fcff92a2700 -1 osd.9 87 heartbeat_check: no reply from 172.21.15.138:6804 osd.2 since back 2024-05-03T21:37:34.605024+0000 front 2024-05-03T21:37:34.604986+0000 (oldest deadline 2024-05-03T21:37:59.904623+0000) 2024-05-03T21:38:03.065 INFO:journalctl@ceph.osd.11.smithi138.stdout:May 03 21:38:03 smithi138 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-11[58547]: 2024-05-03T21:38:03.030+0000 7fdfdcb59700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-03T21:37:21.054826+0000 front 2024-05-03T21:37:21.054670+0000 (oldest deadline 2024-05-03T21:37:45.154864+0000) 2024-05-03T21:38:03.065 INFO:journalctl@ceph.osd.11.smithi138.stdout:May 03 21:38:03 smithi138 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-11[58547]: 2024-05-03T21:38:03.030+0000 7fdfdcb59700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-05-03T21:37:29.256054+0000 front 2024-05-03T21:37:29.256037+0000 (oldest deadline 2024-05-03T21:37:53.956115+0000) 2024-05-03T21:38:03.065 INFO:journalctl@ceph.osd.11.smithi138.stdout:May 03 21:38:03 smithi138 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-11[58547]: 2024-05-03T21:38:03.030+0000 7fdfdcb59700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.138:6804 osd.2 since back 2024-05-03T21:37:33.956668+0000 front 2024-05-03T21:37:33.956702+0000 (oldest deadline 2024-05-03T21:37:59.856824+0000) 2024-05-03T21:38:03.088 DEBUG:teuthology.orchestra.run.smithi138:> sudo pkill -f 'journalctl -f -n 0 -u ceph-713bc57a-0991-11ef-bc97-c7b262605968@osd.5.service' 2024-05-03T21:38:03.166 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-03T21:38:03.167 INFO:tasks.cephadm.osd.5:Stopped osd.5 2024-05-03T21:38:03.167 INFO:tasks.cephadm.osd.6:Stopping osd.6... 2024-05-03T21:38:03.167 DEBUG:teuthology.orchestra.run.smithi067:> sudo systemctl stop ceph-713bc57a-0991-11ef-bc97-c7b262605968@osd.6 2024-05-03T21:38:03.311 INFO:journalctl@ceph.osd.10.smithi082.stdout:May 03 21:38:03 smithi082 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-10[57676]: 2024-05-03T21:38:03.024+0000 7f790b86c700 -1 osd.10 87 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-03T21:37:18.101114+0000 front 2024-05-03T21:37:18.101101+0000 (oldest deadline 2024-05-03T21:37:42.800953+0000) 2024-05-03T21:38:03.311 INFO:journalctl@ceph.osd.10.smithi082.stdout:May 03 21:38:03 smithi082 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-10[57676]: 2024-05-03T21:38:03.024+0000 7f790b86c700 -1 osd.10 87 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-05-03T21:37:28.303836+0000 front 2024-05-03T21:37:28.303941+0000 (oldest deadline 2024-05-03T21:37:51.804075+0000) 2024-05-03T21:38:03.311 INFO:journalctl@ceph.osd.10.smithi082.stdout:May 03 21:38:03 smithi082 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-10[57676]: 2024-05-03T21:38:03.024+0000 7f790b86c700 -1 osd.10 87 heartbeat_check: no reply from 172.21.15.138:6804 osd.2 since back 2024-05-03T21:37:36.406076+0000 front 2024-05-03T21:37:36.406028+0000 (oldest deadline 2024-05-03T21:38:01.705914+0000) 2024-05-03T21:38:03.324 INFO:journalctl@ceph.osd.8.smithi138.stdout:May 03 21:38:03 smithi138 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-8[51920]: 2024-05-03T21:38:03.153+0000 7f4b828df700 -1 osd.8 87 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-05-03T21:37:29.505823+0000 front 2024-05-03T21:37:29.505840+0000 (oldest deadline 2024-05-03T21:37:50.605770+0000) 2024-05-03T21:38:03.324 INFO:journalctl@ceph.osd.8.smithi138.stdout:May 03 21:38:03 smithi138 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-8[51920]: 2024-05-03T21:38:03.153+0000 7f4b828df700 -1 osd.8 87 heartbeat_check: no reply from 172.21.15.138:6804 osd.2 since back 2024-05-03T21:37:34.706687+0000 front 2024-05-03T21:37:34.706869+0000 (oldest deadline 2024-05-03T21:37:58.206880+0000) 2024-05-03T21:38:03.467 INFO:journalctl@ceph.osd.6.smithi067.stdout:May 03 21:38:03 smithi067 systemd[1]: Stopping Ceph osd.6 for 713bc57a-0991-11ef-bc97-c7b262605968... 2024-05-03T21:38:03.721 INFO:journalctl@ceph.osd.9.smithi067.stdout:May 03 21:38:03 smithi067 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-9[62235]: 2024-05-03T21:38:03.674+0000 7fcff92a2700 -1 osd.9 87 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-03T21:37:20.101372+0000 front 2024-05-03T21:37:20.101400+0000 (oldest deadline 2024-05-03T21:37:44.801519+0000) 2024-05-03T21:38:03.721 INFO:journalctl@ceph.osd.9.smithi067.stdout:May 03 21:38:03 smithi067 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-9[62235]: 2024-05-03T21:38:03.674+0000 7fcff92a2700 -1 osd.9 87 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-05-03T21:37:27.702901+0000 front 2024-05-03T21:37:27.702918+0000 (oldest deadline 2024-05-03T21:37:52.402882+0000) 2024-05-03T21:38:03.721 INFO:journalctl@ceph.osd.9.smithi067.stdout:May 03 21:38:03 smithi067 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-9[62235]: 2024-05-03T21:38:03.674+0000 7fcff92a2700 -1 osd.9 87 heartbeat_check: no reply from 172.21.15.138:6804 osd.2 since back 2024-05-03T21:37:34.605024+0000 front 2024-05-03T21:37:34.604986+0000 (oldest deadline 2024-05-03T21:37:59.904623+0000) 2024-05-03T21:38:03.721 INFO:journalctl@ceph.osd.6.smithi067.stdout:May 03 21:38:03 smithi067 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-6[55180]: 2024-05-03T21:38:03.467+0000 7f3895da6700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-03T21:38:03.721 INFO:journalctl@ceph.osd.6.smithi067.stdout:May 03 21:38:03 smithi067 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-6[55180]: 2024-05-03T21:38:03.467+0000 7f3895da6700 -1 osd.6 87 *** Got signal Terminated *** 2024-05-03T21:38:03.721 INFO:journalctl@ceph.osd.6.smithi067.stdout:May 03 21:38:03 smithi067 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-6[55180]: 2024-05-03T21:38:03.467+0000 7f3895da6700 -1 osd.6 87 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-05-03T21:38:03.722 INFO:journalctl@ceph.osd.6.smithi067.stdout:May 03 21:38:03 smithi067 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-6[55180]: 2024-05-03T21:38:03.595+0000 7f389239a700 -1 osd.6 87 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-03T21:37:19.073534+0000 front 2024-05-03T21:37:19.073793+0000 (oldest deadline 2024-05-03T21:37:46.673918+0000) 2024-05-03T21:38:03.722 INFO:journalctl@ceph.osd.6.smithi067.stdout:May 03 21:38:03 smithi067 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-6[55180]: 2024-05-03T21:38:03.595+0000 7f389239a700 -1 osd.6 87 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-05-03T21:37:26.674547+0000 front 2024-05-03T21:37:26.674628+0000 (oldest deadline 2024-05-03T21:37:52.574465+0000) 2024-05-03T21:38:03.722 INFO:journalctl@ceph.osd.6.smithi067.stdout:May 03 21:38:03 smithi067 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-6[55180]: 2024-05-03T21:38:03.595+0000 7f389239a700 -1 osd.6 87 heartbeat_check: no reply from 172.21.15.138:6804 osd.2 since back 2024-05-03T21:37:35.375943+0000 front 2024-05-03T21:37:35.375805+0000 (oldest deadline 2024-05-03T21:38:01.275874+0000) 2024-05-03T21:38:03.745 INFO:journalctl@ceph.osd.7.smithi082.stdout:May 03 21:38:03 smithi082 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-7[50761]: 2024-05-03T21:38:03.309+0000 7f44b54b5700 -1 osd.7 87 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-05-03T21:37:26.771556+0000 front 2024-05-03T21:37:26.771586+0000 (oldest deadline 2024-05-03T21:37:54.372236+0000) 2024-05-03T21:38:03.745 INFO:journalctl@ceph.osd.7.smithi082.stdout:May 03 21:38:03 smithi082 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-7[50761]: 2024-05-03T21:38:03.309+0000 7f44b54b5700 -1 osd.7 87 heartbeat_check: no reply from 172.21.15.138:6804 osd.2 since back 2024-05-03T21:37:34.372911+0000 front 2024-05-03T21:37:34.372806+0000 (oldest deadline 2024-05-03T21:37:59.072727+0000) 2024-05-03T21:38:04.245 INFO:journalctl@ceph.osd.10.smithi082.stdout:May 03 21:38:03 smithi082 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-10[57676]: 2024-05-03T21:38:03.981+0000 7f790b86c700 -1 osd.10 87 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-03T21:37:18.101114+0000 front 2024-05-03T21:37:18.101101+0000 (oldest deadline 2024-05-03T21:37:42.800953+0000) 2024-05-03T21:38:04.245 INFO:journalctl@ceph.osd.10.smithi082.stdout:May 03 21:38:03 smithi082 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-10[57676]: 2024-05-03T21:38:03.981+0000 7f790b86c700 -1 osd.10 87 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-05-03T21:37:28.303836+0000 front 2024-05-03T21:37:28.303941+0000 (oldest deadline 2024-05-03T21:37:51.804075+0000) 2024-05-03T21:38:04.245 INFO:journalctl@ceph.osd.10.smithi082.stdout:May 03 21:38:03 smithi082 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-10[57676]: 2024-05-03T21:38:03.981+0000 7f790b86c700 -1 osd.10 87 heartbeat_check: no reply from 172.21.15.138:6804 osd.2 since back 2024-05-03T21:37:36.406076+0000 front 2024-05-03T21:37:36.406028+0000 (oldest deadline 2024-05-03T21:38:01.705914+0000) 2024-05-03T21:38:04.324 INFO:journalctl@ceph.osd.8.smithi138.stdout:May 03 21:38:04 smithi138 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-8[51920]: 2024-05-03T21:38:04.177+0000 7f4b828df700 -1 osd.8 87 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-05-03T21:37:29.505823+0000 front 2024-05-03T21:37:29.505840+0000 (oldest deadline 2024-05-03T21:37:50.605770+0000) 2024-05-03T21:38:04.324 INFO:journalctl@ceph.osd.8.smithi138.stdout:May 03 21:38:04 smithi138 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-8[51920]: 2024-05-03T21:38:04.177+0000 7f4b828df700 -1 osd.8 87 heartbeat_check: no reply from 172.21.15.138:6804 osd.2 since back 2024-05-03T21:37:34.706687+0000 front 2024-05-03T21:37:34.706869+0000 (oldest deadline 2024-05-03T21:37:58.206880+0000) 2024-05-03T21:38:04.325 INFO:journalctl@ceph.osd.11.smithi138.stdout:May 03 21:38:04 smithi138 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-11[58547]: 2024-05-03T21:38:04.046+0000 7fdfdcb59700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-03T21:37:21.054826+0000 front 2024-05-03T21:37:21.054670+0000 (oldest deadline 2024-05-03T21:37:45.154864+0000) 2024-05-03T21:38:04.325 INFO:journalctl@ceph.osd.11.smithi138.stdout:May 03 21:38:04 smithi138 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-11[58547]: 2024-05-03T21:38:04.046+0000 7fdfdcb59700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-05-03T21:37:29.256054+0000 front 2024-05-03T21:37:29.256037+0000 (oldest deadline 2024-05-03T21:37:53.956115+0000) 2024-05-03T21:38:04.325 INFO:journalctl@ceph.osd.11.smithi138.stdout:May 03 21:38:04 smithi138 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-11[58547]: 2024-05-03T21:38:04.046+0000 7fdfdcb59700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.138:6804 osd.2 since back 2024-05-03T21:37:33.956668+0000 front 2024-05-03T21:37:33.956702+0000 (oldest deadline 2024-05-03T21:37:59.856824+0000) 2024-05-03T21:38:04.745 INFO:journalctl@ceph.osd.7.smithi082.stdout:May 03 21:38:04 smithi082 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-7[50761]: 2024-05-03T21:38:04.277+0000 7f44b54b5700 -1 osd.7 87 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-05-03T21:37:26.771556+0000 front 2024-05-03T21:37:26.771586+0000 (oldest deadline 2024-05-03T21:37:54.372236+0000) 2024-05-03T21:38:04.745 INFO:journalctl@ceph.osd.7.smithi082.stdout:May 03 21:38:04 smithi082 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-7[50761]: 2024-05-03T21:38:04.277+0000 7f44b54b5700 -1 osd.7 87 heartbeat_check: no reply from 172.21.15.138:6804 osd.2 since back 2024-05-03T21:37:34.372911+0000 front 2024-05-03T21:37:34.372806+0000 (oldest deadline 2024-05-03T21:37:59.072727+0000) 2024-05-03T21:38:04.971 INFO:journalctl@ceph.osd.9.smithi067.stdout:May 03 21:38:04 smithi067 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-9[62235]: 2024-05-03T21:38:04.654+0000 7fcff92a2700 -1 osd.9 87 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-03T21:37:20.101372+0000 front 2024-05-03T21:37:20.101400+0000 (oldest deadline 2024-05-03T21:37:44.801519+0000) 2024-05-03T21:38:04.971 INFO:journalctl@ceph.osd.9.smithi067.stdout:May 03 21:38:04 smithi067 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-9[62235]: 2024-05-03T21:38:04.654+0000 7fcff92a2700 -1 osd.9 87 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-05-03T21:37:27.702901+0000 front 2024-05-03T21:37:27.702918+0000 (oldest deadline 2024-05-03T21:37:52.402882+0000) 2024-05-03T21:38:04.971 INFO:journalctl@ceph.osd.9.smithi067.stdout:May 03 21:38:04 smithi067 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-9[62235]: 2024-05-03T21:38:04.654+0000 7fcff92a2700 -1 osd.9 87 heartbeat_check: no reply from 172.21.15.138:6804 osd.2 since back 2024-05-03T21:37:34.605024+0000 front 2024-05-03T21:37:34.604986+0000 (oldest deadline 2024-05-03T21:37:59.904623+0000) 2024-05-03T21:38:04.971 INFO:journalctl@ceph.osd.6.smithi067.stdout:May 03 21:38:04 smithi067 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-6[55180]: 2024-05-03T21:38:04.598+0000 7f389239a700 -1 osd.6 87 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-03T21:37:19.073534+0000 front 2024-05-03T21:37:19.073793+0000 (oldest deadline 2024-05-03T21:37:46.673918+0000) 2024-05-03T21:38:04.972 INFO:journalctl@ceph.osd.6.smithi067.stdout:May 03 21:38:04 smithi067 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-6[55180]: 2024-05-03T21:38:04.598+0000 7f389239a700 -1 osd.6 87 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-05-03T21:37:26.674547+0000 front 2024-05-03T21:37:26.674628+0000 (oldest deadline 2024-05-03T21:37:52.574465+0000) 2024-05-03T21:38:04.972 INFO:journalctl@ceph.osd.6.smithi067.stdout:May 03 21:38:04 smithi067 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-6[55180]: 2024-05-03T21:38:04.598+0000 7f389239a700 -1 osd.6 87 heartbeat_check: no reply from 172.21.15.138:6804 osd.2 since back 2024-05-03T21:37:35.375943+0000 front 2024-05-03T21:37:35.375805+0000 (oldest deadline 2024-05-03T21:38:01.275874+0000) 2024-05-03T21:38:05.245 INFO:journalctl@ceph.osd.10.smithi082.stdout:May 03 21:38:04 smithi082 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-10[57676]: 2024-05-03T21:38:04.957+0000 7f790b86c700 -1 osd.10 87 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-03T21:37:18.101114+0000 front 2024-05-03T21:37:18.101101+0000 (oldest deadline 2024-05-03T21:37:42.800953+0000) 2024-05-03T21:38:05.245 INFO:journalctl@ceph.osd.10.smithi082.stdout:May 03 21:38:04 smithi082 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-10[57676]: 2024-05-03T21:38:04.957+0000 7f790b86c700 -1 osd.10 87 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-05-03T21:37:28.303836+0000 front 2024-05-03T21:37:28.303941+0000 (oldest deadline 2024-05-03T21:37:51.804075+0000) 2024-05-03T21:38:05.245 INFO:journalctl@ceph.osd.10.smithi082.stdout:May 03 21:38:04 smithi082 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-10[57676]: 2024-05-03T21:38:04.957+0000 7f790b86c700 -1 osd.10 87 heartbeat_check: no reply from 172.21.15.138:6804 osd.2 since back 2024-05-03T21:37:36.406076+0000 front 2024-05-03T21:37:36.406028+0000 (oldest deadline 2024-05-03T21:38:01.705914+0000) 2024-05-03T21:38:05.324 INFO:journalctl@ceph.osd.8.smithi138.stdout:May 03 21:38:05 smithi138 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-8[51920]: 2024-05-03T21:38:05.219+0000 7f4b828df700 -1 osd.8 87 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-05-03T21:37:29.505823+0000 front 2024-05-03T21:37:29.505840+0000 (oldest deadline 2024-05-03T21:37:50.605770+0000) 2024-05-03T21:38:05.324 INFO:journalctl@ceph.osd.8.smithi138.stdout:May 03 21:38:05 smithi138 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-8[51920]: 2024-05-03T21:38:05.219+0000 7f4b828df700 -1 osd.8 87 heartbeat_check: no reply from 172.21.15.138:6804 osd.2 since back 2024-05-03T21:37:34.706687+0000 front 2024-05-03T21:37:34.706869+0000 (oldest deadline 2024-05-03T21:37:58.206880+0000) 2024-05-03T21:38:05.324 INFO:journalctl@ceph.osd.11.smithi138.stdout:May 03 21:38:05 smithi138 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-11[58547]: 2024-05-03T21:38:05.046+0000 7fdfdcb59700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-03T21:37:21.054826+0000 front 2024-05-03T21:37:21.054670+0000 (oldest deadline 2024-05-03T21:37:45.154864+0000) 2024-05-03T21:38:05.324 INFO:journalctl@ceph.osd.11.smithi138.stdout:May 03 21:38:05 smithi138 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-11[58547]: 2024-05-03T21:38:05.046+0000 7fdfdcb59700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-05-03T21:37:29.256054+0000 front 2024-05-03T21:37:29.256037+0000 (oldest deadline 2024-05-03T21:37:53.956115+0000) 2024-05-03T21:38:05.325 INFO:journalctl@ceph.osd.11.smithi138.stdout:May 03 21:38:05 smithi138 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-11[58547]: 2024-05-03T21:38:05.046+0000 7fdfdcb59700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.138:6804 osd.2 since back 2024-05-03T21:37:33.956668+0000 front 2024-05-03T21:37:33.956702+0000 (oldest deadline 2024-05-03T21:37:59.856824+0000) 2024-05-03T21:38:05.745 INFO:journalctl@ceph.osd.7.smithi082.stdout:May 03 21:38:05 smithi082 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-7[50761]: 2024-05-03T21:38:05.315+0000 7f44b54b5700 -1 osd.7 87 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-05-03T21:37:26.771556+0000 front 2024-05-03T21:37:26.771586+0000 (oldest deadline 2024-05-03T21:37:54.372236+0000) 2024-05-03T21:38:05.745 INFO:journalctl@ceph.osd.7.smithi082.stdout:May 03 21:38:05 smithi082 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-7[50761]: 2024-05-03T21:38:05.315+0000 7f44b54b5700 -1 osd.7 87 heartbeat_check: no reply from 172.21.15.138:6804 osd.2 since back 2024-05-03T21:37:34.372911+0000 front 2024-05-03T21:37:34.372806+0000 (oldest deadline 2024-05-03T21:37:59.072727+0000) 2024-05-03T21:38:05.872 INFO:journalctl@ceph.osd.6.smithi067.stdout:May 03 21:38:05 smithi067 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-6[55180]: 2024-05-03T21:38:05.604+0000 7f389239a700 -1 osd.6 87 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-03T21:37:19.073534+0000 front 2024-05-03T21:37:19.073793+0000 (oldest deadline 2024-05-03T21:37:46.673918+0000) 2024-05-03T21:38:05.873 INFO:journalctl@ceph.osd.6.smithi067.stdout:May 03 21:38:05 smithi067 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-6[55180]: 2024-05-03T21:38:05.604+0000 7f389239a700 -1 osd.6 87 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-05-03T21:37:26.674547+0000 front 2024-05-03T21:37:26.674628+0000 (oldest deadline 2024-05-03T21:37:52.574465+0000) 2024-05-03T21:38:05.873 INFO:journalctl@ceph.osd.6.smithi067.stdout:May 03 21:38:05 smithi067 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-6[55180]: 2024-05-03T21:38:05.604+0000 7f389239a700 -1 osd.6 87 heartbeat_check: no reply from 172.21.15.138:6804 osd.2 since back 2024-05-03T21:37:35.375943+0000 front 2024-05-03T21:37:35.375805+0000 (oldest deadline 2024-05-03T21:38:01.275874+0000) 2024-05-03T21:38:05.873 INFO:journalctl@ceph.osd.9.smithi067.stdout:May 03 21:38:05 smithi067 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-9[62235]: 2024-05-03T21:38:05.696+0000 7fcff92a2700 -1 osd.9 87 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-03T21:37:20.101372+0000 front 2024-05-03T21:37:20.101400+0000 (oldest deadline 2024-05-03T21:37:44.801519+0000) 2024-05-03T21:38:05.873 INFO:journalctl@ceph.osd.9.smithi067.stdout:May 03 21:38:05 smithi067 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-9[62235]: 2024-05-03T21:38:05.696+0000 7fcff92a2700 -1 osd.9 87 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-05-03T21:37:27.702901+0000 front 2024-05-03T21:37:27.702918+0000 (oldest deadline 2024-05-03T21:37:52.402882+0000) 2024-05-03T21:38:05.873 INFO:journalctl@ceph.osd.9.smithi067.stdout:May 03 21:38:05 smithi067 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-9[62235]: 2024-05-03T21:38:05.696+0000 7fcff92a2700 -1 osd.9 87 heartbeat_check: no reply from 172.21.15.138:6804 osd.2 since back 2024-05-03T21:37:34.605024+0000 front 2024-05-03T21:37:34.604986+0000 (oldest deadline 2024-05-03T21:37:59.904623+0000) 2024-05-03T21:38:06.245 INFO:journalctl@ceph.osd.10.smithi082.stdout:May 03 21:38:05 smithi082 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-10[57676]: 2024-05-03T21:38:05.934+0000 7f790b86c700 -1 osd.10 87 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-03T21:37:18.101114+0000 front 2024-05-03T21:37:18.101101+0000 (oldest deadline 2024-05-03T21:37:42.800953+0000) 2024-05-03T21:38:06.245 INFO:journalctl@ceph.osd.10.smithi082.stdout:May 03 21:38:05 smithi082 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-10[57676]: 2024-05-03T21:38:05.934+0000 7f790b86c700 -1 osd.10 87 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-05-03T21:37:28.303836+0000 front 2024-05-03T21:37:28.303941+0000 (oldest deadline 2024-05-03T21:37:51.804075+0000) 2024-05-03T21:38:06.245 INFO:journalctl@ceph.osd.10.smithi082.stdout:May 03 21:38:05 smithi082 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-10[57676]: 2024-05-03T21:38:05.934+0000 7f790b86c700 -1 osd.10 87 heartbeat_check: no reply from 172.21.15.138:6804 osd.2 since back 2024-05-03T21:37:36.406076+0000 front 2024-05-03T21:37:36.406028+0000 (oldest deadline 2024-05-03T21:38:01.705914+0000) 2024-05-03T21:38:06.574 INFO:journalctl@ceph.osd.8.smithi138.stdout:May 03 21:38:06 smithi138 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-8[51920]: 2024-05-03T21:38:06.244+0000 7f4b828df700 -1 osd.8 87 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-05-03T21:37:29.505823+0000 front 2024-05-03T21:37:29.505840+0000 (oldest deadline 2024-05-03T21:37:50.605770+0000) 2024-05-03T21:38:06.574 INFO:journalctl@ceph.osd.8.smithi138.stdout:May 03 21:38:06 smithi138 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-8[51920]: 2024-05-03T21:38:06.244+0000 7f4b828df700 -1 osd.8 87 heartbeat_check: no reply from 172.21.15.138:6804 osd.2 since back 2024-05-03T21:37:34.706687+0000 front 2024-05-03T21:37:34.706869+0000 (oldest deadline 2024-05-03T21:37:58.206880+0000) 2024-05-03T21:38:06.575 INFO:journalctl@ceph.osd.11.smithi138.stdout:May 03 21:38:06 smithi138 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-11[58547]: 2024-05-03T21:38:06.076+0000 7fdfdcb59700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-03T21:37:21.054826+0000 front 2024-05-03T21:37:21.054670+0000 (oldest deadline 2024-05-03T21:37:45.154864+0000) 2024-05-03T21:38:06.575 INFO:journalctl@ceph.osd.11.smithi138.stdout:May 03 21:38:06 smithi138 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-11[58547]: 2024-05-03T21:38:06.076+0000 7fdfdcb59700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-05-03T21:37:29.256054+0000 front 2024-05-03T21:37:29.256037+0000 (oldest deadline 2024-05-03T21:37:53.956115+0000) 2024-05-03T21:38:06.575 INFO:journalctl@ceph.osd.11.smithi138.stdout:May 03 21:38:06 smithi138 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-11[58547]: 2024-05-03T21:38:06.076+0000 7fdfdcb59700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.138:6804 osd.2 since back 2024-05-03T21:37:33.956668+0000 front 2024-05-03T21:37:33.956702+0000 (oldest deadline 2024-05-03T21:37:59.856824+0000) 2024-05-03T21:38:06.745 INFO:journalctl@ceph.osd.7.smithi082.stdout:May 03 21:38:06 smithi082 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-7[50761]: 2024-05-03T21:38:06.269+0000 7f44b54b5700 -1 osd.7 87 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-05-03T21:37:26.771556+0000 front 2024-05-03T21:37:26.771586+0000 (oldest deadline 2024-05-03T21:37:54.372236+0000) 2024-05-03T21:38:06.745 INFO:journalctl@ceph.osd.7.smithi082.stdout:May 03 21:38:06 smithi082 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-7[50761]: 2024-05-03T21:38:06.269+0000 7f44b54b5700 -1 osd.7 87 heartbeat_check: no reply from 172.21.15.138:6804 osd.2 since back 2024-05-03T21:37:34.372911+0000 front 2024-05-03T21:37:34.372806+0000 (oldest deadline 2024-05-03T21:37:59.072727+0000) 2024-05-03T21:38:06.897 INFO:journalctl@ceph.osd.9.smithi067.stdout:May 03 21:38:06 smithi067 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-9[62235]: 2024-05-03T21:38:06.653+0000 7fcff92a2700 -1 osd.9 87 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-03T21:37:20.101372+0000 front 2024-05-03T21:37:20.101400+0000 (oldest deadline 2024-05-03T21:37:44.801519+0000) 2024-05-03T21:38:06.897 INFO:journalctl@ceph.osd.9.smithi067.stdout:May 03 21:38:06 smithi067 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-9[62235]: 2024-05-03T21:38:06.653+0000 7fcff92a2700 -1 osd.9 87 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-05-03T21:37:27.702901+0000 front 2024-05-03T21:37:27.702918+0000 (oldest deadline 2024-05-03T21:37:52.402882+0000) 2024-05-03T21:38:06.897 INFO:journalctl@ceph.osd.9.smithi067.stdout:May 03 21:38:06 smithi067 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-9[62235]: 2024-05-03T21:38:06.653+0000 7fcff92a2700 -1 osd.9 87 heartbeat_check: no reply from 172.21.15.138:6804 osd.2 since back 2024-05-03T21:37:34.605024+0000 front 2024-05-03T21:37:34.604986+0000 (oldest deadline 2024-05-03T21:37:59.904623+0000) 2024-05-03T21:38:06.897 INFO:journalctl@ceph.osd.6.smithi067.stdout:May 03 21:38:06 smithi067 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-6[55180]: 2024-05-03T21:38:06.599+0000 7f389239a700 -1 osd.6 87 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-03T21:37:19.073534+0000 front 2024-05-03T21:37:19.073793+0000 (oldest deadline 2024-05-03T21:37:46.673918+0000) 2024-05-03T21:38:06.897 INFO:journalctl@ceph.osd.6.smithi067.stdout:May 03 21:38:06 smithi067 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-6[55180]: 2024-05-03T21:38:06.599+0000 7f389239a700 -1 osd.6 87 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-05-03T21:37:26.674547+0000 front 2024-05-03T21:37:26.674628+0000 (oldest deadline 2024-05-03T21:37:52.574465+0000) 2024-05-03T21:38:06.897 INFO:journalctl@ceph.osd.6.smithi067.stdout:May 03 21:38:06 smithi067 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-6[55180]: 2024-05-03T21:38:06.599+0000 7f389239a700 -1 osd.6 87 heartbeat_check: no reply from 172.21.15.138:6804 osd.2 since back 2024-05-03T21:37:35.375943+0000 front 2024-05-03T21:37:35.375805+0000 (oldest deadline 2024-05-03T21:38:01.275874+0000) 2024-05-03T21:38:06.898 INFO:journalctl@ceph.osd.6.smithi067.stdout:May 03 21:38:06 smithi067 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-6[55180]: 2024-05-03T21:38:06.599+0000 7f389239a700 -1 osd.6 87 heartbeat_check: no reply from 172.21.15.67:6814 osd.3 since back 2024-05-03T21:37:42.376768+0000 front 2024-05-03T21:37:42.376801+0000 (oldest deadline 2024-05-03T21:38:05.876948+0000) 2024-05-03T21:38:07.245 INFO:journalctl@ceph.osd.10.smithi082.stdout:May 03 21:38:06 smithi082 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-10[57676]: 2024-05-03T21:38:06.964+0000 7f790b86c700 -1 osd.10 87 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-03T21:37:18.101114+0000 front 2024-05-03T21:37:18.101101+0000 (oldest deadline 2024-05-03T21:37:42.800953+0000) 2024-05-03T21:38:07.245 INFO:journalctl@ceph.osd.10.smithi082.stdout:May 03 21:38:06 smithi082 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-10[57676]: 2024-05-03T21:38:06.964+0000 7f790b86c700 -1 osd.10 87 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-05-03T21:37:28.303836+0000 front 2024-05-03T21:37:28.303941+0000 (oldest deadline 2024-05-03T21:37:51.804075+0000) 2024-05-03T21:38:07.245 INFO:journalctl@ceph.osd.10.smithi082.stdout:May 03 21:38:06 smithi082 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-10[57676]: 2024-05-03T21:38:06.964+0000 7f790b86c700 -1 osd.10 87 heartbeat_check: no reply from 172.21.15.138:6804 osd.2 since back 2024-05-03T21:37:36.406076+0000 front 2024-05-03T21:37:36.406028+0000 (oldest deadline 2024-05-03T21:38:01.705914+0000) 2024-05-03T21:38:07.324 INFO:journalctl@ceph.osd.8.smithi138.stdout:May 03 21:38:07 smithi138 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-8[51920]: 2024-05-03T21:38:07.259+0000 7f4b828df700 -1 osd.8 87 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-05-03T21:37:29.505823+0000 front 2024-05-03T21:37:29.505840+0000 (oldest deadline 2024-05-03T21:37:50.605770+0000) 2024-05-03T21:38:07.324 INFO:journalctl@ceph.osd.8.smithi138.stdout:May 03 21:38:07 smithi138 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-8[51920]: 2024-05-03T21:38:07.259+0000 7f4b828df700 -1 osd.8 87 heartbeat_check: no reply from 172.21.15.138:6804 osd.2 since back 2024-05-03T21:37:34.706687+0000 front 2024-05-03T21:37:34.706869+0000 (oldest deadline 2024-05-03T21:37:58.206880+0000) 2024-05-03T21:38:07.325 INFO:journalctl@ceph.osd.11.smithi138.stdout:May 03 21:38:07 smithi138 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-11[58547]: 2024-05-03T21:38:07.030+0000 7fdfdcb59700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-03T21:37:21.054826+0000 front 2024-05-03T21:37:21.054670+0000 (oldest deadline 2024-05-03T21:37:45.154864+0000) 2024-05-03T21:38:07.325 INFO:journalctl@ceph.osd.11.smithi138.stdout:May 03 21:38:07 smithi138 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-11[58547]: 2024-05-03T21:38:07.030+0000 7fdfdcb59700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-05-03T21:37:29.256054+0000 front 2024-05-03T21:37:29.256037+0000 (oldest deadline 2024-05-03T21:37:53.956115+0000) 2024-05-03T21:38:07.325 INFO:journalctl@ceph.osd.11.smithi138.stdout:May 03 21:38:07 smithi138 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-11[58547]: 2024-05-03T21:38:07.030+0000 7fdfdcb59700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.138:6804 osd.2 since back 2024-05-03T21:37:33.956668+0000 front 2024-05-03T21:37:33.956702+0000 (oldest deadline 2024-05-03T21:37:59.856824+0000) 2024-05-03T21:38:07.745 INFO:journalctl@ceph.osd.7.smithi082.stdout:May 03 21:38:07 smithi082 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-7[50761]: 2024-05-03T21:38:07.276+0000 7f44b54b5700 -1 osd.7 87 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-05-03T21:37:26.771556+0000 front 2024-05-03T21:37:26.771586+0000 (oldest deadline 2024-05-03T21:37:54.372236+0000) 2024-05-03T21:38:07.745 INFO:journalctl@ceph.osd.7.smithi082.stdout:May 03 21:38:07 smithi082 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-7[50761]: 2024-05-03T21:38:07.276+0000 7f44b54b5700 -1 osd.7 87 heartbeat_check: no reply from 172.21.15.138:6804 osd.2 since back 2024-05-03T21:37:34.372911+0000 front 2024-05-03T21:37:34.372806+0000 (oldest deadline 2024-05-03T21:37:59.072727+0000) 2024-05-03T21:38:07.971 INFO:journalctl@ceph.osd.6.smithi067.stdout:May 03 21:38:07 smithi067 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-6[55180]: 2024-05-03T21:38:07.566+0000 7f389239a700 -1 osd.6 87 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-03T21:37:19.073534+0000 front 2024-05-03T21:37:19.073793+0000 (oldest deadline 2024-05-03T21:37:46.673918+0000) 2024-05-03T21:38:07.971 INFO:journalctl@ceph.osd.6.smithi067.stdout:May 03 21:38:07 smithi067 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-6[55180]: 2024-05-03T21:38:07.566+0000 7f389239a700 -1 osd.6 87 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-05-03T21:37:26.674547+0000 front 2024-05-03T21:37:26.674628+0000 (oldest deadline 2024-05-03T21:37:52.574465+0000) 2024-05-03T21:38:07.971 INFO:journalctl@ceph.osd.6.smithi067.stdout:May 03 21:38:07 smithi067 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-6[55180]: 2024-05-03T21:38:07.566+0000 7f389239a700 -1 osd.6 87 heartbeat_check: no reply from 172.21.15.138:6804 osd.2 since back 2024-05-03T21:37:35.375943+0000 front 2024-05-03T21:37:35.375805+0000 (oldest deadline 2024-05-03T21:38:01.275874+0000) 2024-05-03T21:38:07.971 INFO:journalctl@ceph.osd.6.smithi067.stdout:May 03 21:38:07 smithi067 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-6[55180]: 2024-05-03T21:38:07.566+0000 7f389239a700 -1 osd.6 87 heartbeat_check: no reply from 172.21.15.67:6814 osd.3 since back 2024-05-03T21:37:42.376768+0000 front 2024-05-03T21:37:42.376801+0000 (oldest deadline 2024-05-03T21:38:05.876948+0000) 2024-05-03T21:38:07.972 INFO:journalctl@ceph.osd.9.smithi067.stdout:May 03 21:38:07 smithi067 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-9[62235]: 2024-05-03T21:38:07.677+0000 7fcff92a2700 -1 osd.9 87 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-03T21:37:20.101372+0000 front 2024-05-03T21:37:20.101400+0000 (oldest deadline 2024-05-03T21:37:44.801519+0000) 2024-05-03T21:38:07.972 INFO:journalctl@ceph.osd.9.smithi067.stdout:May 03 21:38:07 smithi067 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-9[62235]: 2024-05-03T21:38:07.677+0000 7fcff92a2700 -1 osd.9 87 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-05-03T21:37:27.702901+0000 front 2024-05-03T21:37:27.702918+0000 (oldest deadline 2024-05-03T21:37:52.402882+0000) 2024-05-03T21:38:07.972 INFO:journalctl@ceph.osd.9.smithi067.stdout:May 03 21:38:07 smithi067 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-9[62235]: 2024-05-03T21:38:07.677+0000 7fcff92a2700 -1 osd.9 87 heartbeat_check: no reply from 172.21.15.138:6804 osd.2 since back 2024-05-03T21:37:34.605024+0000 front 2024-05-03T21:37:34.604986+0000 (oldest deadline 2024-05-03T21:37:59.904623+0000) 2024-05-03T21:38:08.245 INFO:journalctl@ceph.osd.10.smithi082.stdout:May 03 21:38:07 smithi082 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-10[57676]: 2024-05-03T21:38:07.935+0000 7f790b86c700 -1 osd.10 87 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-03T21:37:18.101114+0000 front 2024-05-03T21:37:18.101101+0000 (oldest deadline 2024-05-03T21:37:42.800953+0000) 2024-05-03T21:38:08.245 INFO:journalctl@ceph.osd.10.smithi082.stdout:May 03 21:38:07 smithi082 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-10[57676]: 2024-05-03T21:38:07.935+0000 7f790b86c700 -1 osd.10 87 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-05-03T21:37:28.303836+0000 front 2024-05-03T21:37:28.303941+0000 (oldest deadline 2024-05-03T21:37:51.804075+0000) 2024-05-03T21:38:08.245 INFO:journalctl@ceph.osd.10.smithi082.stdout:May 03 21:38:07 smithi082 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-10[57676]: 2024-05-03T21:38:07.935+0000 7f790b86c700 -1 osd.10 87 heartbeat_check: no reply from 172.21.15.138:6804 osd.2 since back 2024-05-03T21:37:36.406076+0000 front 2024-05-03T21:37:36.406028+0000 (oldest deadline 2024-05-03T21:38:01.705914+0000) 2024-05-03T21:38:08.297 INFO:journalctl@ceph.osd.11.smithi138.stdout:May 03 21:38:07 smithi138 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-11[58547]: 2024-05-03T21:38:07.990+0000 7fdfdcb59700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-03T21:37:21.054826+0000 front 2024-05-03T21:37:21.054670+0000 (oldest deadline 2024-05-03T21:37:45.154864+0000) 2024-05-03T21:38:08.297 INFO:journalctl@ceph.osd.11.smithi138.stdout:May 03 21:38:07 smithi138 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-11[58547]: 2024-05-03T21:38:07.990+0000 7fdfdcb59700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-05-03T21:37:29.256054+0000 front 2024-05-03T21:37:29.256037+0000 (oldest deadline 2024-05-03T21:37:53.956115+0000) 2024-05-03T21:38:08.297 INFO:journalctl@ceph.osd.11.smithi138.stdout:May 03 21:38:07 smithi138 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-11[58547]: 2024-05-03T21:38:07.990+0000 7fdfdcb59700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.138:6804 osd.2 since back 2024-05-03T21:37:33.956668+0000 front 2024-05-03T21:37:33.956702+0000 (oldest deadline 2024-05-03T21:37:59.856824+0000) 2024-05-03T21:38:08.574 INFO:journalctl@ceph.osd.8.smithi138.stdout:May 03 21:38:08 smithi138 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-8[51920]: 2024-05-03T21:38:08.295+0000 7f4b828df700 -1 osd.8 87 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-05-03T21:37:29.505823+0000 front 2024-05-03T21:37:29.505840+0000 (oldest deadline 2024-05-03T21:37:50.605770+0000) 2024-05-03T21:38:08.574 INFO:journalctl@ceph.osd.8.smithi138.stdout:May 03 21:38:08 smithi138 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-8[51920]: 2024-05-03T21:38:08.295+0000 7f4b828df700 -1 osd.8 87 heartbeat_check: no reply from 172.21.15.138:6804 osd.2 since back 2024-05-03T21:37:34.706687+0000 front 2024-05-03T21:37:34.706869+0000 (oldest deadline 2024-05-03T21:37:58.206880+0000) 2024-05-03T21:38:08.745 INFO:journalctl@ceph.osd.7.smithi082.stdout:May 03 21:38:08 smithi082 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-7[50761]: 2024-05-03T21:38:08.245+0000 7f44b54b5700 -1 osd.7 87 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-05-03T21:37:26.771556+0000 front 2024-05-03T21:37:26.771586+0000 (oldest deadline 2024-05-03T21:37:54.372236+0000) 2024-05-03T21:38:08.745 INFO:journalctl@ceph.osd.7.smithi082.stdout:May 03 21:38:08 smithi082 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-7[50761]: 2024-05-03T21:38:08.245+0000 7f44b54b5700 -1 osd.7 87 heartbeat_check: no reply from 172.21.15.138:6804 osd.2 since back 2024-05-03T21:37:34.372911+0000 front 2024-05-03T21:37:34.372806+0000 (oldest deadline 2024-05-03T21:37:59.072727+0000) 2024-05-03T21:38:08.970 INFO:journalctl@ceph.osd.6.smithi067.stdout:May 03 21:38:08 smithi067 podman[88004]: 2024-05-03 21:38:08.864956373 +0000 UTC m=+5.496707273 container died d96a92031749bc1c80baecfee1e9678e5cb20b7e33cf55e65db4e4d3f42427b6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:68e15ae152ef5c805c733e76da8fc1d3dd4ee669d5ba025f457e05725baa24dd, name=ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-6, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.build-date=20240430, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_CLEAN=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, RELEASE=wip-rishabh-testing-20240501.193033-9a047fe, maintainer=Guillaume Abrioux , GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b) 2024-05-03T21:38:08.971 INFO:journalctl@ceph.osd.9.smithi067.stdout:May 03 21:38:08 smithi067 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-9[62235]: 2024-05-03T21:38:08.674+0000 7fcff92a2700 -1 osd.9 87 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-03T21:37:20.101372+0000 front 2024-05-03T21:37:20.101400+0000 (oldest deadline 2024-05-03T21:37:44.801519+0000) 2024-05-03T21:38:08.971 INFO:journalctl@ceph.osd.9.smithi067.stdout:May 03 21:38:08 smithi067 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-9[62235]: 2024-05-03T21:38:08.674+0000 7fcff92a2700 -1 osd.9 87 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-05-03T21:37:27.702901+0000 front 2024-05-03T21:37:27.702918+0000 (oldest deadline 2024-05-03T21:37:52.402882+0000) 2024-05-03T21:38:08.971 INFO:journalctl@ceph.osd.9.smithi067.stdout:May 03 21:38:08 smithi067 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-9[62235]: 2024-05-03T21:38:08.674+0000 7fcff92a2700 -1 osd.9 87 heartbeat_check: no reply from 172.21.15.138:6804 osd.2 since back 2024-05-03T21:37:34.605024+0000 front 2024-05-03T21:37:34.604986+0000 (oldest deadline 2024-05-03T21:37:59.904623+0000) 2024-05-03T21:38:08.971 INFO:journalctl@ceph.osd.9.smithi067.stdout:May 03 21:38:08 smithi067 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-9[62235]: 2024-05-03T21:38:08.674+0000 7fcff92a2700 -1 osd.9 87 heartbeat_check: no reply from 172.21.15.67:6814 osd.3 since back 2024-05-03T21:37:42.805355+0000 front 2024-05-03T21:37:42.805441+0000 (oldest deadline 2024-05-03T21:38:08.105521+0000) 2024-05-03T21:38:09.218 INFO:journalctl@ceph.osd.10.smithi082.stdout:May 03 21:38:08 smithi082 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-10[57676]: 2024-05-03T21:38:08.918+0000 7f790b86c700 -1 osd.10 87 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-03T21:37:18.101114+0000 front 2024-05-03T21:37:18.101101+0000 (oldest deadline 2024-05-03T21:37:42.800953+0000) 2024-05-03T21:38:09.218 INFO:journalctl@ceph.osd.10.smithi082.stdout:May 03 21:38:08 smithi082 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-10[57676]: 2024-05-03T21:38:08.918+0000 7f790b86c700 -1 osd.10 87 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-05-03T21:37:28.303836+0000 front 2024-05-03T21:37:28.303941+0000 (oldest deadline 2024-05-03T21:37:51.804075+0000) 2024-05-03T21:38:09.218 INFO:journalctl@ceph.osd.10.smithi082.stdout:May 03 21:38:08 smithi082 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-10[57676]: 2024-05-03T21:38:08.918+0000 7f790b86c700 -1 osd.10 87 heartbeat_check: no reply from 172.21.15.138:6804 osd.2 since back 2024-05-03T21:37:36.406076+0000 front 2024-05-03T21:37:36.406028+0000 (oldest deadline 2024-05-03T21:38:01.705914+0000) 2024-05-03T21:38:09.218 INFO:journalctl@ceph.osd.10.smithi082.stdout:May 03 21:38:08 smithi082 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-10[57676]: 2024-05-03T21:38:08.918+0000 7f790b86c700 -1 osd.10 87 heartbeat_check: no reply from 172.21.15.67:6814 osd.3 since back 2024-05-03T21:37:41.706645+0000 front 2024-05-03T21:37:41.706615+0000 (oldest deadline 2024-05-03T21:38:08.107018+0000) 2024-05-03T21:38:09.324 INFO:journalctl@ceph.osd.11.smithi138.stdout:May 03 21:38:08 smithi138 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-11[58547]: 2024-05-03T21:38:08.973+0000 7fdfdcb59700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-03T21:37:21.054826+0000 front 2024-05-03T21:37:21.054670+0000 (oldest deadline 2024-05-03T21:37:45.154864+0000) 2024-05-03T21:38:09.405 INFO:journalctl@ceph.osd.11.smithi138.stdout:May 03 21:38:08 smithi138 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-11[58547]: 2024-05-03T21:38:08.973+0000 7fdfdcb59700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-05-03T21:37:29.256054+0000 front 2024-05-03T21:37:29.256037+0000 (oldest deadline 2024-05-03T21:37:53.956115+0000) 2024-05-03T21:38:09.405 INFO:journalctl@ceph.osd.11.smithi138.stdout:May 03 21:38:08 smithi138 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-11[58547]: 2024-05-03T21:38:08.973+0000 7fdfdcb59700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.138:6804 osd.2 since back 2024-05-03T21:37:33.956668+0000 front 2024-05-03T21:37:33.956702+0000 (oldest deadline 2024-05-03T21:37:59.856824+0000) 2024-05-03T21:38:09.471 INFO:journalctl@ceph.osd.6.smithi067.stdout:May 03 21:38:09 smithi067 podman[88004]: 2024-05-03 21:38:09.191445833 +0000 UTC m=+5.823196736 container cleanup d96a92031749bc1c80baecfee1e9678e5cb20b7e33cf55e65db4e4d3f42427b6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:68e15ae152ef5c805c733e76da8fc1d3dd4ee669d5ba025f457e05725baa24dd, name=ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-6, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, RELEASE=wip-rishabh-testing-20240501.193033-9a047fe, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, ceph=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240430) 2024-05-03T21:38:09.471 INFO:journalctl@ceph.osd.6.smithi067.stdout:May 03 21:38:09 smithi067 bash[88004]: ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-6 2024-05-03T21:38:09.472 INFO:journalctl@ceph.osd.6.smithi067.stdout:May 03 21:38:09 smithi067 podman[88033]: 2024-05-03 21:38:09.367849852 +0000 UTC m=+0.498662303 container remove d96a92031749bc1c80baecfee1e9678e5cb20b7e33cf55e65db4e4d3f42427b6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:68e15ae152ef5c805c733e76da8fc1d3dd4ee669d5ba025f457e05725baa24dd, name=ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-6, org.label-schema.build-date=20240430, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-rishabh-testing-20240501.193033-9a047fe, GIT_CLEAN=True, ceph=True, io.buildah.version=1.33.5) 2024-05-03T21:38:09.494 INFO:journalctl@ceph.osd.7.smithi082.stdout:May 03 21:38:09 smithi082 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-7[50761]: 2024-05-03T21:38:09.216+0000 7f44b54b5700 -1 osd.7 87 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-05-03T21:37:26.771556+0000 front 2024-05-03T21:37:26.771586+0000 (oldest deadline 2024-05-03T21:37:54.372236+0000) 2024-05-03T21:38:09.495 INFO:journalctl@ceph.osd.7.smithi082.stdout:May 03 21:38:09 smithi082 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-7[50761]: 2024-05-03T21:38:09.216+0000 7f44b54b5700 -1 osd.7 87 heartbeat_check: no reply from 172.21.15.138:6804 osd.2 since back 2024-05-03T21:37:34.372911+0000 front 2024-05-03T21:37:34.372806+0000 (oldest deadline 2024-05-03T21:37:59.072727+0000) 2024-05-03T21:38:09.824 INFO:journalctl@ceph.osd.8.smithi138.stdout:May 03 21:38:09 smithi138 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-8[51920]: 2024-05-03T21:38:09.332+0000 7f4b828df700 -1 osd.8 87 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-05-03T21:37:29.505823+0000 front 2024-05-03T21:37:29.505840+0000 (oldest deadline 2024-05-03T21:37:50.605770+0000) 2024-05-03T21:38:09.824 INFO:journalctl@ceph.osd.8.smithi138.stdout:May 03 21:38:09 smithi138 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-8[51920]: 2024-05-03T21:38:09.333+0000 7f4b828df700 -1 osd.8 87 heartbeat_check: no reply from 172.21.15.138:6804 osd.2 since back 2024-05-03T21:37:34.706687+0000 front 2024-05-03T21:37:34.706869+0000 (oldest deadline 2024-05-03T21:37:58.206880+0000) 2024-05-03T21:38:09.824 INFO:journalctl@ceph.osd.8.smithi138.stdout:May 03 21:38:09 smithi138 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-8[51920]: 2024-05-03T21:38:09.333+0000 7f4b828df700 -1 osd.8 87 heartbeat_check: no reply from 172.21.15.67:6814 osd.3 since back 2024-05-03T21:37:44.508584+0000 front 2024-05-03T21:37:44.508817+0000 (oldest deadline 2024-05-03T21:38:09.208689+0000) 2024-05-03T21:38:09.924 INFO:journalctl@ceph.osd.6.smithi067.stdout:May 03 21:38:09 smithi067 podman[88120]: 2024-05-03 21:38:09.725382937 +0000 UTC m=+0.018821086 image pull ee4d846202496ba6096593e8e8becc0700a5eb7ebd95536031187f5f337d4012 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:68e15ae152ef5c805c733e76da8fc1d3dd4ee669d5ba025f457e05725baa24dd 2024-05-03T21:38:09.924 INFO:journalctl@ceph.osd.6.smithi067.stdout:May 03 21:38:09 smithi067 podman[88120]: 2024-05-03 21:38:09.827075072 +0000 UTC m=+0.120513249 container create fd639a0e603d19a59d375e69709f6a64c7e613f0c820c839d74d8a43cd4150fd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:68e15ae152ef5c805c733e76da8fc1d3dd4ee669d5ba025f457e05725baa24dd, name=ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-6-deactivate, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.build-date=20240430, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, RELEASE=wip-rishabh-testing-20240501.193033-9a047fe, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS) 2024-05-03T21:38:09.925 INFO:journalctl@ceph.osd.9.smithi067.stdout:May 03 21:38:09 smithi067 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-9[62235]: 2024-05-03T21:38:09.628+0000 7fcff92a2700 -1 osd.9 87 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-03T21:37:20.101372+0000 front 2024-05-03T21:37:20.101400+0000 (oldest deadline 2024-05-03T21:37:44.801519+0000) 2024-05-03T21:38:09.925 INFO:journalctl@ceph.osd.9.smithi067.stdout:May 03 21:38:09 smithi067 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-9[62235]: 2024-05-03T21:38:09.628+0000 7fcff92a2700 -1 osd.9 87 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-05-03T21:37:27.702901+0000 front 2024-05-03T21:37:27.702918+0000 (oldest deadline 2024-05-03T21:37:52.402882+0000) 2024-05-03T21:38:09.925 INFO:journalctl@ceph.osd.9.smithi067.stdout:May 03 21:38:09 smithi067 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-9[62235]: 2024-05-03T21:38:09.628+0000 7fcff92a2700 -1 osd.9 87 heartbeat_check: no reply from 172.21.15.138:6804 osd.2 since back 2024-05-03T21:37:34.605024+0000 front 2024-05-03T21:37:34.604986+0000 (oldest deadline 2024-05-03T21:37:59.904623+0000) 2024-05-03T21:38:09.925 INFO:journalctl@ceph.osd.9.smithi067.stdout:May 03 21:38:09 smithi067 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-9[62235]: 2024-05-03T21:38:09.628+0000 7fcff92a2700 -1 osd.9 87 heartbeat_check: no reply from 172.21.15.67:6814 osd.3 since back 2024-05-03T21:37:42.805355+0000 front 2024-05-03T21:37:42.805441+0000 (oldest deadline 2024-05-03T21:38:08.105521+0000) 2024-05-03T21:38:10.221 INFO:journalctl@ceph.osd.6.smithi067.stdout:May 03 21:38:10 smithi067 podman[88120]: 2024-05-03 21:38:10.06289419 +0000 UTC m=+0.356332339 container init fd639a0e603d19a59d375e69709f6a64c7e613f0c820c839d74d8a43cd4150fd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:68e15ae152ef5c805c733e76da8fc1d3dd4ee669d5ba025f457e05725baa24dd, name=ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-6-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240430, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, ceph=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-rishabh-testing-20240501.193033-9a047fe) 2024-05-03T21:38:10.221 INFO:journalctl@ceph.osd.6.smithi067.stdout:May 03 21:38:10 smithi067 podman[88120]: 2024-05-03 21:38:10.066472165 +0000 UTC m=+0.359910301 container start fd639a0e603d19a59d375e69709f6a64c7e613f0c820c839d74d8a43cd4150fd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:68e15ae152ef5c805c733e76da8fc1d3dd4ee669d5ba025f457e05725baa24dd, name=ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-6-deactivate, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, RELEASE=wip-rishabh-testing-20240501.193033-9a047fe, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, ceph=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.build-date=20240430) 2024-05-03T21:38:10.221 INFO:journalctl@ceph.osd.6.smithi067.stdout:May 03 21:38:10 smithi067 podman[88120]: 2024-05-03 21:38:10.162772716 +0000 UTC m=+0.456210862 container attach fd639a0e603d19a59d375e69709f6a64c7e613f0c820c839d74d8a43cd4150fd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:68e15ae152ef5c805c733e76da8fc1d3dd4ee669d5ba025f457e05725baa24dd, name=ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-6-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, ceph=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, RELEASE=wip-rishabh-testing-20240501.193033-9a047fe, org.label-schema.build-date=20240430, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS) 2024-05-03T21:38:10.223 INFO:journalctl@ceph.osd.10.smithi082.stdout:May 03 21:38:09 smithi082 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-10[57676]: 2024-05-03T21:38:09.882+0000 7f790b86c700 -1 osd.10 87 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-03T21:37:18.101114+0000 front 2024-05-03T21:37:18.101101+0000 (oldest deadline 2024-05-03T21:37:42.800953+0000) 2024-05-03T21:38:10.223 INFO:journalctl@ceph.osd.10.smithi082.stdout:May 03 21:38:09 smithi082 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-10[57676]: 2024-05-03T21:38:09.882+0000 7f790b86c700 -1 osd.10 87 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-05-03T21:37:28.303836+0000 front 2024-05-03T21:37:28.303941+0000 (oldest deadline 2024-05-03T21:37:51.804075+0000) 2024-05-03T21:38:10.223 INFO:journalctl@ceph.osd.10.smithi082.stdout:May 03 21:38:09 smithi082 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-10[57676]: 2024-05-03T21:38:09.882+0000 7f790b86c700 -1 osd.10 87 heartbeat_check: no reply from 172.21.15.138:6804 osd.2 since back 2024-05-03T21:37:36.406076+0000 front 2024-05-03T21:37:36.406028+0000 (oldest deadline 2024-05-03T21:38:01.705914+0000) 2024-05-03T21:38:10.223 INFO:journalctl@ceph.osd.10.smithi082.stdout:May 03 21:38:09 smithi082 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-10[57676]: 2024-05-03T21:38:09.882+0000 7f790b86c700 -1 osd.10 87 heartbeat_check: no reply from 172.21.15.67:6814 osd.3 since back 2024-05-03T21:37:41.706645+0000 front 2024-05-03T21:37:41.706615+0000 (oldest deadline 2024-05-03T21:38:08.107018+0000) 2024-05-03T21:38:10.289 INFO:journalctl@ceph.osd.11.smithi138.stdout:May 03 21:38:10 smithi138 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-11[58547]: 2024-05-03T21:38:10.003+0000 7fdfdcb59700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-03T21:37:21.054826+0000 front 2024-05-03T21:37:21.054670+0000 (oldest deadline 2024-05-03T21:37:45.154864+0000) 2024-05-03T21:38:10.290 INFO:journalctl@ceph.osd.11.smithi138.stdout:May 03 21:38:10 smithi138 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-11[58547]: 2024-05-03T21:38:10.003+0000 7fdfdcb59700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-05-03T21:37:29.256054+0000 front 2024-05-03T21:37:29.256037+0000 (oldest deadline 2024-05-03T21:37:53.956115+0000) 2024-05-03T21:38:10.290 INFO:journalctl@ceph.osd.11.smithi138.stdout:May 03 21:38:10 smithi138 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-11[58547]: 2024-05-03T21:38:10.003+0000 7fdfdcb59700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.138:6804 osd.2 since back 2024-05-03T21:37:33.956668+0000 front 2024-05-03T21:37:33.956702+0000 (oldest deadline 2024-05-03T21:37:59.856824+0000) 2024-05-03T21:38:10.290 INFO:journalctl@ceph.osd.11.smithi138.stdout:May 03 21:38:10 smithi138 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-11[58547]: 2024-05-03T21:38:10.003+0000 7fdfdcb59700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.67:6814 osd.3 since back 2024-05-03T21:37:43.759130+0000 front 2024-05-03T21:37:43.759044+0000 (oldest deadline 2024-05-03T21:38:09.058895+0000) 2024-05-03T21:38:10.495 INFO:journalctl@ceph.osd.7.smithi082.stdout:May 03 21:38:10 smithi082 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-7[50761]: 2024-05-03T21:38:10.221+0000 7f44b54b5700 -1 osd.7 87 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-05-03T21:37:26.771556+0000 front 2024-05-03T21:37:26.771586+0000 (oldest deadline 2024-05-03T21:37:54.372236+0000) 2024-05-03T21:38:10.495 INFO:journalctl@ceph.osd.7.smithi082.stdout:May 03 21:38:10 smithi082 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-7[50761]: 2024-05-03T21:38:10.221+0000 7f44b54b5700 -1 osd.7 87 heartbeat_check: no reply from 172.21.15.138:6804 osd.2 since back 2024-05-03T21:37:34.372911+0000 front 2024-05-03T21:37:34.372806+0000 (oldest deadline 2024-05-03T21:37:59.072727+0000) 2024-05-03T21:38:10.574 INFO:journalctl@ceph.osd.8.smithi138.stdout:May 03 21:38:10 smithi138 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-8[51920]: 2024-05-03T21:38:10.288+0000 7f4b828df700 -1 osd.8 87 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-05-03T21:37:29.505823+0000 front 2024-05-03T21:37:29.505840+0000 (oldest deadline 2024-05-03T21:37:50.605770+0000) 2024-05-03T21:38:10.574 INFO:journalctl@ceph.osd.8.smithi138.stdout:May 03 21:38:10 smithi138 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-8[51920]: 2024-05-03T21:38:10.288+0000 7f4b828df700 -1 osd.8 87 heartbeat_check: no reply from 172.21.15.138:6804 osd.2 since back 2024-05-03T21:37:34.706687+0000 front 2024-05-03T21:37:34.706869+0000 (oldest deadline 2024-05-03T21:37:58.206880+0000) 2024-05-03T21:38:10.574 INFO:journalctl@ceph.osd.8.smithi138.stdout:May 03 21:38:10 smithi138 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-8[51920]: 2024-05-03T21:38:10.288+0000 7f4b828df700 -1 osd.8 87 heartbeat_check: no reply from 172.21.15.67:6814 osd.3 since back 2024-05-03T21:37:44.508584+0000 front 2024-05-03T21:37:44.508817+0000 (oldest deadline 2024-05-03T21:38:09.208689+0000) 2024-05-03T21:38:10.624 INFO:journalctl@ceph.osd.6.smithi067.stdout:May 03 21:38:10 smithi067 podman[88120]: 2024-05-03 21:38:10.375535923 +0000 UTC m=+0.668974069 container died fd639a0e603d19a59d375e69709f6a64c7e613f0c820c839d74d8a43cd4150fd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:68e15ae152ef5c805c733e76da8fc1d3dd4ee669d5ba025f457e05725baa24dd, name=ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-6-deactivate, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, RELEASE=wip-rishabh-testing-20240501.193033-9a047fe, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240430) 2024-05-03T21:38:10.950 INFO:journalctl@ceph.osd.9.smithi067.stdout:May 03 21:38:10 smithi067 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-9[62235]: 2024-05-03T21:38:10.622+0000 7fcff92a2700 -1 osd.9 87 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-03T21:37:20.101372+0000 front 2024-05-03T21:37:20.101400+0000 (oldest deadline 2024-05-03T21:37:44.801519+0000) 2024-05-03T21:38:10.951 INFO:journalctl@ceph.osd.9.smithi067.stdout:May 03 21:38:10 smithi067 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-9[62235]: 2024-05-03T21:38:10.622+0000 7fcff92a2700 -1 osd.9 87 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-05-03T21:37:27.702901+0000 front 2024-05-03T21:37:27.702918+0000 (oldest deadline 2024-05-03T21:37:52.402882+0000) 2024-05-03T21:38:10.951 INFO:journalctl@ceph.osd.9.smithi067.stdout:May 03 21:38:10 smithi067 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-9[62235]: 2024-05-03T21:38:10.622+0000 7fcff92a2700 -1 osd.9 87 heartbeat_check: no reply from 172.21.15.138:6804 osd.2 since back 2024-05-03T21:37:34.605024+0000 front 2024-05-03T21:37:34.604986+0000 (oldest deadline 2024-05-03T21:37:59.904623+0000) 2024-05-03T21:38:10.951 INFO:journalctl@ceph.osd.9.smithi067.stdout:May 03 21:38:10 smithi067 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-9[62235]: 2024-05-03T21:38:10.622+0000 7fcff92a2700 -1 osd.9 87 heartbeat_check: no reply from 172.21.15.67:6814 osd.3 since back 2024-05-03T21:37:42.805355+0000 front 2024-05-03T21:37:42.805441+0000 (oldest deadline 2024-05-03T21:38:08.105521+0000) 2024-05-03T21:38:10.966 DEBUG:teuthology.orchestra.run.smithi067:> sudo pkill -f 'journalctl -f -n 0 -u ceph-713bc57a-0991-11ef-bc97-c7b262605968@osd.6.service' 2024-05-03T21:38:11.007 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-03T21:38:11.007 INFO:tasks.cephadm.osd.6:Stopped osd.6 2024-05-03T21:38:11.008 INFO:tasks.cephadm.osd.7:Stopping osd.7... 2024-05-03T21:38:11.008 DEBUG:teuthology.orchestra.run.smithi082:> sudo systemctl stop ceph-713bc57a-0991-11ef-bc97-c7b262605968@osd.7 2024-05-03T21:38:11.183 INFO:journalctl@ceph.osd.7.smithi082.stdout:May 03 21:38:11 smithi082 systemd[1]: Stopping Ceph osd.7 for 713bc57a-0991-11ef-bc97-c7b262605968... 2024-05-03T21:38:11.184 INFO:journalctl@ceph.osd.10.smithi082.stdout:May 03 21:38:10 smithi082 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-10[57676]: 2024-05-03T21:38:10.904+0000 7f790b86c700 -1 osd.10 87 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-03T21:37:18.101114+0000 front 2024-05-03T21:37:18.101101+0000 (oldest deadline 2024-05-03T21:37:42.800953+0000) 2024-05-03T21:38:11.184 INFO:journalctl@ceph.osd.10.smithi082.stdout:May 03 21:38:10 smithi082 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-10[57676]: 2024-05-03T21:38:10.904+0000 7f790b86c700 -1 osd.10 87 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-05-03T21:37:28.303836+0000 front 2024-05-03T21:37:28.303941+0000 (oldest deadline 2024-05-03T21:37:51.804075+0000) 2024-05-03T21:38:11.184 INFO:journalctl@ceph.osd.10.smithi082.stdout:May 03 21:38:10 smithi082 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-10[57676]: 2024-05-03T21:38:10.904+0000 7f790b86c700 -1 osd.10 87 heartbeat_check: no reply from 172.21.15.138:6804 osd.2 since back 2024-05-03T21:37:36.406076+0000 front 2024-05-03T21:37:36.406028+0000 (oldest deadline 2024-05-03T21:38:01.705914+0000) 2024-05-03T21:38:11.184 INFO:journalctl@ceph.osd.10.smithi082.stdout:May 03 21:38:10 smithi082 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-10[57676]: 2024-05-03T21:38:10.904+0000 7f790b86c700 -1 osd.10 87 heartbeat_check: no reply from 172.21.15.67:6814 osd.3 since back 2024-05-03T21:37:41.706645+0000 front 2024-05-03T21:37:41.706615+0000 (oldest deadline 2024-05-03T21:38:08.107018+0000) 2024-05-03T21:38:11.324 INFO:journalctl@ceph.osd.11.smithi138.stdout:May 03 21:38:11 smithi138 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-11[58547]: 2024-05-03T21:38:11.043+0000 7fdfdcb59700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-03T21:37:21.054826+0000 front 2024-05-03T21:37:21.054670+0000 (oldest deadline 2024-05-03T21:37:45.154864+0000) 2024-05-03T21:38:11.324 INFO:journalctl@ceph.osd.11.smithi138.stdout:May 03 21:38:11 smithi138 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-11[58547]: 2024-05-03T21:38:11.043+0000 7fdfdcb59700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-05-03T21:37:29.256054+0000 front 2024-05-03T21:37:29.256037+0000 (oldest deadline 2024-05-03T21:37:53.956115+0000) 2024-05-03T21:38:11.324 INFO:journalctl@ceph.osd.11.smithi138.stdout:May 03 21:38:11 smithi138 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-11[58547]: 2024-05-03T21:38:11.043+0000 7fdfdcb59700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.138:6804 osd.2 since back 2024-05-03T21:37:33.956668+0000 front 2024-05-03T21:37:33.956702+0000 (oldest deadline 2024-05-03T21:37:59.856824+0000) 2024-05-03T21:38:11.324 INFO:journalctl@ceph.osd.11.smithi138.stdout:May 03 21:38:11 smithi138 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-11[58547]: 2024-05-03T21:38:11.043+0000 7fdfdcb59700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.67:6814 osd.3 since back 2024-05-03T21:37:43.759130+0000 front 2024-05-03T21:37:43.759044+0000 (oldest deadline 2024-05-03T21:38:09.058895+0000) 2024-05-03T21:38:11.495 INFO:journalctl@ceph.osd.7.smithi082.stdout:May 03 21:38:11 smithi082 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-7[50761]: 2024-05-03T21:38:11.181+0000 7f44b54b5700 -1 osd.7 87 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-05-03T21:37:26.771556+0000 front 2024-05-03T21:37:26.771586+0000 (oldest deadline 2024-05-03T21:37:54.372236+0000) 2024-05-03T21:38:11.495 INFO:journalctl@ceph.osd.7.smithi082.stdout:May 03 21:38:11 smithi082 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-7[50761]: 2024-05-03T21:38:11.181+0000 7f44b54b5700 -1 osd.7 87 heartbeat_check: no reply from 172.21.15.138:6804 osd.2 since back 2024-05-03T21:37:34.372911+0000 front 2024-05-03T21:37:34.372806+0000 (oldest deadline 2024-05-03T21:37:59.072727+0000) 2024-05-03T21:38:11.496 INFO:journalctl@ceph.osd.7.smithi082.stdout:May 03 21:38:11 smithi082 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-7[50761]: 2024-05-03T21:38:11.181+0000 7f44b54b5700 -1 osd.7 87 heartbeat_check: no reply from 172.21.15.67:6814 osd.3 since back 2024-05-03T21:37:44.973797+0000 front 2024-05-03T21:37:44.973764+0000 (oldest deadline 2024-05-03T21:38:10.873716+0000) 2024-05-03T21:38:11.496 INFO:journalctl@ceph.osd.7.smithi082.stdout:May 03 21:38:11 smithi082 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-7[50761]: 2024-05-03T21:38:11.296+0000 7f44b8ec1700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-03T21:38:11.496 INFO:journalctl@ceph.osd.7.smithi082.stdout:May 03 21:38:11 smithi082 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-7[50761]: 2024-05-03T21:38:11.296+0000 7f44b8ec1700 -1 osd.7 87 *** Got signal Terminated *** 2024-05-03T21:38:11.496 INFO:journalctl@ceph.osd.7.smithi082.stdout:May 03 21:38:11 smithi082 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-7[50761]: 2024-05-03T21:38:11.296+0000 7f44b8ec1700 -1 osd.7 87 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-05-03T21:38:11.824 INFO:journalctl@ceph.osd.8.smithi138.stdout:May 03 21:38:11 smithi138 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-8[51920]: 2024-05-03T21:38:11.337+0000 7f4b828df700 -1 osd.8 87 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-05-03T21:37:29.505823+0000 front 2024-05-03T21:37:29.505840+0000 (oldest deadline 2024-05-03T21:37:50.605770+0000) 2024-05-03T21:38:11.824 INFO:journalctl@ceph.osd.8.smithi138.stdout:May 03 21:38:11 smithi138 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-8[51920]: 2024-05-03T21:38:11.337+0000 7f4b828df700 -1 osd.8 87 heartbeat_check: no reply from 172.21.15.138:6804 osd.2 since back 2024-05-03T21:37:34.706687+0000 front 2024-05-03T21:37:34.706869+0000 (oldest deadline 2024-05-03T21:37:58.206880+0000) 2024-05-03T21:38:11.824 INFO:journalctl@ceph.osd.8.smithi138.stdout:May 03 21:38:11 smithi138 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-8[51920]: 2024-05-03T21:38:11.337+0000 7f4b828df700 -1 osd.8 87 heartbeat_check: no reply from 172.21.15.67:6814 osd.3 since back 2024-05-03T21:37:44.508584+0000 front 2024-05-03T21:37:44.508817+0000 (oldest deadline 2024-05-03T21:38:09.208689+0000) 2024-05-03T21:38:11.971 INFO:journalctl@ceph.osd.9.smithi067.stdout:May 03 21:38:11 smithi067 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-9[62235]: 2024-05-03T21:38:11.620+0000 7fcff92a2700 -1 osd.9 87 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-03T21:37:20.101372+0000 front 2024-05-03T21:37:20.101400+0000 (oldest deadline 2024-05-03T21:37:44.801519+0000) 2024-05-03T21:38:11.971 INFO:journalctl@ceph.osd.9.smithi067.stdout:May 03 21:38:11 smithi067 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-9[62235]: 2024-05-03T21:38:11.620+0000 7fcff92a2700 -1 osd.9 87 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-05-03T21:37:27.702901+0000 front 2024-05-03T21:37:27.702918+0000 (oldest deadline 2024-05-03T21:37:52.402882+0000) 2024-05-03T21:38:11.971 INFO:journalctl@ceph.osd.9.smithi067.stdout:May 03 21:38:11 smithi067 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-9[62235]: 2024-05-03T21:38:11.620+0000 7fcff92a2700 -1 osd.9 87 heartbeat_check: no reply from 172.21.15.138:6804 osd.2 since back 2024-05-03T21:37:34.605024+0000 front 2024-05-03T21:37:34.604986+0000 (oldest deadline 2024-05-03T21:37:59.904623+0000) 2024-05-03T21:38:11.971 INFO:journalctl@ceph.osd.9.smithi067.stdout:May 03 21:38:11 smithi067 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-9[62235]: 2024-05-03T21:38:11.620+0000 7fcff92a2700 -1 osd.9 87 heartbeat_check: no reply from 172.21.15.67:6814 osd.3 since back 2024-05-03T21:37:42.805355+0000 front 2024-05-03T21:37:42.805441+0000 (oldest deadline 2024-05-03T21:38:08.105521+0000) 2024-05-03T21:38:12.221 INFO:journalctl@ceph.osd.10.smithi082.stdout:May 03 21:38:11 smithi082 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-10[57676]: 2024-05-03T21:38:11.944+0000 7f790b86c700 -1 osd.10 87 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-03T21:37:18.101114+0000 front 2024-05-03T21:37:18.101101+0000 (oldest deadline 2024-05-03T21:37:42.800953+0000) 2024-05-03T21:38:12.221 INFO:journalctl@ceph.osd.10.smithi082.stdout:May 03 21:38:11 smithi082 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-10[57676]: 2024-05-03T21:38:11.944+0000 7f790b86c700 -1 osd.10 87 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-05-03T21:37:28.303836+0000 front 2024-05-03T21:37:28.303941+0000 (oldest deadline 2024-05-03T21:37:51.804075+0000) 2024-05-03T21:38:12.221 INFO:journalctl@ceph.osd.10.smithi082.stdout:May 03 21:38:11 smithi082 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-10[57676]: 2024-05-03T21:38:11.944+0000 7f790b86c700 -1 osd.10 87 heartbeat_check: no reply from 172.21.15.138:6804 osd.2 since back 2024-05-03T21:37:36.406076+0000 front 2024-05-03T21:37:36.406028+0000 (oldest deadline 2024-05-03T21:38:01.705914+0000) 2024-05-03T21:38:12.221 INFO:journalctl@ceph.osd.10.smithi082.stdout:May 03 21:38:11 smithi082 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-10[57676]: 2024-05-03T21:38:11.944+0000 7f790b86c700 -1 osd.10 87 heartbeat_check: no reply from 172.21.15.67:6814 osd.3 since back 2024-05-03T21:37:41.706645+0000 front 2024-05-03T21:37:41.706615+0000 (oldest deadline 2024-05-03T21:38:08.107018+0000) 2024-05-03T21:38:12.324 INFO:journalctl@ceph.osd.11.smithi138.stdout:May 03 21:38:12 smithi138 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-11[58547]: 2024-05-03T21:38:12.053+0000 7fdfdcb59700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-03T21:37:21.054826+0000 front 2024-05-03T21:37:21.054670+0000 (oldest deadline 2024-05-03T21:37:45.154864+0000) 2024-05-03T21:38:12.324 INFO:journalctl@ceph.osd.11.smithi138.stdout:May 03 21:38:12 smithi138 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-11[58547]: 2024-05-03T21:38:12.053+0000 7fdfdcb59700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-05-03T21:37:29.256054+0000 front 2024-05-03T21:37:29.256037+0000 (oldest deadline 2024-05-03T21:37:53.956115+0000) 2024-05-03T21:38:12.324 INFO:journalctl@ceph.osd.11.smithi138.stdout:May 03 21:38:12 smithi138 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-11[58547]: 2024-05-03T21:38:12.053+0000 7fdfdcb59700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.138:6804 osd.2 since back 2024-05-03T21:37:33.956668+0000 front 2024-05-03T21:37:33.956702+0000 (oldest deadline 2024-05-03T21:37:59.856824+0000) 2024-05-03T21:38:12.324 INFO:journalctl@ceph.osd.11.smithi138.stdout:May 03 21:38:12 smithi138 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-11[58547]: 2024-05-03T21:38:12.053+0000 7fdfdcb59700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.67:6814 osd.3 since back 2024-05-03T21:37:43.759130+0000 front 2024-05-03T21:37:43.759044+0000 (oldest deadline 2024-05-03T21:38:09.058895+0000) 2024-05-03T21:38:12.495 INFO:journalctl@ceph.osd.7.smithi082.stdout:May 03 21:38:12 smithi082 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-7[50761]: 2024-05-03T21:38:12.219+0000 7f44b54b5700 -1 osd.7 87 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-05-03T21:37:26.771556+0000 front 2024-05-03T21:37:26.771586+0000 (oldest deadline 2024-05-03T21:37:54.372236+0000) 2024-05-03T21:38:12.495 INFO:journalctl@ceph.osd.7.smithi082.stdout:May 03 21:38:12 smithi082 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-7[50761]: 2024-05-03T21:38:12.219+0000 7f44b54b5700 -1 osd.7 87 heartbeat_check: no reply from 172.21.15.138:6804 osd.2 since back 2024-05-03T21:37:34.372911+0000 front 2024-05-03T21:37:34.372806+0000 (oldest deadline 2024-05-03T21:37:59.072727+0000) 2024-05-03T21:38:12.495 INFO:journalctl@ceph.osd.7.smithi082.stdout:May 03 21:38:12 smithi082 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-7[50761]: 2024-05-03T21:38:12.219+0000 7f44b54b5700 -1 osd.7 87 heartbeat_check: no reply from 172.21.15.67:6814 osd.3 since back 2024-05-03T21:37:44.973797+0000 front 2024-05-03T21:37:44.973764+0000 (oldest deadline 2024-05-03T21:38:10.873716+0000) 2024-05-03T21:38:12.824 INFO:journalctl@ceph.osd.8.smithi138.stdout:May 03 21:38:12 smithi138 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-8[51920]: 2024-05-03T21:38:12.354+0000 7f4b828df700 -1 osd.8 87 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-05-03T21:37:29.505823+0000 front 2024-05-03T21:37:29.505840+0000 (oldest deadline 2024-05-03T21:37:50.605770+0000) 2024-05-03T21:38:12.824 INFO:journalctl@ceph.osd.8.smithi138.stdout:May 03 21:38:12 smithi138 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-8[51920]: 2024-05-03T21:38:12.354+0000 7f4b828df700 -1 osd.8 87 heartbeat_check: no reply from 172.21.15.138:6804 osd.2 since back 2024-05-03T21:37:34.706687+0000 front 2024-05-03T21:37:34.706869+0000 (oldest deadline 2024-05-03T21:37:58.206880+0000) 2024-05-03T21:38:12.824 INFO:journalctl@ceph.osd.8.smithi138.stdout:May 03 21:38:12 smithi138 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-8[51920]: 2024-05-03T21:38:12.354+0000 7f4b828df700 -1 osd.8 87 heartbeat_check: no reply from 172.21.15.67:6814 osd.3 since back 2024-05-03T21:37:44.508584+0000 front 2024-05-03T21:37:44.508817+0000 (oldest deadline 2024-05-03T21:38:09.208689+0000) 2024-05-03T21:38:12.971 INFO:journalctl@ceph.osd.9.smithi067.stdout:May 03 21:38:12 smithi067 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-9[62235]: 2024-05-03T21:38:12.597+0000 7fcff92a2700 -1 osd.9 87 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-03T21:37:20.101372+0000 front 2024-05-03T21:37:20.101400+0000 (oldest deadline 2024-05-03T21:37:44.801519+0000) 2024-05-03T21:38:12.971 INFO:journalctl@ceph.osd.9.smithi067.stdout:May 03 21:38:12 smithi067 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-9[62235]: 2024-05-03T21:38:12.597+0000 7fcff92a2700 -1 osd.9 87 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-05-03T21:37:27.702901+0000 front 2024-05-03T21:37:27.702918+0000 (oldest deadline 2024-05-03T21:37:52.402882+0000) 2024-05-03T21:38:12.971 INFO:journalctl@ceph.osd.9.smithi067.stdout:May 03 21:38:12 smithi067 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-9[62235]: 2024-05-03T21:38:12.597+0000 7fcff92a2700 -1 osd.9 87 heartbeat_check: no reply from 172.21.15.138:6804 osd.2 since back 2024-05-03T21:37:34.605024+0000 front 2024-05-03T21:37:34.604986+0000 (oldest deadline 2024-05-03T21:37:59.904623+0000) 2024-05-03T21:38:12.971 INFO:journalctl@ceph.osd.9.smithi067.stdout:May 03 21:38:12 smithi067 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-9[62235]: 2024-05-03T21:38:12.597+0000 7fcff92a2700 -1 osd.9 87 heartbeat_check: no reply from 172.21.15.67:6814 osd.3 since back 2024-05-03T21:37:42.805355+0000 front 2024-05-03T21:37:42.805441+0000 (oldest deadline 2024-05-03T21:38:08.105521+0000) 2024-05-03T21:38:13.219 INFO:journalctl@ceph.osd.10.smithi082.stdout:May 03 21:38:12 smithi082 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-10[57676]: 2024-05-03T21:38:12.948+0000 7f790b86c700 -1 osd.10 87 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-03T21:37:18.101114+0000 front 2024-05-03T21:37:18.101101+0000 (oldest deadline 2024-05-03T21:37:42.800953+0000) 2024-05-03T21:38:13.219 INFO:journalctl@ceph.osd.10.smithi082.stdout:May 03 21:38:12 smithi082 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-10[57676]: 2024-05-03T21:38:12.948+0000 7f790b86c700 -1 osd.10 87 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-05-03T21:37:28.303836+0000 front 2024-05-03T21:37:28.303941+0000 (oldest deadline 2024-05-03T21:37:51.804075+0000) 2024-05-03T21:38:13.220 INFO:journalctl@ceph.osd.10.smithi082.stdout:May 03 21:38:12 smithi082 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-10[57676]: 2024-05-03T21:38:12.948+0000 7f790b86c700 -1 osd.10 87 heartbeat_check: no reply from 172.21.15.138:6804 osd.2 since back 2024-05-03T21:37:36.406076+0000 front 2024-05-03T21:37:36.406028+0000 (oldest deadline 2024-05-03T21:38:01.705914+0000) 2024-05-03T21:38:13.220 INFO:journalctl@ceph.osd.10.smithi082.stdout:May 03 21:38:12 smithi082 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-10[57676]: 2024-05-03T21:38:12.948+0000 7f790b86c700 -1 osd.10 87 heartbeat_check: no reply from 172.21.15.67:6814 osd.3 since back 2024-05-03T21:37:41.706645+0000 front 2024-05-03T21:37:41.706615+0000 (oldest deadline 2024-05-03T21:38:08.107018+0000) 2024-05-03T21:38:13.399 INFO:journalctl@ceph.osd.11.smithi138.stdout:May 03 21:38:13 smithi138 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-11[58547]: 2024-05-03T21:38:13.089+0000 7fdfdcb59700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-03T21:37:21.054826+0000 front 2024-05-03T21:37:21.054670+0000 (oldest deadline 2024-05-03T21:37:45.154864+0000) 2024-05-03T21:38:13.399 INFO:journalctl@ceph.osd.11.smithi138.stdout:May 03 21:38:13 smithi138 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-11[58547]: 2024-05-03T21:38:13.089+0000 7fdfdcb59700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-05-03T21:37:29.256054+0000 front 2024-05-03T21:37:29.256037+0000 (oldest deadline 2024-05-03T21:37:53.956115+0000) 2024-05-03T21:38:13.400 INFO:journalctl@ceph.osd.11.smithi138.stdout:May 03 21:38:13 smithi138 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-11[58547]: 2024-05-03T21:38:13.089+0000 7fdfdcb59700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.138:6804 osd.2 since back 2024-05-03T21:37:33.956668+0000 front 2024-05-03T21:37:33.956702+0000 (oldest deadline 2024-05-03T21:37:59.856824+0000) 2024-05-03T21:38:13.400 INFO:journalctl@ceph.osd.11.smithi138.stdout:May 03 21:38:13 smithi138 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-11[58547]: 2024-05-03T21:38:13.089+0000 7fdfdcb59700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.67:6814 osd.3 since back 2024-05-03T21:37:43.759130+0000 front 2024-05-03T21:37:43.759044+0000 (oldest deadline 2024-05-03T21:38:09.058895+0000) 2024-05-03T21:38:13.494 INFO:journalctl@ceph.osd.7.smithi082.stdout:May 03 21:38:13 smithi082 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-7[50761]: 2024-05-03T21:38:13.218+0000 7f44b54b5700 -1 osd.7 87 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-05-03T21:37:26.771556+0000 front 2024-05-03T21:37:26.771586+0000 (oldest deadline 2024-05-03T21:37:54.372236+0000) 2024-05-03T21:38:13.495 INFO:journalctl@ceph.osd.7.smithi082.stdout:May 03 21:38:13 smithi082 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-7[50761]: 2024-05-03T21:38:13.218+0000 7f44b54b5700 -1 osd.7 87 heartbeat_check: no reply from 172.21.15.138:6804 osd.2 since back 2024-05-03T21:37:34.372911+0000 front 2024-05-03T21:37:34.372806+0000 (oldest deadline 2024-05-03T21:37:59.072727+0000) 2024-05-03T21:38:13.495 INFO:journalctl@ceph.osd.7.smithi082.stdout:May 03 21:38:13 smithi082 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-7[50761]: 2024-05-03T21:38:13.218+0000 7f44b54b5700 -1 osd.7 87 heartbeat_check: no reply from 172.21.15.67:6814 osd.3 since back 2024-05-03T21:37:44.973797+0000 front 2024-05-03T21:37:44.973764+0000 (oldest deadline 2024-05-03T21:38:10.873716+0000) 2024-05-03T21:38:13.824 INFO:journalctl@ceph.osd.8.smithi138.stdout:May 03 21:38:13 smithi138 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-8[51920]: 2024-05-03T21:38:13.398+0000 7f4b828df700 -1 osd.8 87 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-05-03T21:37:29.505823+0000 front 2024-05-03T21:37:29.505840+0000 (oldest deadline 2024-05-03T21:37:50.605770+0000) 2024-05-03T21:38:13.824 INFO:journalctl@ceph.osd.8.smithi138.stdout:May 03 21:38:13 smithi138 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-8[51920]: 2024-05-03T21:38:13.398+0000 7f4b828df700 -1 osd.8 87 heartbeat_check: no reply from 172.21.15.138:6804 osd.2 since back 2024-05-03T21:37:34.706687+0000 front 2024-05-03T21:37:34.706869+0000 (oldest deadline 2024-05-03T21:37:58.206880+0000) 2024-05-03T21:38:13.824 INFO:journalctl@ceph.osd.8.smithi138.stdout:May 03 21:38:13 smithi138 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-8[51920]: 2024-05-03T21:38:13.398+0000 7f4b828df700 -1 osd.8 87 heartbeat_check: no reply from 172.21.15.67:6814 osd.3 since back 2024-05-03T21:37:44.508584+0000 front 2024-05-03T21:37:44.508817+0000 (oldest deadline 2024-05-03T21:38:09.208689+0000) 2024-05-03T21:38:13.970 INFO:journalctl@ceph.osd.9.smithi067.stdout:May 03 21:38:13 smithi067 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-9[62235]: 2024-05-03T21:38:13.605+0000 7fcff92a2700 -1 osd.9 87 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-03T21:37:20.101372+0000 front 2024-05-03T21:37:20.101400+0000 (oldest deadline 2024-05-03T21:37:44.801519+0000) 2024-05-03T21:38:13.971 INFO:journalctl@ceph.osd.9.smithi067.stdout:May 03 21:38:13 smithi067 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-9[62235]: 2024-05-03T21:38:13.605+0000 7fcff92a2700 -1 osd.9 87 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-05-03T21:37:27.702901+0000 front 2024-05-03T21:37:27.702918+0000 (oldest deadline 2024-05-03T21:37:52.402882+0000) 2024-05-03T21:38:13.971 INFO:journalctl@ceph.osd.9.smithi067.stdout:May 03 21:38:13 smithi067 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-9[62235]: 2024-05-03T21:38:13.605+0000 7fcff92a2700 -1 osd.9 87 heartbeat_check: no reply from 172.21.15.138:6804 osd.2 since back 2024-05-03T21:37:34.605024+0000 front 2024-05-03T21:37:34.604986+0000 (oldest deadline 2024-05-03T21:37:59.904623+0000) 2024-05-03T21:38:13.971 INFO:journalctl@ceph.osd.9.smithi067.stdout:May 03 21:38:13 smithi067 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-9[62235]: 2024-05-03T21:38:13.605+0000 7fcff92a2700 -1 osd.9 87 heartbeat_check: no reply from 172.21.15.67:6814 osd.3 since back 2024-05-03T21:37:42.805355+0000 front 2024-05-03T21:37:42.805441+0000 (oldest deadline 2024-05-03T21:38:08.105521+0000) 2024-05-03T21:38:14.245 INFO:journalctl@ceph.osd.7.smithi082.stdout:May 03 21:38:14 smithi082 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-7[50761]: 2024-05-03T21:38:14.202+0000 7f44b54b5700 -1 osd.7 87 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-05-03T21:37:26.771556+0000 front 2024-05-03T21:37:26.771586+0000 (oldest deadline 2024-05-03T21:37:54.372236+0000) 2024-05-03T21:38:14.245 INFO:journalctl@ceph.osd.7.smithi082.stdout:May 03 21:38:14 smithi082 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-7[50761]: 2024-05-03T21:38:14.202+0000 7f44b54b5700 -1 osd.7 87 heartbeat_check: no reply from 172.21.15.138:6804 osd.2 since back 2024-05-03T21:37:34.372911+0000 front 2024-05-03T21:37:34.372806+0000 (oldest deadline 2024-05-03T21:37:59.072727+0000) 2024-05-03T21:38:14.245 INFO:journalctl@ceph.osd.7.smithi082.stdout:May 03 21:38:14 smithi082 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-7[50761]: 2024-05-03T21:38:14.202+0000 7f44b54b5700 -1 osd.7 87 heartbeat_check: no reply from 172.21.15.67:6814 osd.3 since back 2024-05-03T21:37:44.973797+0000 front 2024-05-03T21:37:44.973764+0000 (oldest deadline 2024-05-03T21:38:10.873716+0000) 2024-05-03T21:38:14.246 INFO:journalctl@ceph.osd.10.smithi082.stdout:May 03 21:38:13 smithi082 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-10[57676]: 2024-05-03T21:38:13.972+0000 7f790b86c700 -1 osd.10 87 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-03T21:37:18.101114+0000 front 2024-05-03T21:37:18.101101+0000 (oldest deadline 2024-05-03T21:37:42.800953+0000) 2024-05-03T21:38:14.246 INFO:journalctl@ceph.osd.10.smithi082.stdout:May 03 21:38:13 smithi082 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-10[57676]: 2024-05-03T21:38:13.972+0000 7f790b86c700 -1 osd.10 87 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-05-03T21:37:28.303836+0000 front 2024-05-03T21:37:28.303941+0000 (oldest deadline 2024-05-03T21:37:51.804075+0000) 2024-05-03T21:38:14.246 INFO:journalctl@ceph.osd.10.smithi082.stdout:May 03 21:38:13 smithi082 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-10[57676]: 2024-05-03T21:38:13.972+0000 7f790b86c700 -1 osd.10 87 heartbeat_check: no reply from 172.21.15.138:6804 osd.2 since back 2024-05-03T21:37:36.406076+0000 front 2024-05-03T21:37:36.406028+0000 (oldest deadline 2024-05-03T21:38:01.705914+0000) 2024-05-03T21:38:14.246 INFO:journalctl@ceph.osd.10.smithi082.stdout:May 03 21:38:13 smithi082 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-10[57676]: 2024-05-03T21:38:13.972+0000 7f790b86c700 -1 osd.10 87 heartbeat_check: no reply from 172.21.15.67:6814 osd.3 since back 2024-05-03T21:37:41.706645+0000 front 2024-05-03T21:37:41.706615+0000 (oldest deadline 2024-05-03T21:38:08.107018+0000) 2024-05-03T21:38:14.409 INFO:journalctl@ceph.osd.11.smithi138.stdout:May 03 21:38:14 smithi138 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-11[58547]: 2024-05-03T21:38:14.126+0000 7fdfdcb59700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-03T21:37:21.054826+0000 front 2024-05-03T21:37:21.054670+0000 (oldest deadline 2024-05-03T21:37:45.154864+0000) 2024-05-03T21:38:14.410 INFO:journalctl@ceph.osd.11.smithi138.stdout:May 03 21:38:14 smithi138 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-11[58547]: 2024-05-03T21:38:14.126+0000 7fdfdcb59700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-05-03T21:37:29.256054+0000 front 2024-05-03T21:37:29.256037+0000 (oldest deadline 2024-05-03T21:37:53.956115+0000) 2024-05-03T21:38:14.410 INFO:journalctl@ceph.osd.11.smithi138.stdout:May 03 21:38:14 smithi138 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-11[58547]: 2024-05-03T21:38:14.126+0000 7fdfdcb59700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.138:6804 osd.2 since back 2024-05-03T21:37:33.956668+0000 front 2024-05-03T21:37:33.956702+0000 (oldest deadline 2024-05-03T21:37:59.856824+0000) 2024-05-03T21:38:14.410 INFO:journalctl@ceph.osd.11.smithi138.stdout:May 03 21:38:14 smithi138 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-11[58547]: 2024-05-03T21:38:14.126+0000 7fdfdcb59700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.67:6814 osd.3 since back 2024-05-03T21:37:43.759130+0000 front 2024-05-03T21:37:43.759044+0000 (oldest deadline 2024-05-03T21:38:09.058895+0000) 2024-05-03T21:38:14.824 INFO:journalctl@ceph.osd.8.smithi138.stdout:May 03 21:38:14 smithi138 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-8[51920]: 2024-05-03T21:38:14.408+0000 7f4b828df700 -1 osd.8 87 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-05-03T21:37:29.505823+0000 front 2024-05-03T21:37:29.505840+0000 (oldest deadline 2024-05-03T21:37:50.605770+0000) 2024-05-03T21:38:14.824 INFO:journalctl@ceph.osd.8.smithi138.stdout:May 03 21:38:14 smithi138 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-8[51920]: 2024-05-03T21:38:14.408+0000 7f4b828df700 -1 osd.8 87 heartbeat_check: no reply from 172.21.15.138:6804 osd.2 since back 2024-05-03T21:37:34.706687+0000 front 2024-05-03T21:37:34.706869+0000 (oldest deadline 2024-05-03T21:37:58.206880+0000) 2024-05-03T21:38:14.824 INFO:journalctl@ceph.osd.8.smithi138.stdout:May 03 21:38:14 smithi138 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-8[51920]: 2024-05-03T21:38:14.408+0000 7f4b828df700 -1 osd.8 87 heartbeat_check: no reply from 172.21.15.67:6814 osd.3 since back 2024-05-03T21:37:44.508584+0000 front 2024-05-03T21:37:44.508817+0000 (oldest deadline 2024-05-03T21:38:09.208689+0000) 2024-05-03T21:38:14.970 INFO:journalctl@ceph.osd.9.smithi067.stdout:May 03 21:38:14 smithi067 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-9[62235]: 2024-05-03T21:38:14.571+0000 7fcff92a2700 -1 osd.9 87 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-03T21:37:20.101372+0000 front 2024-05-03T21:37:20.101400+0000 (oldest deadline 2024-05-03T21:37:44.801519+0000) 2024-05-03T21:38:14.971 INFO:journalctl@ceph.osd.9.smithi067.stdout:May 03 21:38:14 smithi067 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-9[62235]: 2024-05-03T21:38:14.571+0000 7fcff92a2700 -1 osd.9 87 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-05-03T21:37:27.702901+0000 front 2024-05-03T21:37:27.702918+0000 (oldest deadline 2024-05-03T21:37:52.402882+0000) 2024-05-03T21:38:14.971 INFO:journalctl@ceph.osd.9.smithi067.stdout:May 03 21:38:14 smithi067 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-9[62235]: 2024-05-03T21:38:14.571+0000 7fcff92a2700 -1 osd.9 87 heartbeat_check: no reply from 172.21.15.138:6804 osd.2 since back 2024-05-03T21:37:34.605024+0000 front 2024-05-03T21:37:34.604986+0000 (oldest deadline 2024-05-03T21:37:59.904623+0000) 2024-05-03T21:38:14.971 INFO:journalctl@ceph.osd.9.smithi067.stdout:May 03 21:38:14 smithi067 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-9[62235]: 2024-05-03T21:38:14.571+0000 7fcff92a2700 -1 osd.9 87 heartbeat_check: no reply from 172.21.15.67:6814 osd.3 since back 2024-05-03T21:37:42.805355+0000 front 2024-05-03T21:37:42.805441+0000 (oldest deadline 2024-05-03T21:38:08.105521+0000) 2024-05-03T21:38:14.971 INFO:journalctl@ceph.osd.9.smithi067.stdout:May 03 21:38:14 smithi067 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-9[62235]: 2024-05-03T21:38:14.571+0000 7fcff92a2700 -1 osd.9 87 heartbeat_check: no reply from 172.21.15.82:6812 osd.4 since back 2024-05-03T21:37:48.106270+0000 front 2024-05-03T21:37:48.106288+0000 (oldest deadline 2024-05-03T21:38:14.005940+0000) 2024-05-03T21:38:15.385 INFO:journalctl@ceph.osd.11.smithi138.stdout:May 03 21:38:15 smithi138 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-11[58547]: 2024-05-03T21:38:15.105+0000 7fdfdcb59700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-03T21:37:21.054826+0000 front 2024-05-03T21:37:21.054670+0000 (oldest deadline 2024-05-03T21:37:45.154864+0000) 2024-05-03T21:38:15.385 INFO:journalctl@ceph.osd.11.smithi138.stdout:May 03 21:38:15 smithi138 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-11[58547]: 2024-05-03T21:38:15.105+0000 7fdfdcb59700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-05-03T21:37:29.256054+0000 front 2024-05-03T21:37:29.256037+0000 (oldest deadline 2024-05-03T21:37:53.956115+0000) 2024-05-03T21:38:15.386 INFO:journalctl@ceph.osd.11.smithi138.stdout:May 03 21:38:15 smithi138 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-11[58547]: 2024-05-03T21:38:15.105+0000 7fdfdcb59700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.138:6804 osd.2 since back 2024-05-03T21:37:33.956668+0000 front 2024-05-03T21:37:33.956702+0000 (oldest deadline 2024-05-03T21:37:59.856824+0000) 2024-05-03T21:38:15.386 INFO:journalctl@ceph.osd.11.smithi138.stdout:May 03 21:38:15 smithi138 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-11[58547]: 2024-05-03T21:38:15.105+0000 7fdfdcb59700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.67:6814 osd.3 since back 2024-05-03T21:37:43.759130+0000 front 2024-05-03T21:37:43.759044+0000 (oldest deadline 2024-05-03T21:38:09.058895+0000) 2024-05-03T21:38:15.386 INFO:journalctl@ceph.osd.11.smithi138.stdout:May 03 21:38:15 smithi138 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-11[58547]: 2024-05-03T21:38:15.105+0000 7fdfdcb59700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.82:6812 osd.4 since back 2024-05-03T21:37:49.059390+0000 front 2024-05-03T21:37:49.059401+0000 (oldest deadline 2024-05-03T21:38:14.959407+0000) 2024-05-03T21:38:15.422 INFO:journalctl@ceph.osd.7.smithi082.stdout:May 03 21:38:15 smithi082 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-7[50761]: 2024-05-03T21:38:15.229+0000 7f44b54b5700 -1 osd.7 87 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-05-03T21:37:26.771556+0000 front 2024-05-03T21:37:26.771586+0000 (oldest deadline 2024-05-03T21:37:54.372236+0000) 2024-05-03T21:38:15.423 INFO:journalctl@ceph.osd.7.smithi082.stdout:May 03 21:38:15 smithi082 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-7[50761]: 2024-05-03T21:38:15.229+0000 7f44b54b5700 -1 osd.7 87 heartbeat_check: no reply from 172.21.15.138:6804 osd.2 since back 2024-05-03T21:37:34.372911+0000 front 2024-05-03T21:37:34.372806+0000 (oldest deadline 2024-05-03T21:37:59.072727+0000) 2024-05-03T21:38:15.423 INFO:journalctl@ceph.osd.7.smithi082.stdout:May 03 21:38:15 smithi082 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-7[50761]: 2024-05-03T21:38:15.229+0000 7f44b54b5700 -1 osd.7 87 heartbeat_check: no reply from 172.21.15.67:6814 osd.3 since back 2024-05-03T21:37:44.973797+0000 front 2024-05-03T21:37:44.973764+0000 (oldest deadline 2024-05-03T21:38:10.873716+0000) 2024-05-03T21:38:15.423 INFO:journalctl@ceph.osd.10.smithi082.stdout:May 03 21:38:14 smithi082 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-10[57676]: 2024-05-03T21:38:14.993+0000 7f790b86c700 -1 osd.10 87 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-03T21:37:18.101114+0000 front 2024-05-03T21:37:18.101101+0000 (oldest deadline 2024-05-03T21:37:42.800953+0000) 2024-05-03T21:38:15.424 INFO:journalctl@ceph.osd.10.smithi082.stdout:May 03 21:38:14 smithi082 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-10[57676]: 2024-05-03T21:38:14.993+0000 7f790b86c700 -1 osd.10 87 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-05-03T21:37:28.303836+0000 front 2024-05-03T21:37:28.303941+0000 (oldest deadline 2024-05-03T21:37:51.804075+0000) 2024-05-03T21:38:15.424 INFO:journalctl@ceph.osd.10.smithi082.stdout:May 03 21:38:14 smithi082 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-10[57676]: 2024-05-03T21:38:14.993+0000 7f790b86c700 -1 osd.10 87 heartbeat_check: no reply from 172.21.15.138:6804 osd.2 since back 2024-05-03T21:37:36.406076+0000 front 2024-05-03T21:37:36.406028+0000 (oldest deadline 2024-05-03T21:38:01.705914+0000) 2024-05-03T21:38:15.424 INFO:journalctl@ceph.osd.10.smithi082.stdout:May 03 21:38:14 smithi082 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-10[57676]: 2024-05-03T21:38:14.993+0000 7f790b86c700 -1 osd.10 87 heartbeat_check: no reply from 172.21.15.67:6814 osd.3 since back 2024-05-03T21:37:41.706645+0000 front 2024-05-03T21:37:41.706615+0000 (oldest deadline 2024-05-03T21:38:08.107018+0000) 2024-05-03T21:38:15.424 INFO:journalctl@ceph.osd.10.smithi082.stdout:May 03 21:38:14 smithi082 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-10[57676]: 2024-05-03T21:38:14.993+0000 7f790b86c700 -1 osd.10 87 heartbeat_check: no reply from 172.21.15.82:6812 osd.4 since back 2024-05-03T21:37:48.107476+0000 front 2024-05-03T21:37:48.107543+0000 (oldest deadline 2024-05-03T21:38:14.007654+0000) 2024-05-03T21:38:15.824 INFO:journalctl@ceph.osd.8.smithi138.stdout:May 03 21:38:15 smithi138 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-8[51920]: 2024-05-03T21:38:15.384+0000 7f4b828df700 -1 osd.8 87 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-05-03T21:37:29.505823+0000 front 2024-05-03T21:37:29.505840+0000 (oldest deadline 2024-05-03T21:37:50.605770+0000) 2024-05-03T21:38:15.824 INFO:journalctl@ceph.osd.8.smithi138.stdout:May 03 21:38:15 smithi138 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-8[51920]: 2024-05-03T21:38:15.384+0000 7f4b828df700 -1 osd.8 87 heartbeat_check: no reply from 172.21.15.138:6804 osd.2 since back 2024-05-03T21:37:34.706687+0000 front 2024-05-03T21:37:34.706869+0000 (oldest deadline 2024-05-03T21:37:58.206880+0000) 2024-05-03T21:38:15.824 INFO:journalctl@ceph.osd.8.smithi138.stdout:May 03 21:38:15 smithi138 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-8[51920]: 2024-05-03T21:38:15.384+0000 7f4b828df700 -1 osd.8 87 heartbeat_check: no reply from 172.21.15.67:6814 osd.3 since back 2024-05-03T21:37:44.508584+0000 front 2024-05-03T21:37:44.508817+0000 (oldest deadline 2024-05-03T21:38:09.208689+0000) 2024-05-03T21:38:15.970 INFO:journalctl@ceph.osd.9.smithi067.stdout:May 03 21:38:15 smithi067 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-9[62235]: 2024-05-03T21:38:15.586+0000 7fcff92a2700 -1 osd.9 87 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-03T21:37:20.101372+0000 front 2024-05-03T21:37:20.101400+0000 (oldest deadline 2024-05-03T21:37:44.801519+0000) 2024-05-03T21:38:15.971 INFO:journalctl@ceph.osd.9.smithi067.stdout:May 03 21:38:15 smithi067 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-9[62235]: 2024-05-03T21:38:15.586+0000 7fcff92a2700 -1 osd.9 87 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-05-03T21:37:27.702901+0000 front 2024-05-03T21:37:27.702918+0000 (oldest deadline 2024-05-03T21:37:52.402882+0000) 2024-05-03T21:38:15.971 INFO:journalctl@ceph.osd.9.smithi067.stdout:May 03 21:38:15 smithi067 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-9[62235]: 2024-05-03T21:38:15.586+0000 7fcff92a2700 -1 osd.9 87 heartbeat_check: no reply from 172.21.15.138:6804 osd.2 since back 2024-05-03T21:37:34.605024+0000 front 2024-05-03T21:37:34.604986+0000 (oldest deadline 2024-05-03T21:37:59.904623+0000) 2024-05-03T21:38:15.971 INFO:journalctl@ceph.osd.9.smithi067.stdout:May 03 21:38:15 smithi067 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-9[62235]: 2024-05-03T21:38:15.586+0000 7fcff92a2700 -1 osd.9 87 heartbeat_check: no reply from 172.21.15.67:6814 osd.3 since back 2024-05-03T21:37:42.805355+0000 front 2024-05-03T21:37:42.805441+0000 (oldest deadline 2024-05-03T21:38:08.105521+0000) 2024-05-03T21:38:15.971 INFO:journalctl@ceph.osd.9.smithi067.stdout:May 03 21:38:15 smithi067 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-9[62235]: 2024-05-03T21:38:15.586+0000 7fcff92a2700 -1 osd.9 87 heartbeat_check: no reply from 172.21.15.82:6812 osd.4 since back 2024-05-03T21:37:48.106270+0000 front 2024-05-03T21:37:48.106288+0000 (oldest deadline 2024-05-03T21:38:14.005940+0000) 2024-05-03T21:38:16.245 INFO:journalctl@ceph.osd.10.smithi082.stdout:May 03 21:38:15 smithi082 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-10[57676]: 2024-05-03T21:38:15.968+0000 7f790b86c700 -1 osd.10 87 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-03T21:37:18.101114+0000 front 2024-05-03T21:37:18.101101+0000 (oldest deadline 2024-05-03T21:37:42.800953+0000) 2024-05-03T21:38:16.245 INFO:journalctl@ceph.osd.10.smithi082.stdout:May 03 21:38:15 smithi082 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-10[57676]: 2024-05-03T21:38:15.968+0000 7f790b86c700 -1 osd.10 87 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-05-03T21:37:28.303836+0000 front 2024-05-03T21:37:28.303941+0000 (oldest deadline 2024-05-03T21:37:51.804075+0000) 2024-05-03T21:38:16.245 INFO:journalctl@ceph.osd.10.smithi082.stdout:May 03 21:38:15 smithi082 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-10[57676]: 2024-05-03T21:38:15.968+0000 7f790b86c700 -1 osd.10 87 heartbeat_check: no reply from 172.21.15.138:6804 osd.2 since back 2024-05-03T21:37:36.406076+0000 front 2024-05-03T21:37:36.406028+0000 (oldest deadline 2024-05-03T21:38:01.705914+0000) 2024-05-03T21:38:16.245 INFO:journalctl@ceph.osd.10.smithi082.stdout:May 03 21:38:15 smithi082 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-10[57676]: 2024-05-03T21:38:15.968+0000 7f790b86c700 -1 osd.10 87 heartbeat_check: no reply from 172.21.15.67:6814 osd.3 since back 2024-05-03T21:37:41.706645+0000 front 2024-05-03T21:37:41.706615+0000 (oldest deadline 2024-05-03T21:38:08.107018+0000) 2024-05-03T21:38:16.245 INFO:journalctl@ceph.osd.10.smithi082.stdout:May 03 21:38:15 smithi082 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-10[57676]: 2024-05-03T21:38:15.968+0000 7f790b86c700 -1 osd.10 87 heartbeat_check: no reply from 172.21.15.82:6812 osd.4 since back 2024-05-03T21:37:48.107476+0000 front 2024-05-03T21:37:48.107543+0000 (oldest deadline 2024-05-03T21:38:14.007654+0000) 2024-05-03T21:38:16.574 INFO:journalctl@ceph.osd.8.smithi138.stdout:May 03 21:38:16 smithi138 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-8[51920]: 2024-05-03T21:38:16.374+0000 7f4b828df700 -1 osd.8 87 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-05-03T21:37:29.505823+0000 front 2024-05-03T21:37:29.505840+0000 (oldest deadline 2024-05-03T21:37:50.605770+0000) 2024-05-03T21:38:16.574 INFO:journalctl@ceph.osd.8.smithi138.stdout:May 03 21:38:16 smithi138 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-8[51920]: 2024-05-03T21:38:16.374+0000 7f4b828df700 -1 osd.8 87 heartbeat_check: no reply from 172.21.15.138:6804 osd.2 since back 2024-05-03T21:37:34.706687+0000 front 2024-05-03T21:37:34.706869+0000 (oldest deadline 2024-05-03T21:37:58.206880+0000) 2024-05-03T21:38:16.574 INFO:journalctl@ceph.osd.8.smithi138.stdout:May 03 21:38:16 smithi138 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-8[51920]: 2024-05-03T21:38:16.374+0000 7f4b828df700 -1 osd.8 87 heartbeat_check: no reply from 172.21.15.67:6814 osd.3 since back 2024-05-03T21:37:44.508584+0000 front 2024-05-03T21:37:44.508817+0000 (oldest deadline 2024-05-03T21:38:09.208689+0000) 2024-05-03T21:38:16.575 INFO:journalctl@ceph.osd.11.smithi138.stdout:May 03 21:38:16 smithi138 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-11[58547]: 2024-05-03T21:38:16.126+0000 7fdfdcb59700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-03T21:37:21.054826+0000 front 2024-05-03T21:37:21.054670+0000 (oldest deadline 2024-05-03T21:37:45.154864+0000) 2024-05-03T21:38:16.575 INFO:journalctl@ceph.osd.11.smithi138.stdout:May 03 21:38:16 smithi138 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-11[58547]: 2024-05-03T21:38:16.126+0000 7fdfdcb59700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-05-03T21:37:29.256054+0000 front 2024-05-03T21:37:29.256037+0000 (oldest deadline 2024-05-03T21:37:53.956115+0000) 2024-05-03T21:38:16.575 INFO:journalctl@ceph.osd.11.smithi138.stdout:May 03 21:38:16 smithi138 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-11[58547]: 2024-05-03T21:38:16.126+0000 7fdfdcb59700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.138:6804 osd.2 since back 2024-05-03T21:37:33.956668+0000 front 2024-05-03T21:37:33.956702+0000 (oldest deadline 2024-05-03T21:37:59.856824+0000) 2024-05-03T21:38:16.575 INFO:journalctl@ceph.osd.11.smithi138.stdout:May 03 21:38:16 smithi138 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-11[58547]: 2024-05-03T21:38:16.126+0000 7fdfdcb59700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.67:6814 osd.3 since back 2024-05-03T21:37:43.759130+0000 front 2024-05-03T21:37:43.759044+0000 (oldest deadline 2024-05-03T21:38:09.058895+0000) 2024-05-03T21:38:16.575 INFO:journalctl@ceph.osd.11.smithi138.stdout:May 03 21:38:16 smithi138 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-11[58547]: 2024-05-03T21:38:16.126+0000 7fdfdcb59700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.82:6812 osd.4 since back 2024-05-03T21:37:49.059390+0000 front 2024-05-03T21:37:49.059401+0000 (oldest deadline 2024-05-03T21:38:14.959407+0000) 2024-05-03T21:38:16.619 INFO:journalctl@ceph.osd.7.smithi082.stdout:May 03 21:38:16 smithi082 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-7[50761]: 2024-05-03T21:38:16.259+0000 7f44b54b5700 -1 osd.7 87 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-05-03T21:37:26.771556+0000 front 2024-05-03T21:37:26.771586+0000 (oldest deadline 2024-05-03T21:37:54.372236+0000) 2024-05-03T21:38:16.620 INFO:journalctl@ceph.osd.7.smithi082.stdout:May 03 21:38:16 smithi082 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-7[50761]: 2024-05-03T21:38:16.259+0000 7f44b54b5700 -1 osd.7 87 heartbeat_check: no reply from 172.21.15.138:6804 osd.2 since back 2024-05-03T21:37:34.372911+0000 front 2024-05-03T21:37:34.372806+0000 (oldest deadline 2024-05-03T21:37:59.072727+0000) 2024-05-03T21:38:16.620 INFO:journalctl@ceph.osd.7.smithi082.stdout:May 03 21:38:16 smithi082 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-7[50761]: 2024-05-03T21:38:16.259+0000 7f44b54b5700 -1 osd.7 87 heartbeat_check: no reply from 172.21.15.67:6814 osd.3 since back 2024-05-03T21:37:44.973797+0000 front 2024-05-03T21:37:44.973764+0000 (oldest deadline 2024-05-03T21:38:10.873716+0000) 2024-05-03T21:38:16.620 INFO:journalctl@ceph.osd.7.smithi082.stdout:May 03 21:38:16 smithi082 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-7[50761]: 2024-05-03T21:38:16.259+0000 7f44b54b5700 -1 osd.7 87 heartbeat_check: no reply from 172.21.15.82:6812 osd.4 since back 2024-05-03T21:37:50.874101+0000 front 2024-05-03T21:37:50.874048+0000 (oldest deadline 2024-05-03T21:38:15.574135+0000) 2024-05-03T21:38:16.897 INFO:journalctl@ceph.osd.7.smithi082.stdout:May 03 21:38:16 smithi082 podman[68700]: 2024-05-03 21:38:16.619271187 +0000 UTC m=+5.416967580 container died 9cfd2cc49f13fe1f0fc3e585817a81d7d2bf3bbc084dc17d9c244f3b064a80f5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:68e15ae152ef5c805c733e76da8fc1d3dd4ee669d5ba025f457e05725baa24dd, name=ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-7, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, RELEASE=wip-rishabh-testing-20240501.193033-9a047fe, ceph=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240430, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5) 2024-05-03T21:38:16.971 INFO:journalctl@ceph.osd.9.smithi067.stdout:May 03 21:38:16 smithi067 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-9[62235]: 2024-05-03T21:38:16.581+0000 7fcff92a2700 -1 osd.9 87 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-03T21:37:20.101372+0000 front 2024-05-03T21:37:20.101400+0000 (oldest deadline 2024-05-03T21:37:44.801519+0000) 2024-05-03T21:38:16.971 INFO:journalctl@ceph.osd.9.smithi067.stdout:May 03 21:38:16 smithi067 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-9[62235]: 2024-05-03T21:38:16.581+0000 7fcff92a2700 -1 osd.9 87 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-05-03T21:37:27.702901+0000 front 2024-05-03T21:37:27.702918+0000 (oldest deadline 2024-05-03T21:37:52.402882+0000) 2024-05-03T21:38:16.971 INFO:journalctl@ceph.osd.9.smithi067.stdout:May 03 21:38:16 smithi067 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-9[62235]: 2024-05-03T21:38:16.581+0000 7fcff92a2700 -1 osd.9 87 heartbeat_check: no reply from 172.21.15.138:6804 osd.2 since back 2024-05-03T21:37:34.605024+0000 front 2024-05-03T21:37:34.604986+0000 (oldest deadline 2024-05-03T21:37:59.904623+0000) 2024-05-03T21:38:16.971 INFO:journalctl@ceph.osd.9.smithi067.stdout:May 03 21:38:16 smithi067 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-9[62235]: 2024-05-03T21:38:16.581+0000 7fcff92a2700 -1 osd.9 87 heartbeat_check: no reply from 172.21.15.67:6814 osd.3 since back 2024-05-03T21:37:42.805355+0000 front 2024-05-03T21:37:42.805441+0000 (oldest deadline 2024-05-03T21:38:08.105521+0000) 2024-05-03T21:38:16.971 INFO:journalctl@ceph.osd.9.smithi067.stdout:May 03 21:38:16 smithi067 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-9[62235]: 2024-05-03T21:38:16.581+0000 7fcff92a2700 -1 osd.9 87 heartbeat_check: no reply from 172.21.15.82:6812 osd.4 since back 2024-05-03T21:37:48.106270+0000 front 2024-05-03T21:37:48.106288+0000 (oldest deadline 2024-05-03T21:38:14.005940+0000) 2024-05-03T21:38:17.214 INFO:journalctl@ceph.osd.7.smithi082.stdout:May 03 21:38:16 smithi082 podman[68700]: 2024-05-03 21:38:16.979301786 +0000 UTC m=+5.776998241 container cleanup 9cfd2cc49f13fe1f0fc3e585817a81d7d2bf3bbc084dc17d9c244f3b064a80f5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:68e15ae152ef5c805c733e76da8fc1d3dd4ee669d5ba025f457e05725baa24dd, name=ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-7, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.vendor=CentOS, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, RELEASE=wip-rishabh-testing-20240501.193033-9a047fe, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240430, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_CLEAN=True) 2024-05-03T21:38:17.214 INFO:journalctl@ceph.osd.7.smithi082.stdout:May 03 21:38:16 smithi082 bash[68700]: ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-7 2024-05-03T21:38:17.214 INFO:journalctl@ceph.osd.10.smithi082.stdout:May 03 21:38:17 smithi082 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-10[57676]: 2024-05-03T21:38:17.011+0000 7f790b86c700 -1 osd.10 87 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-03T21:37:18.101114+0000 front 2024-05-03T21:37:18.101101+0000 (oldest deadline 2024-05-03T21:37:42.800953+0000) 2024-05-03T21:38:17.215 INFO:journalctl@ceph.osd.10.smithi082.stdout:May 03 21:38:17 smithi082 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-10[57676]: 2024-05-03T21:38:17.011+0000 7f790b86c700 -1 osd.10 87 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-05-03T21:37:28.303836+0000 front 2024-05-03T21:37:28.303941+0000 (oldest deadline 2024-05-03T21:37:51.804075+0000) 2024-05-03T21:38:17.215 INFO:journalctl@ceph.osd.10.smithi082.stdout:May 03 21:38:17 smithi082 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-10[57676]: 2024-05-03T21:38:17.011+0000 7f790b86c700 -1 osd.10 87 heartbeat_check: no reply from 172.21.15.138:6804 osd.2 since back 2024-05-03T21:37:36.406076+0000 front 2024-05-03T21:37:36.406028+0000 (oldest deadline 2024-05-03T21:38:01.705914+0000) 2024-05-03T21:38:17.215 INFO:journalctl@ceph.osd.10.smithi082.stdout:May 03 21:38:17 smithi082 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-10[57676]: 2024-05-03T21:38:17.011+0000 7f790b86c700 -1 osd.10 87 heartbeat_check: no reply from 172.21.15.67:6814 osd.3 since back 2024-05-03T21:37:41.706645+0000 front 2024-05-03T21:37:41.706615+0000 (oldest deadline 2024-05-03T21:38:08.107018+0000) 2024-05-03T21:38:17.215 INFO:journalctl@ceph.osd.10.smithi082.stdout:May 03 21:38:17 smithi082 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-10[57676]: 2024-05-03T21:38:17.011+0000 7f790b86c700 -1 osd.10 87 heartbeat_check: no reply from 172.21.15.82:6812 osd.4 since back 2024-05-03T21:37:48.107476+0000 front 2024-05-03T21:37:48.107543+0000 (oldest deadline 2024-05-03T21:38:14.007654+0000) 2024-05-03T21:38:17.324 INFO:journalctl@ceph.osd.11.smithi138.stdout:May 03 21:38:17 smithi138 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-11[58547]: 2024-05-03T21:38:17.096+0000 7fdfdcb59700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-03T21:37:21.054826+0000 front 2024-05-03T21:37:21.054670+0000 (oldest deadline 2024-05-03T21:37:45.154864+0000) 2024-05-03T21:38:17.324 INFO:journalctl@ceph.osd.11.smithi138.stdout:May 03 21:38:17 smithi138 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-11[58547]: 2024-05-03T21:38:17.096+0000 7fdfdcb59700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-05-03T21:37:29.256054+0000 front 2024-05-03T21:37:29.256037+0000 (oldest deadline 2024-05-03T21:37:53.956115+0000) 2024-05-03T21:38:17.324 INFO:journalctl@ceph.osd.11.smithi138.stdout:May 03 21:38:17 smithi138 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-11[58547]: 2024-05-03T21:38:17.096+0000 7fdfdcb59700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.138:6804 osd.2 since back 2024-05-03T21:37:33.956668+0000 front 2024-05-03T21:37:33.956702+0000 (oldest deadline 2024-05-03T21:37:59.856824+0000) 2024-05-03T21:38:17.324 INFO:journalctl@ceph.osd.11.smithi138.stdout:May 03 21:38:17 smithi138 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-11[58547]: 2024-05-03T21:38:17.096+0000 7fdfdcb59700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.67:6814 osd.3 since back 2024-05-03T21:37:43.759130+0000 front 2024-05-03T21:37:43.759044+0000 (oldest deadline 2024-05-03T21:38:09.058895+0000) 2024-05-03T21:38:17.325 INFO:journalctl@ceph.osd.11.smithi138.stdout:May 03 21:38:17 smithi138 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-11[58547]: 2024-05-03T21:38:17.096+0000 7fdfdcb59700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.82:6812 osd.4 since back 2024-05-03T21:37:49.059390+0000 front 2024-05-03T21:37:49.059401+0000 (oldest deadline 2024-05-03T21:38:14.959407+0000) 2024-05-03T21:38:17.495 INFO:journalctl@ceph.osd.7.smithi082.stdout:May 03 21:38:17 smithi082 podman[68729]: 2024-05-03 21:38:17.213519222 +0000 UTC m=+0.594280766 container remove 9cfd2cc49f13fe1f0fc3e585817a81d7d2bf3bbc084dc17d9c244f3b064a80f5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:68e15ae152ef5c805c733e76da8fc1d3dd4ee669d5ba025f457e05725baa24dd, name=ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-7, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240430, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, RELEASE=wip-rishabh-testing-20240501.193033-9a047fe, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.license=GPLv2) 2024-05-03T21:38:17.770 INFO:journalctl@ceph.osd.8.smithi138.stdout:May 03 21:38:17 smithi138 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-8[51920]: 2024-05-03T21:38:17.384+0000 7f4b828df700 -1 osd.8 87 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-05-03T21:37:29.505823+0000 front 2024-05-03T21:37:29.505840+0000 (oldest deadline 2024-05-03T21:37:50.605770+0000) 2024-05-03T21:38:17.771 INFO:journalctl@ceph.osd.8.smithi138.stdout:May 03 21:38:17 smithi138 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-8[51920]: 2024-05-03T21:38:17.384+0000 7f4b828df700 -1 osd.8 87 heartbeat_check: no reply from 172.21.15.138:6804 osd.2 since back 2024-05-03T21:37:34.706687+0000 front 2024-05-03T21:37:34.706869+0000 (oldest deadline 2024-05-03T21:37:58.206880+0000) 2024-05-03T21:38:17.771 INFO:journalctl@ceph.osd.8.smithi138.stdout:May 03 21:38:17 smithi138 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-8[51920]: 2024-05-03T21:38:17.384+0000 7f4b828df700 -1 osd.8 87 heartbeat_check: no reply from 172.21.15.67:6814 osd.3 since back 2024-05-03T21:37:44.508584+0000 front 2024-05-03T21:37:44.508817+0000 (oldest deadline 2024-05-03T21:38:09.208689+0000) 2024-05-03T21:38:17.771 INFO:journalctl@ceph.osd.8.smithi138.stdout:May 03 21:38:17 smithi138 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-8[51920]: 2024-05-03T21:38:17.384+0000 7f4b828df700 -1 osd.8 87 heartbeat_check: no reply from 172.21.15.82:6812 osd.4 since back 2024-05-03T21:37:51.410442+0000 front 2024-05-03T21:37:51.410384+0000 (oldest deadline 2024-05-03T21:38:16.710195+0000) 2024-05-03T21:38:17.843 INFO:journalctl@ceph.osd.7.smithi082.stdout:May 03 21:38:17 smithi082 podman[68817]: 2024-05-03 21:38:17.44019825 +0000 UTC m=+0.018344228 image pull ee4d846202496ba6096593e8e8becc0700a5eb7ebd95536031187f5f337d4012 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:68e15ae152ef5c805c733e76da8fc1d3dd4ee669d5ba025f457e05725baa24dd 2024-05-03T21:38:17.843 INFO:journalctl@ceph.osd.7.smithi082.stdout:May 03 21:38:17 smithi082 podman[68817]: 2024-05-03 21:38:17.564464378 +0000 UTC m=+0.142610312 container create fc699ca5f25adc67cb7f4dc8ec4c722aee076f947f5b9d9c4b9c256a0a6c6837 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:68e15ae152ef5c805c733e76da8fc1d3dd4ee669d5ba025f457e05725baa24dd, name=ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-7-deactivate, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_CLEAN=True, RELEASE=wip-rishabh-testing-20240501.193033-9a047fe, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.build-date=20240430) 2024-05-03T21:38:17.843 INFO:journalctl@ceph.osd.7.smithi082.stdout:May 03 21:38:17 smithi082 podman[68817]: 2024-05-03 21:38:17.758987836 +0000 UTC m=+0.337133773 container init fc699ca5f25adc67cb7f4dc8ec4c722aee076f947f5b9d9c4b9c256a0a6c6837 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:68e15ae152ef5c805c733e76da8fc1d3dd4ee669d5ba025f457e05725baa24dd, name=ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-7-deactivate, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240430, RELEASE=wip-rishabh-testing-20240501.193033-9a047fe, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.schema-version=1.0, GIT_CLEAN=True, CEPH_POINT_RELEASE=, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image) 2024-05-03T21:38:17.843 INFO:journalctl@ceph.osd.7.smithi082.stdout:May 03 21:38:17 smithi082 podman[68817]: 2024-05-03 21:38:17.762075039 +0000 UTC m=+0.340220996 container start fc699ca5f25adc67cb7f4dc8ec4c722aee076f947f5b9d9c4b9c256a0a6c6837 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:68e15ae152ef5c805c733e76da8fc1d3dd4ee669d5ba025f457e05725baa24dd, name=ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-7-deactivate, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240430, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, RELEASE=wip-rishabh-testing-20240501.193033-9a047fe, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.license=GPLv2) 2024-05-03T21:38:17.843 INFO:journalctl@ceph.osd.7.smithi082.stdout:May 03 21:38:17 smithi082 podman[68817]: 2024-05-03 21:38:17.842076207 +0000 UTC m=+0.420222164 container attach fc699ca5f25adc67cb7f4dc8ec4c722aee076f947f5b9d9c4b9c256a0a6c6837 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:68e15ae152ef5c805c733e76da8fc1d3dd4ee669d5ba025f457e05725baa24dd, name=ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-7-deactivate, CEPH_POINT_RELEASE=, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, RELEASE=wip-rishabh-testing-20240501.193033-9a047fe, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.build-date=20240430, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2) 2024-05-03T21:38:17.971 INFO:journalctl@ceph.osd.9.smithi067.stdout:May 03 21:38:17 smithi067 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-9[62235]: 2024-05-03T21:38:17.616+0000 7fcff92a2700 -1 osd.9 87 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-03T21:37:20.101372+0000 front 2024-05-03T21:37:20.101400+0000 (oldest deadline 2024-05-03T21:37:44.801519+0000) 2024-05-03T21:38:17.971 INFO:journalctl@ceph.osd.9.smithi067.stdout:May 03 21:38:17 smithi067 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-9[62235]: 2024-05-03T21:38:17.616+0000 7fcff92a2700 -1 osd.9 87 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-05-03T21:37:27.702901+0000 front 2024-05-03T21:37:27.702918+0000 (oldest deadline 2024-05-03T21:37:52.402882+0000) 2024-05-03T21:38:17.971 INFO:journalctl@ceph.osd.9.smithi067.stdout:May 03 21:38:17 smithi067 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-9[62235]: 2024-05-03T21:38:17.616+0000 7fcff92a2700 -1 osd.9 87 heartbeat_check: no reply from 172.21.15.138:6804 osd.2 since back 2024-05-03T21:37:34.605024+0000 front 2024-05-03T21:37:34.604986+0000 (oldest deadline 2024-05-03T21:37:59.904623+0000) 2024-05-03T21:38:17.971 INFO:journalctl@ceph.osd.9.smithi067.stdout:May 03 21:38:17 smithi067 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-9[62235]: 2024-05-03T21:38:17.616+0000 7fcff92a2700 -1 osd.9 87 heartbeat_check: no reply from 172.21.15.67:6814 osd.3 since back 2024-05-03T21:37:42.805355+0000 front 2024-05-03T21:37:42.805441+0000 (oldest deadline 2024-05-03T21:38:08.105521+0000) 2024-05-03T21:38:17.971 INFO:journalctl@ceph.osd.9.smithi067.stdout:May 03 21:38:17 smithi067 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-9[62235]: 2024-05-03T21:38:17.616+0000 7fcff92a2700 -1 osd.9 87 heartbeat_check: no reply from 172.21.15.82:6812 osd.4 since back 2024-05-03T21:37:48.106270+0000 front 2024-05-03T21:37:48.106288+0000 (oldest deadline 2024-05-03T21:38:14.005940+0000) 2024-05-03T21:38:18.245 INFO:journalctl@ceph.osd.7.smithi082.stdout:May 03 21:38:18 smithi082 podman[68817]: 2024-05-03 21:38:18.054389606 +0000 UTC m=+0.632535575 container died fc699ca5f25adc67cb7f4dc8ec4c722aee076f947f5b9d9c4b9c256a0a6c6837 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:68e15ae152ef5c805c733e76da8fc1d3dd4ee669d5ba025f457e05725baa24dd, name=ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-7-deactivate, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, RELEASE=wip-rishabh-testing-20240501.193033-9a047fe, org.label-schema.build-date=20240430, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True) 2024-05-03T21:38:18.245 INFO:journalctl@ceph.osd.10.smithi082.stdout:May 03 21:38:18 smithi082 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-10[57676]: 2024-05-03T21:38:18.057+0000 7f790b86c700 -1 osd.10 87 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-03T21:37:18.101114+0000 front 2024-05-03T21:37:18.101101+0000 (oldest deadline 2024-05-03T21:37:42.800953+0000) 2024-05-03T21:38:18.246 INFO:journalctl@ceph.osd.10.smithi082.stdout:May 03 21:38:18 smithi082 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-10[57676]: 2024-05-03T21:38:18.057+0000 7f790b86c700 -1 osd.10 87 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-05-03T21:37:28.303836+0000 front 2024-05-03T21:37:28.303941+0000 (oldest deadline 2024-05-03T21:37:51.804075+0000) 2024-05-03T21:38:18.246 INFO:journalctl@ceph.osd.10.smithi082.stdout:May 03 21:38:18 smithi082 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-10[57676]: 2024-05-03T21:38:18.057+0000 7f790b86c700 -1 osd.10 87 heartbeat_check: no reply from 172.21.15.138:6804 osd.2 since back 2024-05-03T21:37:36.406076+0000 front 2024-05-03T21:37:36.406028+0000 (oldest deadline 2024-05-03T21:38:01.705914+0000) 2024-05-03T21:38:18.246 INFO:journalctl@ceph.osd.10.smithi082.stdout:May 03 21:38:18 smithi082 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-10[57676]: 2024-05-03T21:38:18.058+0000 7f790b86c700 -1 osd.10 87 heartbeat_check: no reply from 172.21.15.67:6814 osd.3 since back 2024-05-03T21:37:41.706645+0000 front 2024-05-03T21:37:41.706615+0000 (oldest deadline 2024-05-03T21:38:08.107018+0000) 2024-05-03T21:38:18.246 INFO:journalctl@ceph.osd.10.smithi082.stdout:May 03 21:38:18 smithi082 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-10[57676]: 2024-05-03T21:38:18.058+0000 7f790b86c700 -1 osd.10 87 heartbeat_check: no reply from 172.21.15.82:6812 osd.4 since back 2024-05-03T21:37:48.107476+0000 front 2024-05-03T21:37:48.107543+0000 (oldest deadline 2024-05-03T21:38:14.007654+0000) 2024-05-03T21:38:18.399 INFO:journalctl@ceph.osd.11.smithi138.stdout:May 03 21:38:18 smithi138 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-11[58547]: 2024-05-03T21:38:18.129+0000 7fdfdcb59700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-03T21:37:21.054826+0000 front 2024-05-03T21:37:21.054670+0000 (oldest deadline 2024-05-03T21:37:45.154864+0000) 2024-05-03T21:38:18.399 INFO:journalctl@ceph.osd.11.smithi138.stdout:May 03 21:38:18 smithi138 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-11[58547]: 2024-05-03T21:38:18.129+0000 7fdfdcb59700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-05-03T21:37:29.256054+0000 front 2024-05-03T21:37:29.256037+0000 (oldest deadline 2024-05-03T21:37:53.956115+0000) 2024-05-03T21:38:18.399 INFO:journalctl@ceph.osd.11.smithi138.stdout:May 03 21:38:18 smithi138 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-11[58547]: 2024-05-03T21:38:18.129+0000 7fdfdcb59700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.138:6804 osd.2 since back 2024-05-03T21:37:33.956668+0000 front 2024-05-03T21:37:33.956702+0000 (oldest deadline 2024-05-03T21:37:59.856824+0000) 2024-05-03T21:38:18.399 INFO:journalctl@ceph.osd.11.smithi138.stdout:May 03 21:38:18 smithi138 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-11[58547]: 2024-05-03T21:38:18.129+0000 7fdfdcb59700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.67:6814 osd.3 since back 2024-05-03T21:37:43.759130+0000 front 2024-05-03T21:37:43.759044+0000 (oldest deadline 2024-05-03T21:38:09.058895+0000) 2024-05-03T21:38:18.400 INFO:journalctl@ceph.osd.11.smithi138.stdout:May 03 21:38:18 smithi138 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-11[58547]: 2024-05-03T21:38:18.129+0000 7fdfdcb59700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.82:6812 osd.4 since back 2024-05-03T21:37:49.059390+0000 front 2024-05-03T21:37:49.059401+0000 (oldest deadline 2024-05-03T21:38:14.959407+0000) 2024-05-03T21:38:18.598 INFO:journalctl@ceph.osd.7.smithi082.stdout:May 03 21:38:18 smithi082 podman[68843]: 2024-05-03 21:38:18.57836915 +0000 UTC m=+0.513839526 container remove fc699ca5f25adc67cb7f4dc8ec4c722aee076f947f5b9d9c4b9c256a0a6c6837 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:68e15ae152ef5c805c733e76da8fc1d3dd4ee669d5ba025f457e05725baa24dd, name=ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-7-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-rishabh-testing-20240501.193033-9a047fe, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_CLEAN=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240430, org.label-schema.vendor=CentOS, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.license=GPLv2) 2024-05-03T21:38:18.604 DEBUG:teuthology.orchestra.run.smithi082:> sudo pkill -f 'journalctl -f -n 0 -u ceph-713bc57a-0991-11ef-bc97-c7b262605968@osd.7.service' 2024-05-03T21:38:18.642 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-03T21:38:18.643 INFO:tasks.cephadm.osd.7:Stopped osd.7 2024-05-03T21:38:18.643 INFO:tasks.cephadm.osd.8:Stopping osd.8... 2024-05-03T21:38:18.643 DEBUG:teuthology.orchestra.run.smithi138:> sudo systemctl stop ceph-713bc57a-0991-11ef-bc97-c7b262605968@osd.8 2024-05-03T21:38:18.669 INFO:journalctl@ceph.osd.8.smithi138.stdout:May 03 21:38:18 smithi138 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-8[51920]: 2024-05-03T21:38:18.397+0000 7f4b828df700 -1 osd.8 87 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-05-03T21:37:29.505823+0000 front 2024-05-03T21:37:29.505840+0000 (oldest deadline 2024-05-03T21:37:50.605770+0000) 2024-05-03T21:38:18.670 INFO:journalctl@ceph.osd.8.smithi138.stdout:May 03 21:38:18 smithi138 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-8[51920]: 2024-05-03T21:38:18.397+0000 7f4b828df700 -1 osd.8 87 heartbeat_check: no reply from 172.21.15.138:6804 osd.2 since back 2024-05-03T21:37:34.706687+0000 front 2024-05-03T21:37:34.706869+0000 (oldest deadline 2024-05-03T21:37:58.206880+0000) 2024-05-03T21:38:18.670 INFO:journalctl@ceph.osd.8.smithi138.stdout:May 03 21:38:18 smithi138 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-8[51920]: 2024-05-03T21:38:18.397+0000 7f4b828df700 -1 osd.8 87 heartbeat_check: no reply from 172.21.15.67:6814 osd.3 since back 2024-05-03T21:37:44.508584+0000 front 2024-05-03T21:37:44.508817+0000 (oldest deadline 2024-05-03T21:38:09.208689+0000) 2024-05-03T21:38:18.670 INFO:journalctl@ceph.osd.8.smithi138.stdout:May 03 21:38:18 smithi138 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-8[51920]: 2024-05-03T21:38:18.397+0000 7f4b828df700 -1 osd.8 87 heartbeat_check: no reply from 172.21.15.82:6812 osd.4 since back 2024-05-03T21:37:51.410442+0000 front 2024-05-03T21:37:51.410384+0000 (oldest deadline 2024-05-03T21:38:16.710195+0000) 2024-05-03T21:38:18.934 INFO:journalctl@ceph.osd.8.smithi138.stdout:May 03 21:38:18 smithi138 systemd[1]: Stopping Ceph osd.8 for 713bc57a-0991-11ef-bc97-c7b262605968... 2024-05-03T21:38:18.970 INFO:journalctl@ceph.osd.9.smithi067.stdout:May 03 21:38:18 smithi067 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-9[62235]: 2024-05-03T21:38:18.574+0000 7fcff92a2700 -1 osd.9 87 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-03T21:37:20.101372+0000 front 2024-05-03T21:37:20.101400+0000 (oldest deadline 2024-05-03T21:37:44.801519+0000) 2024-05-03T21:38:18.971 INFO:journalctl@ceph.osd.9.smithi067.stdout:May 03 21:38:18 smithi067 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-9[62235]: 2024-05-03T21:38:18.574+0000 7fcff92a2700 -1 osd.9 87 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-05-03T21:37:27.702901+0000 front 2024-05-03T21:37:27.702918+0000 (oldest deadline 2024-05-03T21:37:52.402882+0000) 2024-05-03T21:38:18.971 INFO:journalctl@ceph.osd.9.smithi067.stdout:May 03 21:38:18 smithi067 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-9[62235]: 2024-05-03T21:38:18.574+0000 7fcff92a2700 -1 osd.9 87 heartbeat_check: no reply from 172.21.15.138:6804 osd.2 since back 2024-05-03T21:37:34.605024+0000 front 2024-05-03T21:37:34.604986+0000 (oldest deadline 2024-05-03T21:37:59.904623+0000) 2024-05-03T21:38:18.971 INFO:journalctl@ceph.osd.9.smithi067.stdout:May 03 21:38:18 smithi067 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-9[62235]: 2024-05-03T21:38:18.574+0000 7fcff92a2700 -1 osd.9 87 heartbeat_check: no reply from 172.21.15.67:6814 osd.3 since back 2024-05-03T21:37:42.805355+0000 front 2024-05-03T21:37:42.805441+0000 (oldest deadline 2024-05-03T21:38:08.105521+0000) 2024-05-03T21:38:18.971 INFO:journalctl@ceph.osd.9.smithi067.stdout:May 03 21:38:18 smithi067 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-9[62235]: 2024-05-03T21:38:18.574+0000 7fcff92a2700 -1 osd.9 87 heartbeat_check: no reply from 172.21.15.82:6812 osd.4 since back 2024-05-03T21:37:48.106270+0000 front 2024-05-03T21:37:48.106288+0000 (oldest deadline 2024-05-03T21:38:14.005940+0000) 2024-05-03T21:38:19.324 INFO:journalctl@ceph.osd.8.smithi138.stdout:May 03 21:38:18 smithi138 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-8[51920]: 2024-05-03T21:38:18.933+0000 7f4b8a2f3700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.8 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-03T21:38:19.324 INFO:journalctl@ceph.osd.8.smithi138.stdout:May 03 21:38:18 smithi138 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-8[51920]: 2024-05-03T21:38:18.933+0000 7f4b8a2f3700 -1 osd.8 87 *** Got signal Terminated *** 2024-05-03T21:38:19.324 INFO:journalctl@ceph.osd.8.smithi138.stdout:May 03 21:38:18 smithi138 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-8[51920]: 2024-05-03T21:38:18.933+0000 7f4b8a2f3700 -1 osd.8 87 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-05-03T21:38:19.325 INFO:journalctl@ceph.osd.11.smithi138.stdout:May 03 21:38:19 smithi138 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-11[58547]: 2024-05-03T21:38:19.114+0000 7fdfdcb59700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-03T21:37:21.054826+0000 front 2024-05-03T21:37:21.054670+0000 (oldest deadline 2024-05-03T21:37:45.154864+0000) 2024-05-03T21:38:19.325 INFO:journalctl@ceph.osd.11.smithi138.stdout:May 03 21:38:19 smithi138 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-11[58547]: 2024-05-03T21:38:19.114+0000 7fdfdcb59700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-05-03T21:37:29.256054+0000 front 2024-05-03T21:37:29.256037+0000 (oldest deadline 2024-05-03T21:37:53.956115+0000) 2024-05-03T21:38:19.325 INFO:journalctl@ceph.osd.11.smithi138.stdout:May 03 21:38:19 smithi138 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-11[58547]: 2024-05-03T21:38:19.114+0000 7fdfdcb59700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.138:6804 osd.2 since back 2024-05-03T21:37:33.956668+0000 front 2024-05-03T21:37:33.956702+0000 (oldest deadline 2024-05-03T21:37:59.856824+0000) 2024-05-03T21:38:19.325 INFO:journalctl@ceph.osd.11.smithi138.stdout:May 03 21:38:19 smithi138 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-11[58547]: 2024-05-03T21:38:19.114+0000 7fdfdcb59700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.67:6814 osd.3 since back 2024-05-03T21:37:43.759130+0000 front 2024-05-03T21:37:43.759044+0000 (oldest deadline 2024-05-03T21:38:09.058895+0000) 2024-05-03T21:38:19.325 INFO:journalctl@ceph.osd.11.smithi138.stdout:May 03 21:38:19 smithi138 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-11[58547]: 2024-05-03T21:38:19.114+0000 7fdfdcb59700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.82:6812 osd.4 since back 2024-05-03T21:37:49.059390+0000 front 2024-05-03T21:37:49.059401+0000 (oldest deadline 2024-05-03T21:38:14.959407+0000) 2024-05-03T21:38:19.495 INFO:journalctl@ceph.osd.10.smithi082.stdout:May 03 21:38:19 smithi082 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-10[57676]: 2024-05-03T21:38:19.083+0000 7f790b86c700 -1 osd.10 87 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-03T21:37:18.101114+0000 front 2024-05-03T21:37:18.101101+0000 (oldest deadline 2024-05-03T21:37:42.800953+0000) 2024-05-03T21:38:19.495 INFO:journalctl@ceph.osd.10.smithi082.stdout:May 03 21:38:19 smithi082 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-10[57676]: 2024-05-03T21:38:19.083+0000 7f790b86c700 -1 osd.10 87 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-05-03T21:37:28.303836+0000 front 2024-05-03T21:37:28.303941+0000 (oldest deadline 2024-05-03T21:37:51.804075+0000) 2024-05-03T21:38:19.495 INFO:journalctl@ceph.osd.10.smithi082.stdout:May 03 21:38:19 smithi082 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-10[57676]: 2024-05-03T21:38:19.083+0000 7f790b86c700 -1 osd.10 87 heartbeat_check: no reply from 172.21.15.138:6804 osd.2 since back 2024-05-03T21:37:36.406076+0000 front 2024-05-03T21:37:36.406028+0000 (oldest deadline 2024-05-03T21:38:01.705914+0000) 2024-05-03T21:38:19.495 INFO:journalctl@ceph.osd.10.smithi082.stdout:May 03 21:38:19 smithi082 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-10[57676]: 2024-05-03T21:38:19.083+0000 7f790b86c700 -1 osd.10 87 heartbeat_check: no reply from 172.21.15.67:6814 osd.3 since back 2024-05-03T21:37:41.706645+0000 front 2024-05-03T21:37:41.706615+0000 (oldest deadline 2024-05-03T21:38:08.107018+0000) 2024-05-03T21:38:19.495 INFO:journalctl@ceph.osd.10.smithi082.stdout:May 03 21:38:19 smithi082 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-10[57676]: 2024-05-03T21:38:19.083+0000 7f790b86c700 -1 osd.10 87 heartbeat_check: no reply from 172.21.15.82:6812 osd.4 since back 2024-05-03T21:37:48.107476+0000 front 2024-05-03T21:37:48.107543+0000 (oldest deadline 2024-05-03T21:38:14.007654+0000) 2024-05-03T21:38:19.824 INFO:journalctl@ceph.osd.8.smithi138.stdout:May 03 21:38:19 smithi138 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-8[51920]: 2024-05-03T21:38:19.427+0000 7f4b828df700 -1 osd.8 87 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-05-03T21:37:29.505823+0000 front 2024-05-03T21:37:29.505840+0000 (oldest deadline 2024-05-03T21:37:50.605770+0000) 2024-05-03T21:38:19.824 INFO:journalctl@ceph.osd.8.smithi138.stdout:May 03 21:38:19 smithi138 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-8[51920]: 2024-05-03T21:38:19.427+0000 7f4b828df700 -1 osd.8 87 heartbeat_check: no reply from 172.21.15.138:6804 osd.2 since back 2024-05-03T21:37:34.706687+0000 front 2024-05-03T21:37:34.706869+0000 (oldest deadline 2024-05-03T21:37:58.206880+0000) 2024-05-03T21:38:19.824 INFO:journalctl@ceph.osd.8.smithi138.stdout:May 03 21:38:19 smithi138 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-8[51920]: 2024-05-03T21:38:19.427+0000 7f4b828df700 -1 osd.8 87 heartbeat_check: no reply from 172.21.15.67:6814 osd.3 since back 2024-05-03T21:37:44.508584+0000 front 2024-05-03T21:37:44.508817+0000 (oldest deadline 2024-05-03T21:38:09.208689+0000) 2024-05-03T21:38:19.824 INFO:journalctl@ceph.osd.8.smithi138.stdout:May 03 21:38:19 smithi138 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-8[51920]: 2024-05-03T21:38:19.427+0000 7f4b828df700 -1 osd.8 87 heartbeat_check: no reply from 172.21.15.82:6812 osd.4 since back 2024-05-03T21:37:51.410442+0000 front 2024-05-03T21:37:51.410384+0000 (oldest deadline 2024-05-03T21:38:16.710195+0000) 2024-05-03T21:38:19.971 INFO:journalctl@ceph.osd.9.smithi067.stdout:May 03 21:38:19 smithi067 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-9[62235]: 2024-05-03T21:38:19.598+0000 7fcff92a2700 -1 osd.9 87 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-03T21:37:20.101372+0000 front 2024-05-03T21:37:20.101400+0000 (oldest deadline 2024-05-03T21:37:44.801519+0000) 2024-05-03T21:38:19.971 INFO:journalctl@ceph.osd.9.smithi067.stdout:May 03 21:38:19 smithi067 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-9[62235]: 2024-05-03T21:38:19.598+0000 7fcff92a2700 -1 osd.9 87 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-05-03T21:37:27.702901+0000 front 2024-05-03T21:37:27.702918+0000 (oldest deadline 2024-05-03T21:37:52.402882+0000) 2024-05-03T21:38:19.971 INFO:journalctl@ceph.osd.9.smithi067.stdout:May 03 21:38:19 smithi067 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-9[62235]: 2024-05-03T21:38:19.598+0000 7fcff92a2700 -1 osd.9 87 heartbeat_check: no reply from 172.21.15.138:6804 osd.2 since back 2024-05-03T21:37:34.605024+0000 front 2024-05-03T21:37:34.604986+0000 (oldest deadline 2024-05-03T21:37:59.904623+0000) 2024-05-03T21:38:19.971 INFO:journalctl@ceph.osd.9.smithi067.stdout:May 03 21:38:19 smithi067 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-9[62235]: 2024-05-03T21:38:19.598+0000 7fcff92a2700 -1 osd.9 87 heartbeat_check: no reply from 172.21.15.67:6814 osd.3 since back 2024-05-03T21:37:42.805355+0000 front 2024-05-03T21:37:42.805441+0000 (oldest deadline 2024-05-03T21:38:08.105521+0000) 2024-05-03T21:38:19.971 INFO:journalctl@ceph.osd.9.smithi067.stdout:May 03 21:38:19 smithi067 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-9[62235]: 2024-05-03T21:38:19.598+0000 7fcff92a2700 -1 osd.9 87 heartbeat_check: no reply from 172.21.15.82:6812 osd.4 since back 2024-05-03T21:37:48.106270+0000 front 2024-05-03T21:37:48.106288+0000 (oldest deadline 2024-05-03T21:38:14.005940+0000) 2024-05-03T21:38:20.495 INFO:journalctl@ceph.osd.10.smithi082.stdout:May 03 21:38:20 smithi082 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-10[57676]: 2024-05-03T21:38:20.036+0000 7f790b86c700 -1 osd.10 87 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-03T21:37:18.101114+0000 front 2024-05-03T21:37:18.101101+0000 (oldest deadline 2024-05-03T21:37:42.800953+0000) 2024-05-03T21:38:20.495 INFO:journalctl@ceph.osd.10.smithi082.stdout:May 03 21:38:20 smithi082 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-10[57676]: 2024-05-03T21:38:20.036+0000 7f790b86c700 -1 osd.10 87 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-05-03T21:37:28.303836+0000 front 2024-05-03T21:37:28.303941+0000 (oldest deadline 2024-05-03T21:37:51.804075+0000) 2024-05-03T21:38:20.495 INFO:journalctl@ceph.osd.10.smithi082.stdout:May 03 21:38:20 smithi082 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-10[57676]: 2024-05-03T21:38:20.036+0000 7f790b86c700 -1 osd.10 87 heartbeat_check: no reply from 172.21.15.138:6804 osd.2 since back 2024-05-03T21:37:36.406076+0000 front 2024-05-03T21:37:36.406028+0000 (oldest deadline 2024-05-03T21:38:01.705914+0000) 2024-05-03T21:38:20.495 INFO:journalctl@ceph.osd.10.smithi082.stdout:May 03 21:38:20 smithi082 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-10[57676]: 2024-05-03T21:38:20.036+0000 7f790b86c700 -1 osd.10 87 heartbeat_check: no reply from 172.21.15.67:6814 osd.3 since back 2024-05-03T21:37:41.706645+0000 front 2024-05-03T21:37:41.706615+0000 (oldest deadline 2024-05-03T21:38:08.107018+0000) 2024-05-03T21:38:20.495 INFO:journalctl@ceph.osd.10.smithi082.stdout:May 03 21:38:20 smithi082 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-10[57676]: 2024-05-03T21:38:20.036+0000 7f790b86c700 -1 osd.10 87 heartbeat_check: no reply from 172.21.15.82:6812 osd.4 since back 2024-05-03T21:37:48.107476+0000 front 2024-05-03T21:37:48.107543+0000 (oldest deadline 2024-05-03T21:38:14.007654+0000) 2024-05-03T21:38:20.574 INFO:journalctl@ceph.osd.8.smithi138.stdout:May 03 21:38:20 smithi138 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-8[51920]: 2024-05-03T21:38:20.382+0000 7f4b828df700 -1 osd.8 87 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-05-03T21:37:29.505823+0000 front 2024-05-03T21:37:29.505840+0000 (oldest deadline 2024-05-03T21:37:50.605770+0000) 2024-05-03T21:38:20.574 INFO:journalctl@ceph.osd.8.smithi138.stdout:May 03 21:38:20 smithi138 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-8[51920]: 2024-05-03T21:38:20.382+0000 7f4b828df700 -1 osd.8 87 heartbeat_check: no reply from 172.21.15.138:6804 osd.2 since back 2024-05-03T21:37:34.706687+0000 front 2024-05-03T21:37:34.706869+0000 (oldest deadline 2024-05-03T21:37:58.206880+0000) 2024-05-03T21:38:20.574 INFO:journalctl@ceph.osd.8.smithi138.stdout:May 03 21:38:20 smithi138 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-8[51920]: 2024-05-03T21:38:20.382+0000 7f4b828df700 -1 osd.8 87 heartbeat_check: no reply from 172.21.15.67:6814 osd.3 since back 2024-05-03T21:37:44.508584+0000 front 2024-05-03T21:37:44.508817+0000 (oldest deadline 2024-05-03T21:38:09.208689+0000) 2024-05-03T21:38:20.574 INFO:journalctl@ceph.osd.8.smithi138.stdout:May 03 21:38:20 smithi138 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-8[51920]: 2024-05-03T21:38:20.382+0000 7f4b828df700 -1 osd.8 87 heartbeat_check: no reply from 172.21.15.82:6812 osd.4 since back 2024-05-03T21:37:51.410442+0000 front 2024-05-03T21:37:51.410384+0000 (oldest deadline 2024-05-03T21:38:16.710195+0000) 2024-05-03T21:38:20.575 INFO:journalctl@ceph.osd.11.smithi138.stdout:May 03 21:38:20 smithi138 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-11[58547]: 2024-05-03T21:38:20.153+0000 7fdfdcb59700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-03T21:37:21.054826+0000 front 2024-05-03T21:37:21.054670+0000 (oldest deadline 2024-05-03T21:37:45.154864+0000) 2024-05-03T21:38:20.575 INFO:journalctl@ceph.osd.11.smithi138.stdout:May 03 21:38:20 smithi138 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-11[58547]: 2024-05-03T21:38:20.153+0000 7fdfdcb59700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-05-03T21:37:29.256054+0000 front 2024-05-03T21:37:29.256037+0000 (oldest deadline 2024-05-03T21:37:53.956115+0000) 2024-05-03T21:38:20.575 INFO:journalctl@ceph.osd.11.smithi138.stdout:May 03 21:38:20 smithi138 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-11[58547]: 2024-05-03T21:38:20.153+0000 7fdfdcb59700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.138:6804 osd.2 since back 2024-05-03T21:37:33.956668+0000 front 2024-05-03T21:37:33.956702+0000 (oldest deadline 2024-05-03T21:37:59.856824+0000) 2024-05-03T21:38:20.575 INFO:journalctl@ceph.osd.11.smithi138.stdout:May 03 21:38:20 smithi138 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-11[58547]: 2024-05-03T21:38:20.153+0000 7fdfdcb59700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.67:6814 osd.3 since back 2024-05-03T21:37:43.759130+0000 front 2024-05-03T21:37:43.759044+0000 (oldest deadline 2024-05-03T21:38:09.058895+0000) 2024-05-03T21:38:20.575 INFO:journalctl@ceph.osd.11.smithi138.stdout:May 03 21:38:20 smithi138 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-11[58547]: 2024-05-03T21:38:20.153+0000 7fdfdcb59700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.82:6812 osd.4 since back 2024-05-03T21:37:49.059390+0000 front 2024-05-03T21:37:49.059401+0000 (oldest deadline 2024-05-03T21:38:14.959407+0000) 2024-05-03T21:38:20.971 INFO:journalctl@ceph.osd.9.smithi067.stdout:May 03 21:38:20 smithi067 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-9[62235]: 2024-05-03T21:38:20.616+0000 7fcff92a2700 -1 osd.9 87 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-03T21:37:20.101372+0000 front 2024-05-03T21:37:20.101400+0000 (oldest deadline 2024-05-03T21:37:44.801519+0000) 2024-05-03T21:38:20.971 INFO:journalctl@ceph.osd.9.smithi067.stdout:May 03 21:38:20 smithi067 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-9[62235]: 2024-05-03T21:38:20.616+0000 7fcff92a2700 -1 osd.9 87 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-05-03T21:37:27.702901+0000 front 2024-05-03T21:37:27.702918+0000 (oldest deadline 2024-05-03T21:37:52.402882+0000) 2024-05-03T21:38:20.971 INFO:journalctl@ceph.osd.9.smithi067.stdout:May 03 21:38:20 smithi067 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-9[62235]: 2024-05-03T21:38:20.616+0000 7fcff92a2700 -1 osd.9 87 heartbeat_check: no reply from 172.21.15.138:6804 osd.2 since back 2024-05-03T21:37:34.605024+0000 front 2024-05-03T21:37:34.604986+0000 (oldest deadline 2024-05-03T21:37:59.904623+0000) 2024-05-03T21:38:20.971 INFO:journalctl@ceph.osd.9.smithi067.stdout:May 03 21:38:20 smithi067 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-9[62235]: 2024-05-03T21:38:20.616+0000 7fcff92a2700 -1 osd.9 87 heartbeat_check: no reply from 172.21.15.67:6814 osd.3 since back 2024-05-03T21:37:42.805355+0000 front 2024-05-03T21:37:42.805441+0000 (oldest deadline 2024-05-03T21:38:08.105521+0000) 2024-05-03T21:38:20.971 INFO:journalctl@ceph.osd.9.smithi067.stdout:May 03 21:38:20 smithi067 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-9[62235]: 2024-05-03T21:38:20.616+0000 7fcff92a2700 -1 osd.9 87 heartbeat_check: no reply from 172.21.15.82:6812 osd.4 since back 2024-05-03T21:37:48.106270+0000 front 2024-05-03T21:37:48.106288+0000 (oldest deadline 2024-05-03T21:38:14.005940+0000) 2024-05-03T21:38:21.495 INFO:journalctl@ceph.osd.10.smithi082.stdout:May 03 21:38:21 smithi082 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-10[57676]: 2024-05-03T21:38:21.072+0000 7f790b86c700 -1 osd.10 87 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-03T21:37:18.101114+0000 front 2024-05-03T21:37:18.101101+0000 (oldest deadline 2024-05-03T21:37:42.800953+0000) 2024-05-03T21:38:21.495 INFO:journalctl@ceph.osd.10.smithi082.stdout:May 03 21:38:21 smithi082 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-10[57676]: 2024-05-03T21:38:21.072+0000 7f790b86c700 -1 osd.10 87 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-05-03T21:37:28.303836+0000 front 2024-05-03T21:37:28.303941+0000 (oldest deadline 2024-05-03T21:37:51.804075+0000) 2024-05-03T21:38:21.495 INFO:journalctl@ceph.osd.10.smithi082.stdout:May 03 21:38:21 smithi082 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-10[57676]: 2024-05-03T21:38:21.072+0000 7f790b86c700 -1 osd.10 87 heartbeat_check: no reply from 172.21.15.138:6804 osd.2 since back 2024-05-03T21:37:36.406076+0000 front 2024-05-03T21:37:36.406028+0000 (oldest deadline 2024-05-03T21:38:01.705914+0000) 2024-05-03T21:38:21.495 INFO:journalctl@ceph.osd.10.smithi082.stdout:May 03 21:38:21 smithi082 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-10[57676]: 2024-05-03T21:38:21.072+0000 7f790b86c700 -1 osd.10 87 heartbeat_check: no reply from 172.21.15.67:6814 osd.3 since back 2024-05-03T21:37:41.706645+0000 front 2024-05-03T21:37:41.706615+0000 (oldest deadline 2024-05-03T21:38:08.107018+0000) 2024-05-03T21:38:21.495 INFO:journalctl@ceph.osd.10.smithi082.stdout:May 03 21:38:21 smithi082 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-10[57676]: 2024-05-03T21:38:21.072+0000 7f790b86c700 -1 osd.10 87 heartbeat_check: no reply from 172.21.15.82:6812 osd.4 since back 2024-05-03T21:37:48.107476+0000 front 2024-05-03T21:37:48.107543+0000 (oldest deadline 2024-05-03T21:38:14.007654+0000) 2024-05-03T21:38:21.574 INFO:journalctl@ceph.osd.8.smithi138.stdout:May 03 21:38:21 smithi138 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-8[51920]: 2024-05-03T21:38:21.402+0000 7f4b828df700 -1 osd.8 87 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-05-03T21:37:29.505823+0000 front 2024-05-03T21:37:29.505840+0000 (oldest deadline 2024-05-03T21:37:50.605770+0000) 2024-05-03T21:38:21.575 INFO:journalctl@ceph.osd.8.smithi138.stdout:May 03 21:38:21 smithi138 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-8[51920]: 2024-05-03T21:38:21.402+0000 7f4b828df700 -1 osd.8 87 heartbeat_check: no reply from 172.21.15.138:6804 osd.2 since back 2024-05-03T21:37:34.706687+0000 front 2024-05-03T21:37:34.706869+0000 (oldest deadline 2024-05-03T21:37:58.206880+0000) 2024-05-03T21:38:21.575 INFO:journalctl@ceph.osd.8.smithi138.stdout:May 03 21:38:21 smithi138 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-8[51920]: 2024-05-03T21:38:21.402+0000 7f4b828df700 -1 osd.8 87 heartbeat_check: no reply from 172.21.15.67:6814 osd.3 since back 2024-05-03T21:37:44.508584+0000 front 2024-05-03T21:37:44.508817+0000 (oldest deadline 2024-05-03T21:38:09.208689+0000) 2024-05-03T21:38:21.575 INFO:journalctl@ceph.osd.8.smithi138.stdout:May 03 21:38:21 smithi138 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-8[51920]: 2024-05-03T21:38:21.402+0000 7f4b828df700 -1 osd.8 87 heartbeat_check: no reply from 172.21.15.82:6812 osd.4 since back 2024-05-03T21:37:51.410442+0000 front 2024-05-03T21:37:51.410384+0000 (oldest deadline 2024-05-03T21:38:16.710195+0000) 2024-05-03T21:38:21.575 INFO:journalctl@ceph.osd.11.smithi138.stdout:May 03 21:38:21 smithi138 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-11[58547]: 2024-05-03T21:38:21.202+0000 7fdfdcb59700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-03T21:37:21.054826+0000 front 2024-05-03T21:37:21.054670+0000 (oldest deadline 2024-05-03T21:37:45.154864+0000) 2024-05-03T21:38:21.575 INFO:journalctl@ceph.osd.11.smithi138.stdout:May 03 21:38:21 smithi138 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-11[58547]: 2024-05-03T21:38:21.202+0000 7fdfdcb59700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-05-03T21:37:29.256054+0000 front 2024-05-03T21:37:29.256037+0000 (oldest deadline 2024-05-03T21:37:53.956115+0000) 2024-05-03T21:38:21.576 INFO:journalctl@ceph.osd.11.smithi138.stdout:May 03 21:38:21 smithi138 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-11[58547]: 2024-05-03T21:38:21.202+0000 7fdfdcb59700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.138:6804 osd.2 since back 2024-05-03T21:37:33.956668+0000 front 2024-05-03T21:37:33.956702+0000 (oldest deadline 2024-05-03T21:37:59.856824+0000) 2024-05-03T21:38:21.576 INFO:journalctl@ceph.osd.11.smithi138.stdout:May 03 21:38:21 smithi138 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-11[58547]: 2024-05-03T21:38:21.202+0000 7fdfdcb59700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.67:6814 osd.3 since back 2024-05-03T21:37:43.759130+0000 front 2024-05-03T21:37:43.759044+0000 (oldest deadline 2024-05-03T21:38:09.058895+0000) 2024-05-03T21:38:21.576 INFO:journalctl@ceph.osd.11.smithi138.stdout:May 03 21:38:21 smithi138 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-11[58547]: 2024-05-03T21:38:21.202+0000 7fdfdcb59700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.82:6812 osd.4 since back 2024-05-03T21:37:49.059390+0000 front 2024-05-03T21:37:49.059401+0000 (oldest deadline 2024-05-03T21:38:14.959407+0000) 2024-05-03T21:38:21.971 INFO:journalctl@ceph.osd.9.smithi067.stdout:May 03 21:38:21 smithi067 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-9[62235]: 2024-05-03T21:38:21.634+0000 7fcff92a2700 -1 osd.9 87 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-03T21:37:20.101372+0000 front 2024-05-03T21:37:20.101400+0000 (oldest deadline 2024-05-03T21:37:44.801519+0000) 2024-05-03T21:38:21.971 INFO:journalctl@ceph.osd.9.smithi067.stdout:May 03 21:38:21 smithi067 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-9[62235]: 2024-05-03T21:38:21.634+0000 7fcff92a2700 -1 osd.9 87 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-05-03T21:37:27.702901+0000 front 2024-05-03T21:37:27.702918+0000 (oldest deadline 2024-05-03T21:37:52.402882+0000) 2024-05-03T21:38:21.971 INFO:journalctl@ceph.osd.9.smithi067.stdout:May 03 21:38:21 smithi067 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-9[62235]: 2024-05-03T21:38:21.634+0000 7fcff92a2700 -1 osd.9 87 heartbeat_check: no reply from 172.21.15.138:6804 osd.2 since back 2024-05-03T21:37:34.605024+0000 front 2024-05-03T21:37:34.604986+0000 (oldest deadline 2024-05-03T21:37:59.904623+0000) 2024-05-03T21:38:21.971 INFO:journalctl@ceph.osd.9.smithi067.stdout:May 03 21:38:21 smithi067 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-9[62235]: 2024-05-03T21:38:21.634+0000 7fcff92a2700 -1 osd.9 87 heartbeat_check: no reply from 172.21.15.67:6814 osd.3 since back 2024-05-03T21:37:42.805355+0000 front 2024-05-03T21:37:42.805441+0000 (oldest deadline 2024-05-03T21:38:08.105521+0000) 2024-05-03T21:38:21.971 INFO:journalctl@ceph.osd.9.smithi067.stdout:May 03 21:38:21 smithi067 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-9[62235]: 2024-05-03T21:38:21.634+0000 7fcff92a2700 -1 osd.9 87 heartbeat_check: no reply from 172.21.15.82:6812 osd.4 since back 2024-05-03T21:37:48.106270+0000 front 2024-05-03T21:37:48.106288+0000 (oldest deadline 2024-05-03T21:38:14.005940+0000) 2024-05-03T21:38:22.495 INFO:journalctl@ceph.osd.10.smithi082.stdout:May 03 21:38:22 smithi082 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-10[57676]: 2024-05-03T21:38:22.058+0000 7f790b86c700 -1 osd.10 87 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-03T21:37:18.101114+0000 front 2024-05-03T21:37:18.101101+0000 (oldest deadline 2024-05-03T21:37:42.800953+0000) 2024-05-03T21:38:22.495 INFO:journalctl@ceph.osd.10.smithi082.stdout:May 03 21:38:22 smithi082 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-10[57676]: 2024-05-03T21:38:22.058+0000 7f790b86c700 -1 osd.10 87 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-05-03T21:37:28.303836+0000 front 2024-05-03T21:37:28.303941+0000 (oldest deadline 2024-05-03T21:37:51.804075+0000) 2024-05-03T21:38:22.495 INFO:journalctl@ceph.osd.10.smithi082.stdout:May 03 21:38:22 smithi082 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-10[57676]: 2024-05-03T21:38:22.058+0000 7f790b86c700 -1 osd.10 87 heartbeat_check: no reply from 172.21.15.138:6804 osd.2 since back 2024-05-03T21:37:36.406076+0000 front 2024-05-03T21:37:36.406028+0000 (oldest deadline 2024-05-03T21:38:01.705914+0000) 2024-05-03T21:38:22.495 INFO:journalctl@ceph.osd.10.smithi082.stdout:May 03 21:38:22 smithi082 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-10[57676]: 2024-05-03T21:38:22.058+0000 7f790b86c700 -1 osd.10 87 heartbeat_check: no reply from 172.21.15.67:6814 osd.3 since back 2024-05-03T21:37:41.706645+0000 front 2024-05-03T21:37:41.706615+0000 (oldest deadline 2024-05-03T21:38:08.107018+0000) 2024-05-03T21:38:22.495 INFO:journalctl@ceph.osd.10.smithi082.stdout:May 03 21:38:22 smithi082 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-10[57676]: 2024-05-03T21:38:22.058+0000 7f790b86c700 -1 osd.10 87 heartbeat_check: no reply from 172.21.15.82:6812 osd.4 since back 2024-05-03T21:37:48.107476+0000 front 2024-05-03T21:37:48.107543+0000 (oldest deadline 2024-05-03T21:38:14.007654+0000) 2024-05-03T21:38:22.574 INFO:journalctl@ceph.osd.8.smithi138.stdout:May 03 21:38:22 smithi138 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-8[51920]: 2024-05-03T21:38:22.375+0000 7f4b828df700 -1 osd.8 87 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-05-03T21:37:29.505823+0000 front 2024-05-03T21:37:29.505840+0000 (oldest deadline 2024-05-03T21:37:50.605770+0000) 2024-05-03T21:38:22.574 INFO:journalctl@ceph.osd.8.smithi138.stdout:May 03 21:38:22 smithi138 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-8[51920]: 2024-05-03T21:38:22.375+0000 7f4b828df700 -1 osd.8 87 heartbeat_check: no reply from 172.21.15.138:6804 osd.2 since back 2024-05-03T21:37:34.706687+0000 front 2024-05-03T21:37:34.706869+0000 (oldest deadline 2024-05-03T21:37:58.206880+0000) 2024-05-03T21:38:22.574 INFO:journalctl@ceph.osd.8.smithi138.stdout:May 03 21:38:22 smithi138 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-8[51920]: 2024-05-03T21:38:22.375+0000 7f4b828df700 -1 osd.8 87 heartbeat_check: no reply from 172.21.15.67:6814 osd.3 since back 2024-05-03T21:37:44.508584+0000 front 2024-05-03T21:37:44.508817+0000 (oldest deadline 2024-05-03T21:38:09.208689+0000) 2024-05-03T21:38:22.574 INFO:journalctl@ceph.osd.8.smithi138.stdout:May 03 21:38:22 smithi138 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-8[51920]: 2024-05-03T21:38:22.375+0000 7f4b828df700 -1 osd.8 87 heartbeat_check: no reply from 172.21.15.82:6812 osd.4 since back 2024-05-03T21:37:51.410442+0000 front 2024-05-03T21:37:51.410384+0000 (oldest deadline 2024-05-03T21:38:16.710195+0000) 2024-05-03T21:38:22.575 INFO:journalctl@ceph.osd.11.smithi138.stdout:May 03 21:38:22 smithi138 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-11[58547]: 2024-05-03T21:38:22.222+0000 7fdfdcb59700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-03T21:37:21.054826+0000 front 2024-05-03T21:37:21.054670+0000 (oldest deadline 2024-05-03T21:37:45.154864+0000) 2024-05-03T21:38:22.575 INFO:journalctl@ceph.osd.11.smithi138.stdout:May 03 21:38:22 smithi138 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-11[58547]: 2024-05-03T21:38:22.222+0000 7fdfdcb59700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-05-03T21:37:29.256054+0000 front 2024-05-03T21:37:29.256037+0000 (oldest deadline 2024-05-03T21:37:53.956115+0000) 2024-05-03T21:38:22.575 INFO:journalctl@ceph.osd.11.smithi138.stdout:May 03 21:38:22 smithi138 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-11[58547]: 2024-05-03T21:38:22.222+0000 7fdfdcb59700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.138:6804 osd.2 since back 2024-05-03T21:37:33.956668+0000 front 2024-05-03T21:37:33.956702+0000 (oldest deadline 2024-05-03T21:37:59.856824+0000) 2024-05-03T21:38:22.575 INFO:journalctl@ceph.osd.11.smithi138.stdout:May 03 21:38:22 smithi138 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-11[58547]: 2024-05-03T21:38:22.222+0000 7fdfdcb59700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.67:6814 osd.3 since back 2024-05-03T21:37:43.759130+0000 front 2024-05-03T21:37:43.759044+0000 (oldest deadline 2024-05-03T21:38:09.058895+0000) 2024-05-03T21:38:22.575 INFO:journalctl@ceph.osd.11.smithi138.stdout:May 03 21:38:22 smithi138 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-11[58547]: 2024-05-03T21:38:22.222+0000 7fdfdcb59700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.82:6812 osd.4 since back 2024-05-03T21:37:49.059390+0000 front 2024-05-03T21:37:49.059401+0000 (oldest deadline 2024-05-03T21:38:14.959407+0000) 2024-05-03T21:38:22.575 INFO:journalctl@ceph.osd.11.smithi138.stdout:May 03 21:38:22 smithi138 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-11[58547]: 2024-05-03T21:38:22.222+0000 7fdfdcb59700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.138:6812 osd.5 since back 2024-05-03T21:37:56.660334+0000 front 2024-05-03T21:37:56.660386+0000 (oldest deadline 2024-05-03T21:38:21.960518+0000) 2024-05-03T21:38:22.970 INFO:journalctl@ceph.osd.9.smithi067.stdout:May 03 21:38:22 smithi067 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-9[62235]: 2024-05-03T21:38:22.625+0000 7fcff92a2700 -1 osd.9 87 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-03T21:37:20.101372+0000 front 2024-05-03T21:37:20.101400+0000 (oldest deadline 2024-05-03T21:37:44.801519+0000) 2024-05-03T21:38:22.971 INFO:journalctl@ceph.osd.9.smithi067.stdout:May 03 21:38:22 smithi067 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-9[62235]: 2024-05-03T21:38:22.625+0000 7fcff92a2700 -1 osd.9 87 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-05-03T21:37:27.702901+0000 front 2024-05-03T21:37:27.702918+0000 (oldest deadline 2024-05-03T21:37:52.402882+0000) 2024-05-03T21:38:22.971 INFO:journalctl@ceph.osd.9.smithi067.stdout:May 03 21:38:22 smithi067 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-9[62235]: 2024-05-03T21:38:22.625+0000 7fcff92a2700 -1 osd.9 87 heartbeat_check: no reply from 172.21.15.138:6804 osd.2 since back 2024-05-03T21:37:34.605024+0000 front 2024-05-03T21:37:34.604986+0000 (oldest deadline 2024-05-03T21:37:59.904623+0000) 2024-05-03T21:38:22.971 INFO:journalctl@ceph.osd.9.smithi067.stdout:May 03 21:38:22 smithi067 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-9[62235]: 2024-05-03T21:38:22.625+0000 7fcff92a2700 -1 osd.9 87 heartbeat_check: no reply from 172.21.15.67:6814 osd.3 since back 2024-05-03T21:37:42.805355+0000 front 2024-05-03T21:37:42.805441+0000 (oldest deadline 2024-05-03T21:38:08.105521+0000) 2024-05-03T21:38:22.971 INFO:journalctl@ceph.osd.9.smithi067.stdout:May 03 21:38:22 smithi067 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-9[62235]: 2024-05-03T21:38:22.625+0000 7fcff92a2700 -1 osd.9 87 heartbeat_check: no reply from 172.21.15.82:6812 osd.4 since back 2024-05-03T21:37:48.106270+0000 front 2024-05-03T21:37:48.106288+0000 (oldest deadline 2024-05-03T21:38:14.005940+0000) 2024-05-03T21:38:23.495 INFO:journalctl@ceph.osd.10.smithi082.stdout:May 03 21:38:23 smithi082 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-10[57676]: 2024-05-03T21:38:23.036+0000 7f790b86c700 -1 osd.10 87 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-03T21:37:18.101114+0000 front 2024-05-03T21:37:18.101101+0000 (oldest deadline 2024-05-03T21:37:42.800953+0000) 2024-05-03T21:38:23.495 INFO:journalctl@ceph.osd.10.smithi082.stdout:May 03 21:38:23 smithi082 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-10[57676]: 2024-05-03T21:38:23.036+0000 7f790b86c700 -1 osd.10 87 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-05-03T21:37:28.303836+0000 front 2024-05-03T21:37:28.303941+0000 (oldest deadline 2024-05-03T21:37:51.804075+0000) 2024-05-03T21:38:23.495 INFO:journalctl@ceph.osd.10.smithi082.stdout:May 03 21:38:23 smithi082 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-10[57676]: 2024-05-03T21:38:23.036+0000 7f790b86c700 -1 osd.10 87 heartbeat_check: no reply from 172.21.15.138:6804 osd.2 since back 2024-05-03T21:37:36.406076+0000 front 2024-05-03T21:37:36.406028+0000 (oldest deadline 2024-05-03T21:38:01.705914+0000) 2024-05-03T21:38:23.495 INFO:journalctl@ceph.osd.10.smithi082.stdout:May 03 21:38:23 smithi082 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-10[57676]: 2024-05-03T21:38:23.036+0000 7f790b86c700 -1 osd.10 87 heartbeat_check: no reply from 172.21.15.67:6814 osd.3 since back 2024-05-03T21:37:41.706645+0000 front 2024-05-03T21:37:41.706615+0000 (oldest deadline 2024-05-03T21:38:08.107018+0000) 2024-05-03T21:38:23.496 INFO:journalctl@ceph.osd.10.smithi082.stdout:May 03 21:38:23 smithi082 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-10[57676]: 2024-05-03T21:38:23.036+0000 7f790b86c700 -1 osd.10 87 heartbeat_check: no reply from 172.21.15.82:6812 osd.4 since back 2024-05-03T21:37:48.107476+0000 front 2024-05-03T21:37:48.107543+0000 (oldest deadline 2024-05-03T21:38:14.007654+0000) 2024-05-03T21:38:23.574 INFO:journalctl@ceph.osd.8.smithi138.stdout:May 03 21:38:23 smithi138 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-8[51920]: 2024-05-03T21:38:23.391+0000 7f4b828df700 -1 osd.8 87 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-05-03T21:37:29.505823+0000 front 2024-05-03T21:37:29.505840+0000 (oldest deadline 2024-05-03T21:37:50.605770+0000) 2024-05-03T21:38:23.574 INFO:journalctl@ceph.osd.8.smithi138.stdout:May 03 21:38:23 smithi138 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-8[51920]: 2024-05-03T21:38:23.391+0000 7f4b828df700 -1 osd.8 87 heartbeat_check: no reply from 172.21.15.138:6804 osd.2 since back 2024-05-03T21:37:34.706687+0000 front 2024-05-03T21:37:34.706869+0000 (oldest deadline 2024-05-03T21:37:58.206880+0000) 2024-05-03T21:38:23.574 INFO:journalctl@ceph.osd.8.smithi138.stdout:May 03 21:38:23 smithi138 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-8[51920]: 2024-05-03T21:38:23.391+0000 7f4b828df700 -1 osd.8 87 heartbeat_check: no reply from 172.21.15.67:6814 osd.3 since back 2024-05-03T21:37:44.508584+0000 front 2024-05-03T21:37:44.508817+0000 (oldest deadline 2024-05-03T21:38:09.208689+0000) 2024-05-03T21:38:23.574 INFO:journalctl@ceph.osd.8.smithi138.stdout:May 03 21:38:23 smithi138 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-8[51920]: 2024-05-03T21:38:23.391+0000 7f4b828df700 -1 osd.8 87 heartbeat_check: no reply from 172.21.15.82:6812 osd.4 since back 2024-05-03T21:37:51.410442+0000 front 2024-05-03T21:37:51.410384+0000 (oldest deadline 2024-05-03T21:38:16.710195+0000) 2024-05-03T21:38:23.574 INFO:journalctl@ceph.osd.11.smithi138.stdout:May 03 21:38:23 smithi138 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-11[58547]: 2024-05-03T21:38:23.246+0000 7fdfdcb59700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-03T21:37:21.054826+0000 front 2024-05-03T21:37:21.054670+0000 (oldest deadline 2024-05-03T21:37:45.154864+0000) 2024-05-03T21:38:23.575 INFO:journalctl@ceph.osd.11.smithi138.stdout:May 03 21:38:23 smithi138 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-11[58547]: 2024-05-03T21:38:23.246+0000 7fdfdcb59700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-05-03T21:37:29.256054+0000 front 2024-05-03T21:37:29.256037+0000 (oldest deadline 2024-05-03T21:37:53.956115+0000) 2024-05-03T21:38:23.575 INFO:journalctl@ceph.osd.11.smithi138.stdout:May 03 21:38:23 smithi138 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-11[58547]: 2024-05-03T21:38:23.246+0000 7fdfdcb59700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.138:6804 osd.2 since back 2024-05-03T21:37:33.956668+0000 front 2024-05-03T21:37:33.956702+0000 (oldest deadline 2024-05-03T21:37:59.856824+0000) 2024-05-03T21:38:23.575 INFO:journalctl@ceph.osd.11.smithi138.stdout:May 03 21:38:23 smithi138 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-11[58547]: 2024-05-03T21:38:23.246+0000 7fdfdcb59700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.67:6814 osd.3 since back 2024-05-03T21:37:43.759130+0000 front 2024-05-03T21:37:43.759044+0000 (oldest deadline 2024-05-03T21:38:09.058895+0000) 2024-05-03T21:38:23.575 INFO:journalctl@ceph.osd.11.smithi138.stdout:May 03 21:38:23 smithi138 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-11[58547]: 2024-05-03T21:38:23.246+0000 7fdfdcb59700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.82:6812 osd.4 since back 2024-05-03T21:37:49.059390+0000 front 2024-05-03T21:37:49.059401+0000 (oldest deadline 2024-05-03T21:38:14.959407+0000) 2024-05-03T21:38:23.575 INFO:journalctl@ceph.osd.11.smithi138.stdout:May 03 21:38:23 smithi138 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-11[58547]: 2024-05-03T21:38:23.246+0000 7fdfdcb59700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.138:6812 osd.5 since back 2024-05-03T21:37:56.660334+0000 front 2024-05-03T21:37:56.660386+0000 (oldest deadline 2024-05-03T21:38:21.960518+0000) 2024-05-03T21:38:23.971 INFO:journalctl@ceph.osd.9.smithi067.stdout:May 03 21:38:23 smithi067 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-9[62235]: 2024-05-03T21:38:23.660+0000 7fcff92a2700 -1 osd.9 87 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-03T21:37:20.101372+0000 front 2024-05-03T21:37:20.101400+0000 (oldest deadline 2024-05-03T21:37:44.801519+0000) 2024-05-03T21:38:23.971 INFO:journalctl@ceph.osd.9.smithi067.stdout:May 03 21:38:23 smithi067 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-9[62235]: 2024-05-03T21:38:23.660+0000 7fcff92a2700 -1 osd.9 87 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-05-03T21:37:27.702901+0000 front 2024-05-03T21:37:27.702918+0000 (oldest deadline 2024-05-03T21:37:52.402882+0000) 2024-05-03T21:38:23.971 INFO:journalctl@ceph.osd.9.smithi067.stdout:May 03 21:38:23 smithi067 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-9[62235]: 2024-05-03T21:38:23.660+0000 7fcff92a2700 -1 osd.9 87 heartbeat_check: no reply from 172.21.15.138:6804 osd.2 since back 2024-05-03T21:37:34.605024+0000 front 2024-05-03T21:37:34.604986+0000 (oldest deadline 2024-05-03T21:37:59.904623+0000) 2024-05-03T21:38:23.971 INFO:journalctl@ceph.osd.9.smithi067.stdout:May 03 21:38:23 smithi067 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-9[62235]: 2024-05-03T21:38:23.660+0000 7fcff92a2700 -1 osd.9 87 heartbeat_check: no reply from 172.21.15.67:6814 osd.3 since back 2024-05-03T21:37:42.805355+0000 front 2024-05-03T21:37:42.805441+0000 (oldest deadline 2024-05-03T21:38:08.105521+0000) 2024-05-03T21:38:23.971 INFO:journalctl@ceph.osd.9.smithi067.stdout:May 03 21:38:23 smithi067 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-9[62235]: 2024-05-03T21:38:23.660+0000 7fcff92a2700 -1 osd.9 87 heartbeat_check: no reply from 172.21.15.82:6812 osd.4 since back 2024-05-03T21:37:48.106270+0000 front 2024-05-03T21:37:48.106288+0000 (oldest deadline 2024-05-03T21:38:14.005940+0000) 2024-05-03T21:38:24.245 INFO:journalctl@ceph.osd.10.smithi082.stdout:May 03 21:38:23 smithi082 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-10[57676]: 2024-05-03T21:38:23.989+0000 7f790b86c700 -1 osd.10 87 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-03T21:37:18.101114+0000 front 2024-05-03T21:37:18.101101+0000 (oldest deadline 2024-05-03T21:37:42.800953+0000) 2024-05-03T21:38:24.245 INFO:journalctl@ceph.osd.10.smithi082.stdout:May 03 21:38:23 smithi082 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-10[57676]: 2024-05-03T21:38:23.989+0000 7f790b86c700 -1 osd.10 87 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-05-03T21:37:28.303836+0000 front 2024-05-03T21:37:28.303941+0000 (oldest deadline 2024-05-03T21:37:51.804075+0000) 2024-05-03T21:38:24.245 INFO:journalctl@ceph.osd.10.smithi082.stdout:May 03 21:38:23 smithi082 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-10[57676]: 2024-05-03T21:38:23.989+0000 7f790b86c700 -1 osd.10 87 heartbeat_check: no reply from 172.21.15.138:6804 osd.2 since back 2024-05-03T21:37:36.406076+0000 front 2024-05-03T21:37:36.406028+0000 (oldest deadline 2024-05-03T21:38:01.705914+0000) 2024-05-03T21:38:24.246 INFO:journalctl@ceph.osd.10.smithi082.stdout:May 03 21:38:23 smithi082 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-10[57676]: 2024-05-03T21:38:23.989+0000 7f790b86c700 -1 osd.10 87 heartbeat_check: no reply from 172.21.15.67:6814 osd.3 since back 2024-05-03T21:37:41.706645+0000 front 2024-05-03T21:37:41.706615+0000 (oldest deadline 2024-05-03T21:38:08.107018+0000) 2024-05-03T21:38:24.246 INFO:journalctl@ceph.osd.10.smithi082.stdout:May 03 21:38:23 smithi082 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-10[57676]: 2024-05-03T21:38:23.989+0000 7f790b86c700 -1 osd.10 87 heartbeat_check: no reply from 172.21.15.82:6812 osd.4 since back 2024-05-03T21:37:48.107476+0000 front 2024-05-03T21:37:48.107543+0000 (oldest deadline 2024-05-03T21:38:14.007654+0000) 2024-05-03T21:38:24.246 INFO:journalctl@ceph.osd.10.smithi082.stdout:May 03 21:38:23 smithi082 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-10[57676]: 2024-05-03T21:38:23.989+0000 7f790b86c700 -1 osd.10 87 heartbeat_check: no reply from 172.21.15.138:6812 osd.5 since back 2024-05-03T21:38:00.409331+0000 front 2024-05-03T21:38:00.409486+0000 (oldest deadline 2024-05-03T21:38:23.909130+0000) 2024-05-03T21:38:24.574 INFO:journalctl@ceph.osd.8.smithi138.stdout:May 03 21:38:24 smithi138 podman[69102]: 2024-05-03 21:38:24.386852219 +0000 UTC m=+5.552018030 container died 0726426ae7066319c36d84e0f94e95758c15fd0c14f263e9d24c65f97f0672e3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:68e15ae152ef5c805c733e76da8fc1d3dd4ee669d5ba025f457e05725baa24dd, name=ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-8, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, RELEASE=wip-rishabh-testing-20240501.193033-9a047fe, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, ceph=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240430) 2024-05-03T21:38:24.574 INFO:journalctl@ceph.osd.11.smithi138.stdout:May 03 21:38:24 smithi138 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-11[58547]: 2024-05-03T21:38:24.277+0000 7fdfdcb59700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-03T21:37:21.054826+0000 front 2024-05-03T21:37:21.054670+0000 (oldest deadline 2024-05-03T21:37:45.154864+0000) 2024-05-03T21:38:24.575 INFO:journalctl@ceph.osd.11.smithi138.stdout:May 03 21:38:24 smithi138 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-11[58547]: 2024-05-03T21:38:24.277+0000 7fdfdcb59700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-05-03T21:37:29.256054+0000 front 2024-05-03T21:37:29.256037+0000 (oldest deadline 2024-05-03T21:37:53.956115+0000) 2024-05-03T21:38:24.575 INFO:journalctl@ceph.osd.11.smithi138.stdout:May 03 21:38:24 smithi138 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-11[58547]: 2024-05-03T21:38:24.277+0000 7fdfdcb59700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.138:6804 osd.2 since back 2024-05-03T21:37:33.956668+0000 front 2024-05-03T21:37:33.956702+0000 (oldest deadline 2024-05-03T21:37:59.856824+0000) 2024-05-03T21:38:24.575 INFO:journalctl@ceph.osd.11.smithi138.stdout:May 03 21:38:24 smithi138 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-11[58547]: 2024-05-03T21:38:24.277+0000 7fdfdcb59700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.67:6814 osd.3 since back 2024-05-03T21:37:43.759130+0000 front 2024-05-03T21:37:43.759044+0000 (oldest deadline 2024-05-03T21:38:09.058895+0000) 2024-05-03T21:38:24.575 INFO:journalctl@ceph.osd.11.smithi138.stdout:May 03 21:38:24 smithi138 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-11[58547]: 2024-05-03T21:38:24.277+0000 7fdfdcb59700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.82:6812 osd.4 since back 2024-05-03T21:37:49.059390+0000 front 2024-05-03T21:37:49.059401+0000 (oldest deadline 2024-05-03T21:38:14.959407+0000) 2024-05-03T21:38:24.575 INFO:journalctl@ceph.osd.11.smithi138.stdout:May 03 21:38:24 smithi138 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-11[58547]: 2024-05-03T21:38:24.277+0000 7fdfdcb59700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.138:6812 osd.5 since back 2024-05-03T21:37:56.660334+0000 front 2024-05-03T21:37:56.660386+0000 (oldest deadline 2024-05-03T21:38:21.960518+0000) 2024-05-03T21:38:24.971 INFO:journalctl@ceph.osd.9.smithi067.stdout:May 03 21:38:24 smithi067 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-9[62235]: 2024-05-03T21:38:24.655+0000 7fcff92a2700 -1 osd.9 87 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-03T21:37:20.101372+0000 front 2024-05-03T21:37:20.101400+0000 (oldest deadline 2024-05-03T21:37:44.801519+0000) 2024-05-03T21:38:24.971 INFO:journalctl@ceph.osd.9.smithi067.stdout:May 03 21:38:24 smithi067 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-9[62235]: 2024-05-03T21:38:24.655+0000 7fcff92a2700 -1 osd.9 87 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-05-03T21:37:27.702901+0000 front 2024-05-03T21:37:27.702918+0000 (oldest deadline 2024-05-03T21:37:52.402882+0000) 2024-05-03T21:38:24.971 INFO:journalctl@ceph.osd.9.smithi067.stdout:May 03 21:38:24 smithi067 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-9[62235]: 2024-05-03T21:38:24.655+0000 7fcff92a2700 -1 osd.9 87 heartbeat_check: no reply from 172.21.15.138:6804 osd.2 since back 2024-05-03T21:37:34.605024+0000 front 2024-05-03T21:37:34.604986+0000 (oldest deadline 2024-05-03T21:37:59.904623+0000) 2024-05-03T21:38:24.971 INFO:journalctl@ceph.osd.9.smithi067.stdout:May 03 21:38:24 smithi067 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-9[62235]: 2024-05-03T21:38:24.655+0000 7fcff92a2700 -1 osd.9 87 heartbeat_check: no reply from 172.21.15.67:6814 osd.3 since back 2024-05-03T21:37:42.805355+0000 front 2024-05-03T21:37:42.805441+0000 (oldest deadline 2024-05-03T21:38:08.105521+0000) 2024-05-03T21:38:24.971 INFO:journalctl@ceph.osd.9.smithi067.stdout:May 03 21:38:24 smithi067 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-9[62235]: 2024-05-03T21:38:24.655+0000 7fcff92a2700 -1 osd.9 87 heartbeat_check: no reply from 172.21.15.82:6812 osd.4 since back 2024-05-03T21:37:48.106270+0000 front 2024-05-03T21:37:48.106288+0000 (oldest deadline 2024-05-03T21:38:14.005940+0000) 2024-05-03T21:38:25.018 INFO:journalctl@ceph.osd.8.smithi138.stdout:May 03 21:38:24 smithi138 podman[69102]: 2024-05-03 21:38:24.741314236 +0000 UTC m=+5.906480039 container cleanup 0726426ae7066319c36d84e0f94e95758c15fd0c14f263e9d24c65f97f0672e3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:68e15ae152ef5c805c733e76da8fc1d3dd4ee669d5ba025f457e05725baa24dd, name=ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-8, ceph=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240430, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, RELEASE=wip-rishabh-testing-20240501.193033-9a047fe, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image) 2024-05-03T21:38:25.018 INFO:journalctl@ceph.osd.8.smithi138.stdout:May 03 21:38:24 smithi138 bash[69102]: ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-8 2024-05-03T21:38:25.320 INFO:journalctl@ceph.osd.8.smithi138.stdout:May 03 21:38:25 smithi138 podman[69119]: 2024-05-03 21:38:25.017675738 +0000 UTC m=+0.633426658 container remove 0726426ae7066319c36d84e0f94e95758c15fd0c14f263e9d24c65f97f0672e3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:68e15ae152ef5c805c733e76da8fc1d3dd4ee669d5ba025f457e05725baa24dd, name=ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-8, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.build-date=20240430, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , ceph=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-rishabh-testing-20240501.193033-9a047fe, org.label-schema.vendor=CentOS, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, CEPH_POINT_RELEASE=) 2024-05-03T21:38:25.495 INFO:journalctl@ceph.osd.10.smithi082.stdout:May 03 21:38:25 smithi082 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-10[57676]: 2024-05-03T21:38:25.021+0000 7f790b86c700 -1 osd.10 87 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-03T21:37:18.101114+0000 front 2024-05-03T21:37:18.101101+0000 (oldest deadline 2024-05-03T21:37:42.800953+0000) 2024-05-03T21:38:25.495 INFO:journalctl@ceph.osd.10.smithi082.stdout:May 03 21:38:25 smithi082 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-10[57676]: 2024-05-03T21:38:25.021+0000 7f790b86c700 -1 osd.10 87 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-05-03T21:37:28.303836+0000 front 2024-05-03T21:37:28.303941+0000 (oldest deadline 2024-05-03T21:37:51.804075+0000) 2024-05-03T21:38:25.495 INFO:journalctl@ceph.osd.10.smithi082.stdout:May 03 21:38:25 smithi082 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-10[57676]: 2024-05-03T21:38:25.021+0000 7f790b86c700 -1 osd.10 87 heartbeat_check: no reply from 172.21.15.138:6804 osd.2 since back 2024-05-03T21:37:36.406076+0000 front 2024-05-03T21:37:36.406028+0000 (oldest deadline 2024-05-03T21:38:01.705914+0000) 2024-05-03T21:38:25.495 INFO:journalctl@ceph.osd.10.smithi082.stdout:May 03 21:38:25 smithi082 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-10[57676]: 2024-05-03T21:38:25.021+0000 7f790b86c700 -1 osd.10 87 heartbeat_check: no reply from 172.21.15.67:6814 osd.3 since back 2024-05-03T21:37:41.706645+0000 front 2024-05-03T21:37:41.706615+0000 (oldest deadline 2024-05-03T21:38:08.107018+0000) 2024-05-03T21:38:25.495 INFO:journalctl@ceph.osd.10.smithi082.stdout:May 03 21:38:25 smithi082 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-10[57676]: 2024-05-03T21:38:25.021+0000 7f790b86c700 -1 osd.10 87 heartbeat_check: no reply from 172.21.15.82:6812 osd.4 since back 2024-05-03T21:37:48.107476+0000 front 2024-05-03T21:37:48.107543+0000 (oldest deadline 2024-05-03T21:38:14.007654+0000) 2024-05-03T21:38:25.495 INFO:journalctl@ceph.osd.10.smithi082.stdout:May 03 21:38:25 smithi082 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-10[57676]: 2024-05-03T21:38:25.021+0000 7f790b86c700 -1 osd.10 87 heartbeat_check: no reply from 172.21.15.138:6812 osd.5 since back 2024-05-03T21:38:00.409331+0000 front 2024-05-03T21:38:00.409486+0000 (oldest deadline 2024-05-03T21:38:23.909130+0000) 2024-05-03T21:38:25.574 INFO:journalctl@ceph.osd.8.smithi138.stdout:May 03 21:38:25 smithi138 podman[69210]: 2024-05-03 21:38:25.256146092 +0000 UTC m=+0.017678149 image pull ee4d846202496ba6096593e8e8becc0700a5eb7ebd95536031187f5f337d4012 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:68e15ae152ef5c805c733e76da8fc1d3dd4ee669d5ba025f457e05725baa24dd 2024-05-03T21:38:25.574 INFO:journalctl@ceph.osd.8.smithi138.stdout:May 03 21:38:25 smithi138 podman[69210]: 2024-05-03 21:38:25.376823117 +0000 UTC m=+0.138355170 container create 9981774505a7f2debb56feaca6bb86ff054781a4f37b4fd9614bdcce6a8a29bd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:68e15ae152ef5c805c733e76da8fc1d3dd4ee669d5ba025f457e05725baa24dd, name=ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-8-deactivate, ceph=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.license=GPLv2, RELEASE=wip-rishabh-testing-20240501.193033-9a047fe, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240430, maintainer=Guillaume Abrioux ) 2024-05-03T21:38:25.574 INFO:journalctl@ceph.osd.11.smithi138.stdout:May 03 21:38:25 smithi138 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-11[58547]: 2024-05-03T21:38:25.319+0000 7fdfdcb59700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-03T21:37:21.054826+0000 front 2024-05-03T21:37:21.054670+0000 (oldest deadline 2024-05-03T21:37:45.154864+0000) 2024-05-03T21:38:25.575 INFO:journalctl@ceph.osd.11.smithi138.stdout:May 03 21:38:25 smithi138 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-11[58547]: 2024-05-03T21:38:25.319+0000 7fdfdcb59700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-05-03T21:37:29.256054+0000 front 2024-05-03T21:37:29.256037+0000 (oldest deadline 2024-05-03T21:37:53.956115+0000) 2024-05-03T21:38:25.575 INFO:journalctl@ceph.osd.11.smithi138.stdout:May 03 21:38:25 smithi138 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-11[58547]: 2024-05-03T21:38:25.319+0000 7fdfdcb59700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.138:6804 osd.2 since back 2024-05-03T21:37:33.956668+0000 front 2024-05-03T21:37:33.956702+0000 (oldest deadline 2024-05-03T21:37:59.856824+0000) 2024-05-03T21:38:25.575 INFO:journalctl@ceph.osd.11.smithi138.stdout:May 03 21:38:25 smithi138 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-11[58547]: 2024-05-03T21:38:25.319+0000 7fdfdcb59700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.67:6814 osd.3 since back 2024-05-03T21:37:43.759130+0000 front 2024-05-03T21:37:43.759044+0000 (oldest deadline 2024-05-03T21:38:09.058895+0000) 2024-05-03T21:38:25.575 INFO:journalctl@ceph.osd.11.smithi138.stdout:May 03 21:38:25 smithi138 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-11[58547]: 2024-05-03T21:38:25.319+0000 7fdfdcb59700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.82:6812 osd.4 since back 2024-05-03T21:37:49.059390+0000 front 2024-05-03T21:37:49.059401+0000 (oldest deadline 2024-05-03T21:38:14.959407+0000) 2024-05-03T21:38:25.575 INFO:journalctl@ceph.osd.11.smithi138.stdout:May 03 21:38:25 smithi138 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-11[58547]: 2024-05-03T21:38:25.319+0000 7fdfdcb59700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.138:6812 osd.5 since back 2024-05-03T21:37:56.660334+0000 front 2024-05-03T21:37:56.660386+0000 (oldest deadline 2024-05-03T21:38:21.960518+0000) 2024-05-03T21:38:25.876 INFO:journalctl@ceph.osd.8.smithi138.stdout:May 03 21:38:25 smithi138 podman[69210]: 2024-05-03 21:38:25.579136312 +0000 UTC m=+0.340668366 container init 9981774505a7f2debb56feaca6bb86ff054781a4f37b4fd9614bdcce6a8a29bd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:68e15ae152ef5c805c733e76da8fc1d3dd4ee669d5ba025f457e05725baa24dd, name=ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-8-deactivate, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240430, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, RELEASE=wip-rishabh-testing-20240501.193033-9a047fe, org.label-schema.name=CentOS Stream 8 Base Image) 2024-05-03T21:38:25.877 INFO:journalctl@ceph.osd.8.smithi138.stdout:May 03 21:38:25 smithi138 podman[69210]: 2024-05-03 21:38:25.582290929 +0000 UTC m=+0.343822982 container start 9981774505a7f2debb56feaca6bb86ff054781a4f37b4fd9614bdcce6a8a29bd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:68e15ae152ef5c805c733e76da8fc1d3dd4ee669d5ba025f457e05725baa24dd, name=ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-8-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240430, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, RELEASE=wip-rishabh-testing-20240501.193033-9a047fe, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_CLEAN=True) 2024-05-03T21:38:25.877 INFO:journalctl@ceph.osd.8.smithi138.stdout:May 03 21:38:25 smithi138 podman[69210]: 2024-05-03 21:38:25.646046697 +0000 UTC m=+0.407578755 container attach 9981774505a7f2debb56feaca6bb86ff054781a4f37b4fd9614bdcce6a8a29bd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:68e15ae152ef5c805c733e76da8fc1d3dd4ee669d5ba025f457e05725baa24dd, name=ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-8-deactivate, org.label-schema.license=GPLv2, ceph=True, RELEASE=wip-rishabh-testing-20240501.193033-9a047fe, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240430, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux ) 2024-05-03T21:38:25.970 INFO:journalctl@ceph.osd.9.smithi067.stdout:May 03 21:38:25 smithi067 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-9[62235]: 2024-05-03T21:38:25.677+0000 7fcff92a2700 -1 osd.9 87 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-03T21:37:20.101372+0000 front 2024-05-03T21:37:20.101400+0000 (oldest deadline 2024-05-03T21:37:44.801519+0000) 2024-05-03T21:38:25.971 INFO:journalctl@ceph.osd.9.smithi067.stdout:May 03 21:38:25 smithi067 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-9[62235]: 2024-05-03T21:38:25.677+0000 7fcff92a2700 -1 osd.9 87 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-05-03T21:37:27.702901+0000 front 2024-05-03T21:37:27.702918+0000 (oldest deadline 2024-05-03T21:37:52.402882+0000) 2024-05-03T21:38:25.971 INFO:journalctl@ceph.osd.9.smithi067.stdout:May 03 21:38:25 smithi067 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-9[62235]: 2024-05-03T21:38:25.677+0000 7fcff92a2700 -1 osd.9 87 heartbeat_check: no reply from 172.21.15.138:6804 osd.2 since back 2024-05-03T21:37:34.605024+0000 front 2024-05-03T21:37:34.604986+0000 (oldest deadline 2024-05-03T21:37:59.904623+0000) 2024-05-03T21:38:25.971 INFO:journalctl@ceph.osd.9.smithi067.stdout:May 03 21:38:25 smithi067 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-9[62235]: 2024-05-03T21:38:25.677+0000 7fcff92a2700 -1 osd.9 87 heartbeat_check: no reply from 172.21.15.67:6814 osd.3 since back 2024-05-03T21:37:42.805355+0000 front 2024-05-03T21:37:42.805441+0000 (oldest deadline 2024-05-03T21:38:08.105521+0000) 2024-05-03T21:38:25.971 INFO:journalctl@ceph.osd.9.smithi067.stdout:May 03 21:38:25 smithi067 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-9[62235]: 2024-05-03T21:38:25.677+0000 7fcff92a2700 -1 osd.9 87 heartbeat_check: no reply from 172.21.15.82:6812 osd.4 since back 2024-05-03T21:37:48.106270+0000 front 2024-05-03T21:37:48.106288+0000 (oldest deadline 2024-05-03T21:38:14.005940+0000) 2024-05-03T21:38:25.971 INFO:journalctl@ceph.osd.9.smithi067.stdout:May 03 21:38:25 smithi067 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-9[62235]: 2024-05-03T21:38:25.677+0000 7fcff92a2700 -1 osd.9 87 heartbeat_check: no reply from 172.21.15.138:6812 osd.5 since back 2024-05-03T21:38:00.307891+0000 front 2024-05-03T21:38:00.307916+0000 (oldest deadline 2024-05-03T21:38:25.607766+0000) 2024-05-03T21:38:26.158 INFO:journalctl@ceph.osd.8.smithi138.stdout:May 03 21:38:25 smithi138 podman[69210]: 2024-05-03 21:38:25.876678213 +0000 UTC m=+0.638210258 container died 9981774505a7f2debb56feaca6bb86ff054781a4f37b4fd9614bdcce6a8a29bd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:68e15ae152ef5c805c733e76da8fc1d3dd4ee669d5ba025f457e05725baa24dd, name=ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-8-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240430, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, maintainer=Guillaume Abrioux , GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, RELEASE=wip-rishabh-testing-20240501.193033-9a047fe, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0) 2024-05-03T21:38:26.245 INFO:journalctl@ceph.osd.10.smithi082.stdout:May 03 21:38:25 smithi082 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-10[57676]: 2024-05-03T21:38:25.985+0000 7f790b86c700 -1 osd.10 87 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-03T21:37:18.101114+0000 front 2024-05-03T21:37:18.101101+0000 (oldest deadline 2024-05-03T21:37:42.800953+0000) 2024-05-03T21:38:26.245 INFO:journalctl@ceph.osd.10.smithi082.stdout:May 03 21:38:25 smithi082 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-10[57676]: 2024-05-03T21:38:25.985+0000 7f790b86c700 -1 osd.10 87 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-05-03T21:37:28.303836+0000 front 2024-05-03T21:37:28.303941+0000 (oldest deadline 2024-05-03T21:37:51.804075+0000) 2024-05-03T21:38:26.245 INFO:journalctl@ceph.osd.10.smithi082.stdout:May 03 21:38:25 smithi082 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-10[57676]: 2024-05-03T21:38:25.985+0000 7f790b86c700 -1 osd.10 87 heartbeat_check: no reply from 172.21.15.138:6804 osd.2 since back 2024-05-03T21:37:36.406076+0000 front 2024-05-03T21:37:36.406028+0000 (oldest deadline 2024-05-03T21:38:01.705914+0000) 2024-05-03T21:38:26.245 INFO:journalctl@ceph.osd.10.smithi082.stdout:May 03 21:38:25 smithi082 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-10[57676]: 2024-05-03T21:38:25.985+0000 7f790b86c700 -1 osd.10 87 heartbeat_check: no reply from 172.21.15.67:6814 osd.3 since back 2024-05-03T21:37:41.706645+0000 front 2024-05-03T21:37:41.706615+0000 (oldest deadline 2024-05-03T21:38:08.107018+0000) 2024-05-03T21:38:26.245 INFO:journalctl@ceph.osd.10.smithi082.stdout:May 03 21:38:25 smithi082 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-10[57676]: 2024-05-03T21:38:25.985+0000 7f790b86c700 -1 osd.10 87 heartbeat_check: no reply from 172.21.15.82:6812 osd.4 since back 2024-05-03T21:37:48.107476+0000 front 2024-05-03T21:37:48.107543+0000 (oldest deadline 2024-05-03T21:38:14.007654+0000) 2024-05-03T21:38:26.245 INFO:journalctl@ceph.osd.10.smithi082.stdout:May 03 21:38:25 smithi082 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-10[57676]: 2024-05-03T21:38:25.985+0000 7f790b86c700 -1 osd.10 87 heartbeat_check: no reply from 172.21.15.138:6812 osd.5 since back 2024-05-03T21:38:00.409331+0000 front 2024-05-03T21:38:00.409486+0000 (oldest deadline 2024-05-03T21:38:23.909130+0000) 2024-05-03T21:38:26.416 INFO:journalctl@ceph.osd.8.smithi138.stdout:May 03 21:38:26 smithi138 podman[69236]: 2024-05-03 21:38:26.39907794 +0000 UTC m=+0.517264599 container remove 9981774505a7f2debb56feaca6bb86ff054781a4f37b4fd9614bdcce6a8a29bd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:68e15ae152ef5c805c733e76da8fc1d3dd4ee669d5ba025f457e05725baa24dd, name=ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-8-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, RELEASE=wip-rishabh-testing-20240501.193033-9a047fe, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240430, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_CLEAN=True, CEPH_POINT_RELEASE=) 2024-05-03T21:38:26.417 INFO:journalctl@ceph.osd.11.smithi138.stdout:May 03 21:38:26 smithi138 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-11[58547]: 2024-05-03T21:38:26.273+0000 7fdfdcb59700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-03T21:37:21.054826+0000 front 2024-05-03T21:37:21.054670+0000 (oldest deadline 2024-05-03T21:37:45.154864+0000) 2024-05-03T21:38:26.417 INFO:journalctl@ceph.osd.11.smithi138.stdout:May 03 21:38:26 smithi138 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-11[58547]: 2024-05-03T21:38:26.273+0000 7fdfdcb59700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-05-03T21:37:29.256054+0000 front 2024-05-03T21:37:29.256037+0000 (oldest deadline 2024-05-03T21:37:53.956115+0000) 2024-05-03T21:38:26.417 INFO:journalctl@ceph.osd.11.smithi138.stdout:May 03 21:38:26 smithi138 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-11[58547]: 2024-05-03T21:38:26.273+0000 7fdfdcb59700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.138:6804 osd.2 since back 2024-05-03T21:37:33.956668+0000 front 2024-05-03T21:37:33.956702+0000 (oldest deadline 2024-05-03T21:37:59.856824+0000) 2024-05-03T21:38:26.417 INFO:journalctl@ceph.osd.11.smithi138.stdout:May 03 21:38:26 smithi138 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-11[58547]: 2024-05-03T21:38:26.273+0000 7fdfdcb59700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.67:6814 osd.3 since back 2024-05-03T21:37:43.759130+0000 front 2024-05-03T21:37:43.759044+0000 (oldest deadline 2024-05-03T21:38:09.058895+0000) 2024-05-03T21:38:26.417 INFO:journalctl@ceph.osd.11.smithi138.stdout:May 03 21:38:26 smithi138 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-11[58547]: 2024-05-03T21:38:26.273+0000 7fdfdcb59700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.82:6812 osd.4 since back 2024-05-03T21:37:49.059390+0000 front 2024-05-03T21:37:49.059401+0000 (oldest deadline 2024-05-03T21:38:14.959407+0000) 2024-05-03T21:38:26.418 INFO:journalctl@ceph.osd.11.smithi138.stdout:May 03 21:38:26 smithi138 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-11[58547]: 2024-05-03T21:38:26.273+0000 7fdfdcb59700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.138:6812 osd.5 since back 2024-05-03T21:37:56.660334+0000 front 2024-05-03T21:37:56.660386+0000 (oldest deadline 2024-05-03T21:38:21.960518+0000) 2024-05-03T21:38:26.422 DEBUG:teuthology.orchestra.run.smithi138:> sudo pkill -f 'journalctl -f -n 0 -u ceph-713bc57a-0991-11ef-bc97-c7b262605968@osd.8.service' 2024-05-03T21:38:26.499 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-03T21:38:26.499 INFO:tasks.cephadm.osd.8:Stopped osd.8 2024-05-03T21:38:26.499 INFO:tasks.cephadm.osd.9:Stopping osd.9... 2024-05-03T21:38:26.500 DEBUG:teuthology.orchestra.run.smithi067:> sudo systemctl stop ceph-713bc57a-0991-11ef-bc97-c7b262605968@osd.9 2024-05-03T21:38:26.795 INFO:journalctl@ceph.osd.9.smithi067.stdout:May 03 21:38:26 smithi067 systemd[1]: Stopping Ceph osd.9 for 713bc57a-0991-11ef-bc97-c7b262605968... 2024-05-03T21:38:26.795 INFO:journalctl@ceph.osd.9.smithi067.stdout:May 03 21:38:26 smithi067 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-9[62235]: 2024-05-03T21:38:26.678+0000 7fcff92a2700 -1 osd.9 87 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-03T21:37:20.101372+0000 front 2024-05-03T21:37:20.101400+0000 (oldest deadline 2024-05-03T21:37:44.801519+0000) 2024-05-03T21:38:26.796 INFO:journalctl@ceph.osd.9.smithi067.stdout:May 03 21:38:26 smithi067 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-9[62235]: 2024-05-03T21:38:26.678+0000 7fcff92a2700 -1 osd.9 87 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-05-03T21:37:27.702901+0000 front 2024-05-03T21:37:27.702918+0000 (oldest deadline 2024-05-03T21:37:52.402882+0000) 2024-05-03T21:38:26.796 INFO:journalctl@ceph.osd.9.smithi067.stdout:May 03 21:38:26 smithi067 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-9[62235]: 2024-05-03T21:38:26.678+0000 7fcff92a2700 -1 osd.9 87 heartbeat_check: no reply from 172.21.15.138:6804 osd.2 since back 2024-05-03T21:37:34.605024+0000 front 2024-05-03T21:37:34.604986+0000 (oldest deadline 2024-05-03T21:37:59.904623+0000) 2024-05-03T21:38:26.796 INFO:journalctl@ceph.osd.9.smithi067.stdout:May 03 21:38:26 smithi067 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-9[62235]: 2024-05-03T21:38:26.678+0000 7fcff92a2700 -1 osd.9 87 heartbeat_check: no reply from 172.21.15.67:6814 osd.3 since back 2024-05-03T21:37:42.805355+0000 front 2024-05-03T21:37:42.805441+0000 (oldest deadline 2024-05-03T21:38:08.105521+0000) 2024-05-03T21:38:26.796 INFO:journalctl@ceph.osd.9.smithi067.stdout:May 03 21:38:26 smithi067 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-9[62235]: 2024-05-03T21:38:26.678+0000 7fcff92a2700 -1 osd.9 87 heartbeat_check: no reply from 172.21.15.82:6812 osd.4 since back 2024-05-03T21:37:48.106270+0000 front 2024-05-03T21:37:48.106288+0000 (oldest deadline 2024-05-03T21:38:14.005940+0000) 2024-05-03T21:38:26.796 INFO:journalctl@ceph.osd.9.smithi067.stdout:May 03 21:38:26 smithi067 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-9[62235]: 2024-05-03T21:38:26.678+0000 7fcff92a2700 -1 osd.9 87 heartbeat_check: no reply from 172.21.15.138:6812 osd.5 since back 2024-05-03T21:38:00.307891+0000 front 2024-05-03T21:38:00.307916+0000 (oldest deadline 2024-05-03T21:38:25.607766+0000) 2024-05-03T21:38:26.796 INFO:journalctl@ceph.osd.9.smithi067.stdout:May 03 21:38:26 smithi067 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-9[62235]: 2024-05-03T21:38:26.793+0000 7fd000cb6700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.9 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-03T21:38:26.796 INFO:journalctl@ceph.osd.9.smithi067.stdout:May 03 21:38:26 smithi067 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-9[62235]: 2024-05-03T21:38:26.793+0000 7fd000cb6700 -1 osd.9 87 *** Got signal Terminated *** 2024-05-03T21:38:26.796 INFO:journalctl@ceph.osd.9.smithi067.stdout:May 03 21:38:26 smithi067 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-9[62235]: 2024-05-03T21:38:26.793+0000 7fd000cb6700 -1 osd.9 87 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-05-03T21:38:27.245 INFO:journalctl@ceph.osd.10.smithi082.stdout:May 03 21:38:26 smithi082 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-10[57676]: 2024-05-03T21:38:26.975+0000 7f790b86c700 -1 osd.10 87 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-03T21:37:18.101114+0000 front 2024-05-03T21:37:18.101101+0000 (oldest deadline 2024-05-03T21:37:42.800953+0000) 2024-05-03T21:38:27.245 INFO:journalctl@ceph.osd.10.smithi082.stdout:May 03 21:38:26 smithi082 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-10[57676]: 2024-05-03T21:38:26.975+0000 7f790b86c700 -1 osd.10 87 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-05-03T21:37:28.303836+0000 front 2024-05-03T21:37:28.303941+0000 (oldest deadline 2024-05-03T21:37:51.804075+0000) 2024-05-03T21:38:27.245 INFO:journalctl@ceph.osd.10.smithi082.stdout:May 03 21:38:26 smithi082 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-10[57676]: 2024-05-03T21:38:26.975+0000 7f790b86c700 -1 osd.10 87 heartbeat_check: no reply from 172.21.15.138:6804 osd.2 since back 2024-05-03T21:37:36.406076+0000 front 2024-05-03T21:37:36.406028+0000 (oldest deadline 2024-05-03T21:38:01.705914+0000) 2024-05-03T21:38:27.245 INFO:journalctl@ceph.osd.10.smithi082.stdout:May 03 21:38:26 smithi082 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-10[57676]: 2024-05-03T21:38:26.975+0000 7f790b86c700 -1 osd.10 87 heartbeat_check: no reply from 172.21.15.67:6814 osd.3 since back 2024-05-03T21:37:41.706645+0000 front 2024-05-03T21:37:41.706615+0000 (oldest deadline 2024-05-03T21:38:08.107018+0000) 2024-05-03T21:38:27.246 INFO:journalctl@ceph.osd.10.smithi082.stdout:May 03 21:38:26 smithi082 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-10[57676]: 2024-05-03T21:38:26.975+0000 7f790b86c700 -1 osd.10 87 heartbeat_check: no reply from 172.21.15.82:6812 osd.4 since back 2024-05-03T21:37:48.107476+0000 front 2024-05-03T21:37:48.107543+0000 (oldest deadline 2024-05-03T21:38:14.007654+0000) 2024-05-03T21:38:27.246 INFO:journalctl@ceph.osd.10.smithi082.stdout:May 03 21:38:26 smithi082 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-10[57676]: 2024-05-03T21:38:26.975+0000 7f790b86c700 -1 osd.10 87 heartbeat_check: no reply from 172.21.15.138:6812 osd.5 since back 2024-05-03T21:38:00.409331+0000 front 2024-05-03T21:38:00.409486+0000 (oldest deadline 2024-05-03T21:38:23.909130+0000) 2024-05-03T21:38:27.574 INFO:journalctl@ceph.osd.11.smithi138.stdout:May 03 21:38:27 smithi138 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-11[58547]: 2024-05-03T21:38:27.312+0000 7fdfdcb59700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-03T21:37:21.054826+0000 front 2024-05-03T21:37:21.054670+0000 (oldest deadline 2024-05-03T21:37:45.154864+0000) 2024-05-03T21:38:27.574 INFO:journalctl@ceph.osd.11.smithi138.stdout:May 03 21:38:27 smithi138 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-11[58547]: 2024-05-03T21:38:27.312+0000 7fdfdcb59700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-05-03T21:37:29.256054+0000 front 2024-05-03T21:37:29.256037+0000 (oldest deadline 2024-05-03T21:37:53.956115+0000) 2024-05-03T21:38:27.574 INFO:journalctl@ceph.osd.11.smithi138.stdout:May 03 21:38:27 smithi138 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-11[58547]: 2024-05-03T21:38:27.312+0000 7fdfdcb59700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.138:6804 osd.2 since back 2024-05-03T21:37:33.956668+0000 front 2024-05-03T21:37:33.956702+0000 (oldest deadline 2024-05-03T21:37:59.856824+0000) 2024-05-03T21:38:27.574 INFO:journalctl@ceph.osd.11.smithi138.stdout:May 03 21:38:27 smithi138 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-11[58547]: 2024-05-03T21:38:27.312+0000 7fdfdcb59700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.67:6814 osd.3 since back 2024-05-03T21:37:43.759130+0000 front 2024-05-03T21:37:43.759044+0000 (oldest deadline 2024-05-03T21:38:09.058895+0000) 2024-05-03T21:38:27.575 INFO:journalctl@ceph.osd.11.smithi138.stdout:May 03 21:38:27 smithi138 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-11[58547]: 2024-05-03T21:38:27.312+0000 7fdfdcb59700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.82:6812 osd.4 since back 2024-05-03T21:37:49.059390+0000 front 2024-05-03T21:37:49.059401+0000 (oldest deadline 2024-05-03T21:38:14.959407+0000) 2024-05-03T21:38:27.575 INFO:journalctl@ceph.osd.11.smithi138.stdout:May 03 21:38:27 smithi138 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-11[58547]: 2024-05-03T21:38:27.312+0000 7fdfdcb59700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.138:6812 osd.5 since back 2024-05-03T21:37:56.660334+0000 front 2024-05-03T21:37:56.660386+0000 (oldest deadline 2024-05-03T21:38:21.960518+0000) 2024-05-03T21:38:27.971 INFO:journalctl@ceph.osd.9.smithi067.stdout:May 03 21:38:27 smithi067 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-9[62235]: 2024-05-03T21:38:27.653+0000 7fcff92a2700 -1 osd.9 87 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-03T21:37:20.101372+0000 front 2024-05-03T21:37:20.101400+0000 (oldest deadline 2024-05-03T21:37:44.801519+0000) 2024-05-03T21:38:27.971 INFO:journalctl@ceph.osd.9.smithi067.stdout:May 03 21:38:27 smithi067 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-9[62235]: 2024-05-03T21:38:27.653+0000 7fcff92a2700 -1 osd.9 87 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-05-03T21:37:27.702901+0000 front 2024-05-03T21:37:27.702918+0000 (oldest deadline 2024-05-03T21:37:52.402882+0000) 2024-05-03T21:38:27.972 INFO:journalctl@ceph.osd.9.smithi067.stdout:May 03 21:38:27 smithi067 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-9[62235]: 2024-05-03T21:38:27.653+0000 7fcff92a2700 -1 osd.9 87 heartbeat_check: no reply from 172.21.15.138:6804 osd.2 since back 2024-05-03T21:37:34.605024+0000 front 2024-05-03T21:37:34.604986+0000 (oldest deadline 2024-05-03T21:37:59.904623+0000) 2024-05-03T21:38:27.972 INFO:journalctl@ceph.osd.9.smithi067.stdout:May 03 21:38:27 smithi067 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-9[62235]: 2024-05-03T21:38:27.653+0000 7fcff92a2700 -1 osd.9 87 heartbeat_check: no reply from 172.21.15.67:6814 osd.3 since back 2024-05-03T21:37:42.805355+0000 front 2024-05-03T21:37:42.805441+0000 (oldest deadline 2024-05-03T21:38:08.105521+0000) 2024-05-03T21:38:27.972 INFO:journalctl@ceph.osd.9.smithi067.stdout:May 03 21:38:27 smithi067 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-9[62235]: 2024-05-03T21:38:27.653+0000 7fcff92a2700 -1 osd.9 87 heartbeat_check: no reply from 172.21.15.82:6812 osd.4 since back 2024-05-03T21:37:48.106270+0000 front 2024-05-03T21:37:48.106288+0000 (oldest deadline 2024-05-03T21:38:14.005940+0000) 2024-05-03T21:38:27.972 INFO:journalctl@ceph.osd.9.smithi067.stdout:May 03 21:38:27 smithi067 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-9[62235]: 2024-05-03T21:38:27.653+0000 7fcff92a2700 -1 osd.9 87 heartbeat_check: no reply from 172.21.15.138:6812 osd.5 since back 2024-05-03T21:38:00.307891+0000 front 2024-05-03T21:38:00.307916+0000 (oldest deadline 2024-05-03T21:38:25.607766+0000) 2024-05-03T21:38:28.245 INFO:journalctl@ceph.osd.10.smithi082.stdout:May 03 21:38:27 smithi082 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-10[57676]: 2024-05-03T21:38:27.973+0000 7f790b86c700 -1 osd.10 87 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-03T21:37:18.101114+0000 front 2024-05-03T21:37:18.101101+0000 (oldest deadline 2024-05-03T21:37:42.800953+0000) 2024-05-03T21:38:28.245 INFO:journalctl@ceph.osd.10.smithi082.stdout:May 03 21:38:27 smithi082 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-10[57676]: 2024-05-03T21:38:27.973+0000 7f790b86c700 -1 osd.10 87 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-05-03T21:37:28.303836+0000 front 2024-05-03T21:37:28.303941+0000 (oldest deadline 2024-05-03T21:37:51.804075+0000) 2024-05-03T21:38:28.245 INFO:journalctl@ceph.osd.10.smithi082.stdout:May 03 21:38:27 smithi082 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-10[57676]: 2024-05-03T21:38:27.973+0000 7f790b86c700 -1 osd.10 87 heartbeat_check: no reply from 172.21.15.138:6804 osd.2 since back 2024-05-03T21:37:36.406076+0000 front 2024-05-03T21:37:36.406028+0000 (oldest deadline 2024-05-03T21:38:01.705914+0000) 2024-05-03T21:38:28.246 INFO:journalctl@ceph.osd.10.smithi082.stdout:May 03 21:38:27 smithi082 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-10[57676]: 2024-05-03T21:38:27.973+0000 7f790b86c700 -1 osd.10 87 heartbeat_check: no reply from 172.21.15.67:6814 osd.3 since back 2024-05-03T21:37:41.706645+0000 front 2024-05-03T21:37:41.706615+0000 (oldest deadline 2024-05-03T21:38:08.107018+0000) 2024-05-03T21:38:28.246 INFO:journalctl@ceph.osd.10.smithi082.stdout:May 03 21:38:27 smithi082 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-10[57676]: 2024-05-03T21:38:27.973+0000 7f790b86c700 -1 osd.10 87 heartbeat_check: no reply from 172.21.15.82:6812 osd.4 since back 2024-05-03T21:37:48.107476+0000 front 2024-05-03T21:37:48.107543+0000 (oldest deadline 2024-05-03T21:38:14.007654+0000) 2024-05-03T21:38:28.246 INFO:journalctl@ceph.osd.10.smithi082.stdout:May 03 21:38:27 smithi082 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-10[57676]: 2024-05-03T21:38:27.973+0000 7f790b86c700 -1 osd.10 87 heartbeat_check: no reply from 172.21.15.138:6812 osd.5 since back 2024-05-03T21:38:00.409331+0000 front 2024-05-03T21:38:00.409486+0000 (oldest deadline 2024-05-03T21:38:23.909130+0000) 2024-05-03T21:38:28.824 INFO:journalctl@ceph.osd.11.smithi138.stdout:May 03 21:38:28 smithi138 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-11[58547]: 2024-05-03T21:38:28.341+0000 7fdfdcb59700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-03T21:37:21.054826+0000 front 2024-05-03T21:37:21.054670+0000 (oldest deadline 2024-05-03T21:37:45.154864+0000) 2024-05-03T21:38:28.824 INFO:journalctl@ceph.osd.11.smithi138.stdout:May 03 21:38:28 smithi138 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-11[58547]: 2024-05-03T21:38:28.341+0000 7fdfdcb59700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-05-03T21:37:29.256054+0000 front 2024-05-03T21:37:29.256037+0000 (oldest deadline 2024-05-03T21:37:53.956115+0000) 2024-05-03T21:38:28.825 INFO:journalctl@ceph.osd.11.smithi138.stdout:May 03 21:38:28 smithi138 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-11[58547]: 2024-05-03T21:38:28.341+0000 7fdfdcb59700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.138:6804 osd.2 since back 2024-05-03T21:37:33.956668+0000 front 2024-05-03T21:37:33.956702+0000 (oldest deadline 2024-05-03T21:37:59.856824+0000) 2024-05-03T21:38:28.825 INFO:journalctl@ceph.osd.11.smithi138.stdout:May 03 21:38:28 smithi138 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-11[58547]: 2024-05-03T21:38:28.341+0000 7fdfdcb59700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.67:6814 osd.3 since back 2024-05-03T21:37:43.759130+0000 front 2024-05-03T21:37:43.759044+0000 (oldest deadline 2024-05-03T21:38:09.058895+0000) 2024-05-03T21:38:28.825 INFO:journalctl@ceph.osd.11.smithi138.stdout:May 03 21:38:28 smithi138 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-11[58547]: 2024-05-03T21:38:28.341+0000 7fdfdcb59700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.82:6812 osd.4 since back 2024-05-03T21:37:49.059390+0000 front 2024-05-03T21:37:49.059401+0000 (oldest deadline 2024-05-03T21:38:14.959407+0000) 2024-05-03T21:38:28.825 INFO:journalctl@ceph.osd.11.smithi138.stdout:May 03 21:38:28 smithi138 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-11[58547]: 2024-05-03T21:38:28.341+0000 7fdfdcb59700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.138:6812 osd.5 since back 2024-05-03T21:37:56.660334+0000 front 2024-05-03T21:37:56.660386+0000 (oldest deadline 2024-05-03T21:38:21.960518+0000) 2024-05-03T21:38:28.971 INFO:journalctl@ceph.osd.9.smithi067.stdout:May 03 21:38:28 smithi067 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-9[62235]: 2024-05-03T21:38:28.688+0000 7fcff92a2700 -1 osd.9 87 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-03T21:37:20.101372+0000 front 2024-05-03T21:37:20.101400+0000 (oldest deadline 2024-05-03T21:37:44.801519+0000) 2024-05-03T21:38:28.971 INFO:journalctl@ceph.osd.9.smithi067.stdout:May 03 21:38:28 smithi067 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-9[62235]: 2024-05-03T21:38:28.688+0000 7fcff92a2700 -1 osd.9 87 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-05-03T21:37:27.702901+0000 front 2024-05-03T21:37:27.702918+0000 (oldest deadline 2024-05-03T21:37:52.402882+0000) 2024-05-03T21:38:28.971 INFO:journalctl@ceph.osd.9.smithi067.stdout:May 03 21:38:28 smithi067 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-9[62235]: 2024-05-03T21:38:28.688+0000 7fcff92a2700 -1 osd.9 87 heartbeat_check: no reply from 172.21.15.138:6804 osd.2 since back 2024-05-03T21:37:34.605024+0000 front 2024-05-03T21:37:34.604986+0000 (oldest deadline 2024-05-03T21:37:59.904623+0000) 2024-05-03T21:38:28.971 INFO:journalctl@ceph.osd.9.smithi067.stdout:May 03 21:38:28 smithi067 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-9[62235]: 2024-05-03T21:38:28.688+0000 7fcff92a2700 -1 osd.9 87 heartbeat_check: no reply from 172.21.15.67:6814 osd.3 since back 2024-05-03T21:37:42.805355+0000 front 2024-05-03T21:37:42.805441+0000 (oldest deadline 2024-05-03T21:38:08.105521+0000) 2024-05-03T21:38:28.971 INFO:journalctl@ceph.osd.9.smithi067.stdout:May 03 21:38:28 smithi067 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-9[62235]: 2024-05-03T21:38:28.688+0000 7fcff92a2700 -1 osd.9 87 heartbeat_check: no reply from 172.21.15.82:6812 osd.4 since back 2024-05-03T21:37:48.106270+0000 front 2024-05-03T21:37:48.106288+0000 (oldest deadline 2024-05-03T21:38:14.005940+0000) 2024-05-03T21:38:28.971 INFO:journalctl@ceph.osd.9.smithi067.stdout:May 03 21:38:28 smithi067 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-9[62235]: 2024-05-03T21:38:28.688+0000 7fcff92a2700 -1 osd.9 87 heartbeat_check: no reply from 172.21.15.138:6812 osd.5 since back 2024-05-03T21:38:00.307891+0000 front 2024-05-03T21:38:00.307916+0000 (oldest deadline 2024-05-03T21:38:25.607766+0000) 2024-05-03T21:38:29.495 INFO:journalctl@ceph.osd.10.smithi082.stdout:May 03 21:38:29 smithi082 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-10[57676]: 2024-05-03T21:38:29.005+0000 7f790b86c700 -1 osd.10 87 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-03T21:37:18.101114+0000 front 2024-05-03T21:37:18.101101+0000 (oldest deadline 2024-05-03T21:37:42.800953+0000) 2024-05-03T21:38:29.495 INFO:journalctl@ceph.osd.10.smithi082.stdout:May 03 21:38:29 smithi082 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-10[57676]: 2024-05-03T21:38:29.005+0000 7f790b86c700 -1 osd.10 87 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-05-03T21:37:28.303836+0000 front 2024-05-03T21:37:28.303941+0000 (oldest deadline 2024-05-03T21:37:51.804075+0000) 2024-05-03T21:38:29.495 INFO:journalctl@ceph.osd.10.smithi082.stdout:May 03 21:38:29 smithi082 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-10[57676]: 2024-05-03T21:38:29.005+0000 7f790b86c700 -1 osd.10 87 heartbeat_check: no reply from 172.21.15.138:6804 osd.2 since back 2024-05-03T21:37:36.406076+0000 front 2024-05-03T21:37:36.406028+0000 (oldest deadline 2024-05-03T21:38:01.705914+0000) 2024-05-03T21:38:29.495 INFO:journalctl@ceph.osd.10.smithi082.stdout:May 03 21:38:29 smithi082 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-10[57676]: 2024-05-03T21:38:29.005+0000 7f790b86c700 -1 osd.10 87 heartbeat_check: no reply from 172.21.15.67:6814 osd.3 since back 2024-05-03T21:37:41.706645+0000 front 2024-05-03T21:37:41.706615+0000 (oldest deadline 2024-05-03T21:38:08.107018+0000) 2024-05-03T21:38:29.496 INFO:journalctl@ceph.osd.10.smithi082.stdout:May 03 21:38:29 smithi082 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-10[57676]: 2024-05-03T21:38:29.005+0000 7f790b86c700 -1 osd.10 87 heartbeat_check: no reply from 172.21.15.82:6812 osd.4 since back 2024-05-03T21:37:48.107476+0000 front 2024-05-03T21:37:48.107543+0000 (oldest deadline 2024-05-03T21:38:14.007654+0000) 2024-05-03T21:38:29.496 INFO:journalctl@ceph.osd.10.smithi082.stdout:May 03 21:38:29 smithi082 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-10[57676]: 2024-05-03T21:38:29.005+0000 7f790b86c700 -1 osd.10 87 heartbeat_check: no reply from 172.21.15.138:6812 osd.5 since back 2024-05-03T21:38:00.409331+0000 front 2024-05-03T21:38:00.409486+0000 (oldest deadline 2024-05-03T21:38:23.909130+0000) 2024-05-03T21:38:29.824 INFO:journalctl@ceph.osd.11.smithi138.stdout:May 03 21:38:29 smithi138 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-11[58547]: 2024-05-03T21:38:29.385+0000 7fdfdcb59700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-03T21:37:21.054826+0000 front 2024-05-03T21:37:21.054670+0000 (oldest deadline 2024-05-03T21:37:45.154864+0000) 2024-05-03T21:38:29.824 INFO:journalctl@ceph.osd.11.smithi138.stdout:May 03 21:38:29 smithi138 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-11[58547]: 2024-05-03T21:38:29.385+0000 7fdfdcb59700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-05-03T21:37:29.256054+0000 front 2024-05-03T21:37:29.256037+0000 (oldest deadline 2024-05-03T21:37:53.956115+0000) 2024-05-03T21:38:29.824 INFO:journalctl@ceph.osd.11.smithi138.stdout:May 03 21:38:29 smithi138 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-11[58547]: 2024-05-03T21:38:29.385+0000 7fdfdcb59700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.138:6804 osd.2 since back 2024-05-03T21:37:33.956668+0000 front 2024-05-03T21:37:33.956702+0000 (oldest deadline 2024-05-03T21:37:59.856824+0000) 2024-05-03T21:38:29.824 INFO:journalctl@ceph.osd.11.smithi138.stdout:May 03 21:38:29 smithi138 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-11[58547]: 2024-05-03T21:38:29.385+0000 7fdfdcb59700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.67:6814 osd.3 since back 2024-05-03T21:37:43.759130+0000 front 2024-05-03T21:37:43.759044+0000 (oldest deadline 2024-05-03T21:38:09.058895+0000) 2024-05-03T21:38:29.825 INFO:journalctl@ceph.osd.11.smithi138.stdout:May 03 21:38:29 smithi138 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-11[58547]: 2024-05-03T21:38:29.385+0000 7fdfdcb59700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.82:6812 osd.4 since back 2024-05-03T21:37:49.059390+0000 front 2024-05-03T21:37:49.059401+0000 (oldest deadline 2024-05-03T21:38:14.959407+0000) 2024-05-03T21:38:29.825 INFO:journalctl@ceph.osd.11.smithi138.stdout:May 03 21:38:29 smithi138 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-11[58547]: 2024-05-03T21:38:29.385+0000 7fdfdcb59700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.138:6812 osd.5 since back 2024-05-03T21:37:56.660334+0000 front 2024-05-03T21:37:56.660386+0000 (oldest deadline 2024-05-03T21:38:21.960518+0000) 2024-05-03T21:38:29.971 INFO:journalctl@ceph.osd.9.smithi067.stdout:May 03 21:38:29 smithi067 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-9[62235]: 2024-05-03T21:38:29.640+0000 7fcff92a2700 -1 osd.9 87 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-03T21:37:20.101372+0000 front 2024-05-03T21:37:20.101400+0000 (oldest deadline 2024-05-03T21:37:44.801519+0000) 2024-05-03T21:38:29.971 INFO:journalctl@ceph.osd.9.smithi067.stdout:May 03 21:38:29 smithi067 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-9[62235]: 2024-05-03T21:38:29.640+0000 7fcff92a2700 -1 osd.9 87 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-05-03T21:37:27.702901+0000 front 2024-05-03T21:37:27.702918+0000 (oldest deadline 2024-05-03T21:37:52.402882+0000) 2024-05-03T21:38:29.971 INFO:journalctl@ceph.osd.9.smithi067.stdout:May 03 21:38:29 smithi067 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-9[62235]: 2024-05-03T21:38:29.640+0000 7fcff92a2700 -1 osd.9 87 heartbeat_check: no reply from 172.21.15.138:6804 osd.2 since back 2024-05-03T21:37:34.605024+0000 front 2024-05-03T21:37:34.604986+0000 (oldest deadline 2024-05-03T21:37:59.904623+0000) 2024-05-03T21:38:29.971 INFO:journalctl@ceph.osd.9.smithi067.stdout:May 03 21:38:29 smithi067 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-9[62235]: 2024-05-03T21:38:29.640+0000 7fcff92a2700 -1 osd.9 87 heartbeat_check: no reply from 172.21.15.67:6814 osd.3 since back 2024-05-03T21:37:42.805355+0000 front 2024-05-03T21:37:42.805441+0000 (oldest deadline 2024-05-03T21:38:08.105521+0000) 2024-05-03T21:38:29.971 INFO:journalctl@ceph.osd.9.smithi067.stdout:May 03 21:38:29 smithi067 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-9[62235]: 2024-05-03T21:38:29.640+0000 7fcff92a2700 -1 osd.9 87 heartbeat_check: no reply from 172.21.15.82:6812 osd.4 since back 2024-05-03T21:37:48.106270+0000 front 2024-05-03T21:37:48.106288+0000 (oldest deadline 2024-05-03T21:38:14.005940+0000) 2024-05-03T21:38:29.971 INFO:journalctl@ceph.osd.9.smithi067.stdout:May 03 21:38:29 smithi067 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-9[62235]: 2024-05-03T21:38:29.640+0000 7fcff92a2700 -1 osd.9 87 heartbeat_check: no reply from 172.21.15.138:6812 osd.5 since back 2024-05-03T21:38:00.307891+0000 front 2024-05-03T21:38:00.307916+0000 (oldest deadline 2024-05-03T21:38:25.607766+0000) 2024-05-03T21:38:30.495 INFO:journalctl@ceph.osd.10.smithi082.stdout:May 03 21:38:30 smithi082 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-10[57676]: 2024-05-03T21:38:30.036+0000 7f790b86c700 -1 osd.10 87 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-03T21:37:18.101114+0000 front 2024-05-03T21:37:18.101101+0000 (oldest deadline 2024-05-03T21:37:42.800953+0000) 2024-05-03T21:38:30.495 INFO:journalctl@ceph.osd.10.smithi082.stdout:May 03 21:38:30 smithi082 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-10[57676]: 2024-05-03T21:38:30.036+0000 7f790b86c700 -1 osd.10 87 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-05-03T21:37:28.303836+0000 front 2024-05-03T21:37:28.303941+0000 (oldest deadline 2024-05-03T21:37:51.804075+0000) 2024-05-03T21:38:30.495 INFO:journalctl@ceph.osd.10.smithi082.stdout:May 03 21:38:30 smithi082 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-10[57676]: 2024-05-03T21:38:30.036+0000 7f790b86c700 -1 osd.10 87 heartbeat_check: no reply from 172.21.15.138:6804 osd.2 since back 2024-05-03T21:37:36.406076+0000 front 2024-05-03T21:37:36.406028+0000 (oldest deadline 2024-05-03T21:38:01.705914+0000) 2024-05-03T21:38:30.495 INFO:journalctl@ceph.osd.10.smithi082.stdout:May 03 21:38:30 smithi082 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-10[57676]: 2024-05-03T21:38:30.036+0000 7f790b86c700 -1 osd.10 87 heartbeat_check: no reply from 172.21.15.67:6814 osd.3 since back 2024-05-03T21:37:41.706645+0000 front 2024-05-03T21:37:41.706615+0000 (oldest deadline 2024-05-03T21:38:08.107018+0000) 2024-05-03T21:38:30.495 INFO:journalctl@ceph.osd.10.smithi082.stdout:May 03 21:38:30 smithi082 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-10[57676]: 2024-05-03T21:38:30.036+0000 7f790b86c700 -1 osd.10 87 heartbeat_check: no reply from 172.21.15.82:6812 osd.4 since back 2024-05-03T21:37:48.107476+0000 front 2024-05-03T21:37:48.107543+0000 (oldest deadline 2024-05-03T21:38:14.007654+0000) 2024-05-03T21:38:30.495 INFO:journalctl@ceph.osd.10.smithi082.stdout:May 03 21:38:30 smithi082 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-10[57676]: 2024-05-03T21:38:30.036+0000 7f790b86c700 -1 osd.10 87 heartbeat_check: no reply from 172.21.15.138:6812 osd.5 since back 2024-05-03T21:38:00.409331+0000 front 2024-05-03T21:38:00.409486+0000 (oldest deadline 2024-05-03T21:38:23.909130+0000) 2024-05-03T21:38:30.496 INFO:journalctl@ceph.osd.10.smithi082.stdout:May 03 21:38:30 smithi082 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-10[57676]: 2024-05-03T21:38:30.036+0000 7f790b86c700 -1 osd.10 87 heartbeat_check: no reply from 172.21.15.67:6822 osd.6 since back 2024-05-03T21:38:07.409859+0000 front 2024-05-03T21:38:07.409980+0000 (oldest deadline 2024-05-03T21:38:29.010268+0000) 2024-05-03T21:38:30.824 INFO:journalctl@ceph.osd.11.smithi138.stdout:May 03 21:38:30 smithi138 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-11[58547]: 2024-05-03T21:38:30.366+0000 7fdfdcb59700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-03T21:37:21.054826+0000 front 2024-05-03T21:37:21.054670+0000 (oldest deadline 2024-05-03T21:37:45.154864+0000) 2024-05-03T21:38:30.824 INFO:journalctl@ceph.osd.11.smithi138.stdout:May 03 21:38:30 smithi138 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-11[58547]: 2024-05-03T21:38:30.366+0000 7fdfdcb59700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-05-03T21:37:29.256054+0000 front 2024-05-03T21:37:29.256037+0000 (oldest deadline 2024-05-03T21:37:53.956115+0000) 2024-05-03T21:38:30.824 INFO:journalctl@ceph.osd.11.smithi138.stdout:May 03 21:38:30 smithi138 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-11[58547]: 2024-05-03T21:38:30.366+0000 7fdfdcb59700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.138:6804 osd.2 since back 2024-05-03T21:37:33.956668+0000 front 2024-05-03T21:37:33.956702+0000 (oldest deadline 2024-05-03T21:37:59.856824+0000) 2024-05-03T21:38:30.824 INFO:journalctl@ceph.osd.11.smithi138.stdout:May 03 21:38:30 smithi138 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-11[58547]: 2024-05-03T21:38:30.366+0000 7fdfdcb59700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.67:6814 osd.3 since back 2024-05-03T21:37:43.759130+0000 front 2024-05-03T21:37:43.759044+0000 (oldest deadline 2024-05-03T21:38:09.058895+0000) 2024-05-03T21:38:30.824 INFO:journalctl@ceph.osd.11.smithi138.stdout:May 03 21:38:30 smithi138 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-11[58547]: 2024-05-03T21:38:30.366+0000 7fdfdcb59700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.82:6812 osd.4 since back 2024-05-03T21:37:49.059390+0000 front 2024-05-03T21:37:49.059401+0000 (oldest deadline 2024-05-03T21:38:14.959407+0000) 2024-05-03T21:38:30.825 INFO:journalctl@ceph.osd.11.smithi138.stdout:May 03 21:38:30 smithi138 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-11[58547]: 2024-05-03T21:38:30.366+0000 7fdfdcb59700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.138:6812 osd.5 since back 2024-05-03T21:37:56.660334+0000 front 2024-05-03T21:37:56.660386+0000 (oldest deadline 2024-05-03T21:38:21.960518+0000) 2024-05-03T21:38:30.970 INFO:journalctl@ceph.osd.9.smithi067.stdout:May 03 21:38:30 smithi067 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-9[62235]: 2024-05-03T21:38:30.678+0000 7fcff92a2700 -1 osd.9 87 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-03T21:37:20.101372+0000 front 2024-05-03T21:37:20.101400+0000 (oldest deadline 2024-05-03T21:37:44.801519+0000) 2024-05-03T21:38:30.971 INFO:journalctl@ceph.osd.9.smithi067.stdout:May 03 21:38:30 smithi067 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-9[62235]: 2024-05-03T21:38:30.678+0000 7fcff92a2700 -1 osd.9 87 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-05-03T21:37:27.702901+0000 front 2024-05-03T21:37:27.702918+0000 (oldest deadline 2024-05-03T21:37:52.402882+0000) 2024-05-03T21:38:30.971 INFO:journalctl@ceph.osd.9.smithi067.stdout:May 03 21:38:30 smithi067 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-9[62235]: 2024-05-03T21:38:30.678+0000 7fcff92a2700 -1 osd.9 87 heartbeat_check: no reply from 172.21.15.138:6804 osd.2 since back 2024-05-03T21:37:34.605024+0000 front 2024-05-03T21:37:34.604986+0000 (oldest deadline 2024-05-03T21:37:59.904623+0000) 2024-05-03T21:38:30.971 INFO:journalctl@ceph.osd.9.smithi067.stdout:May 03 21:38:30 smithi067 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-9[62235]: 2024-05-03T21:38:30.678+0000 7fcff92a2700 -1 osd.9 87 heartbeat_check: no reply from 172.21.15.67:6814 osd.3 since back 2024-05-03T21:37:42.805355+0000 front 2024-05-03T21:37:42.805441+0000 (oldest deadline 2024-05-03T21:38:08.105521+0000) 2024-05-03T21:38:30.971 INFO:journalctl@ceph.osd.9.smithi067.stdout:May 03 21:38:30 smithi067 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-9[62235]: 2024-05-03T21:38:30.678+0000 7fcff92a2700 -1 osd.9 87 heartbeat_check: no reply from 172.21.15.82:6812 osd.4 since back 2024-05-03T21:37:48.106270+0000 front 2024-05-03T21:37:48.106288+0000 (oldest deadline 2024-05-03T21:38:14.005940+0000) 2024-05-03T21:38:30.971 INFO:journalctl@ceph.osd.9.smithi067.stdout:May 03 21:38:30 smithi067 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-9[62235]: 2024-05-03T21:38:30.678+0000 7fcff92a2700 -1 osd.9 87 heartbeat_check: no reply from 172.21.15.138:6812 osd.5 since back 2024-05-03T21:38:00.307891+0000 front 2024-05-03T21:38:00.307916+0000 (oldest deadline 2024-05-03T21:38:25.607766+0000) 2024-05-03T21:38:31.495 INFO:journalctl@ceph.osd.10.smithi082.stdout:May 03 21:38:31 smithi082 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-10[57676]: 2024-05-03T21:38:31.053+0000 7f790b86c700 -1 osd.10 87 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-03T21:37:18.101114+0000 front 2024-05-03T21:37:18.101101+0000 (oldest deadline 2024-05-03T21:37:42.800953+0000) 2024-05-03T21:38:31.495 INFO:journalctl@ceph.osd.10.smithi082.stdout:May 03 21:38:31 smithi082 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-10[57676]: 2024-05-03T21:38:31.053+0000 7f790b86c700 -1 osd.10 87 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-05-03T21:37:28.303836+0000 front 2024-05-03T21:37:28.303941+0000 (oldest deadline 2024-05-03T21:37:51.804075+0000) 2024-05-03T21:38:31.495 INFO:journalctl@ceph.osd.10.smithi082.stdout:May 03 21:38:31 smithi082 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-10[57676]: 2024-05-03T21:38:31.053+0000 7f790b86c700 -1 osd.10 87 heartbeat_check: no reply from 172.21.15.138:6804 osd.2 since back 2024-05-03T21:37:36.406076+0000 front 2024-05-03T21:37:36.406028+0000 (oldest deadline 2024-05-03T21:38:01.705914+0000) 2024-05-03T21:38:31.495 INFO:journalctl@ceph.osd.10.smithi082.stdout:May 03 21:38:31 smithi082 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-10[57676]: 2024-05-03T21:38:31.053+0000 7f790b86c700 -1 osd.10 87 heartbeat_check: no reply from 172.21.15.67:6814 osd.3 since back 2024-05-03T21:37:41.706645+0000 front 2024-05-03T21:37:41.706615+0000 (oldest deadline 2024-05-03T21:38:08.107018+0000) 2024-05-03T21:38:31.495 INFO:journalctl@ceph.osd.10.smithi082.stdout:May 03 21:38:31 smithi082 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-10[57676]: 2024-05-03T21:38:31.053+0000 7f790b86c700 -1 osd.10 87 heartbeat_check: no reply from 172.21.15.82:6812 osd.4 since back 2024-05-03T21:37:48.107476+0000 front 2024-05-03T21:37:48.107543+0000 (oldest deadline 2024-05-03T21:38:14.007654+0000) 2024-05-03T21:38:31.495 INFO:journalctl@ceph.osd.10.smithi082.stdout:May 03 21:38:31 smithi082 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-10[57676]: 2024-05-03T21:38:31.053+0000 7f790b86c700 -1 osd.10 87 heartbeat_check: no reply from 172.21.15.138:6812 osd.5 since back 2024-05-03T21:38:00.409331+0000 front 2024-05-03T21:38:00.409486+0000 (oldest deadline 2024-05-03T21:38:23.909130+0000) 2024-05-03T21:38:31.496 INFO:journalctl@ceph.osd.10.smithi082.stdout:May 03 21:38:31 smithi082 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-10[57676]: 2024-05-03T21:38:31.053+0000 7f790b86c700 -1 osd.10 87 heartbeat_check: no reply from 172.21.15.67:6822 osd.6 since back 2024-05-03T21:38:07.409859+0000 front 2024-05-03T21:38:07.409980+0000 (oldest deadline 2024-05-03T21:38:29.010268+0000) 2024-05-03T21:38:31.824 INFO:journalctl@ceph.osd.11.smithi138.stdout:May 03 21:38:31 smithi138 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-11[58547]: 2024-05-03T21:38:31.408+0000 7fdfdcb59700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-03T21:37:21.054826+0000 front 2024-05-03T21:37:21.054670+0000 (oldest deadline 2024-05-03T21:37:45.154864+0000) 2024-05-03T21:38:31.824 INFO:journalctl@ceph.osd.11.smithi138.stdout:May 03 21:38:31 smithi138 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-11[58547]: 2024-05-03T21:38:31.408+0000 7fdfdcb59700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-05-03T21:37:29.256054+0000 front 2024-05-03T21:37:29.256037+0000 (oldest deadline 2024-05-03T21:37:53.956115+0000) 2024-05-03T21:38:31.824 INFO:journalctl@ceph.osd.11.smithi138.stdout:May 03 21:38:31 smithi138 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-11[58547]: 2024-05-03T21:38:31.408+0000 7fdfdcb59700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.138:6804 osd.2 since back 2024-05-03T21:37:33.956668+0000 front 2024-05-03T21:37:33.956702+0000 (oldest deadline 2024-05-03T21:37:59.856824+0000) 2024-05-03T21:38:31.825 INFO:journalctl@ceph.osd.11.smithi138.stdout:May 03 21:38:31 smithi138 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-11[58547]: 2024-05-03T21:38:31.408+0000 7fdfdcb59700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.67:6814 osd.3 since back 2024-05-03T21:37:43.759130+0000 front 2024-05-03T21:37:43.759044+0000 (oldest deadline 2024-05-03T21:38:09.058895+0000) 2024-05-03T21:38:31.825 INFO:journalctl@ceph.osd.11.smithi138.stdout:May 03 21:38:31 smithi138 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-11[58547]: 2024-05-03T21:38:31.409+0000 7fdfdcb59700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.82:6812 osd.4 since back 2024-05-03T21:37:49.059390+0000 front 2024-05-03T21:37:49.059401+0000 (oldest deadline 2024-05-03T21:38:14.959407+0000) 2024-05-03T21:38:31.825 INFO:journalctl@ceph.osd.11.smithi138.stdout:May 03 21:38:31 smithi138 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-11[58547]: 2024-05-03T21:38:31.409+0000 7fdfdcb59700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.138:6812 osd.5 since back 2024-05-03T21:37:56.660334+0000 front 2024-05-03T21:37:56.660386+0000 (oldest deadline 2024-05-03T21:38:21.960518+0000) 2024-05-03T21:38:31.971 INFO:journalctl@ceph.osd.9.smithi067.stdout:May 03 21:38:31 smithi067 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-9[62235]: 2024-05-03T21:38:31.707+0000 7fcff92a2700 -1 osd.9 87 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-03T21:37:20.101372+0000 front 2024-05-03T21:37:20.101400+0000 (oldest deadline 2024-05-03T21:37:44.801519+0000) 2024-05-03T21:38:31.971 INFO:journalctl@ceph.osd.9.smithi067.stdout:May 03 21:38:31 smithi067 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-9[62235]: 2024-05-03T21:38:31.707+0000 7fcff92a2700 -1 osd.9 87 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-05-03T21:37:27.702901+0000 front 2024-05-03T21:37:27.702918+0000 (oldest deadline 2024-05-03T21:37:52.402882+0000) 2024-05-03T21:38:31.971 INFO:journalctl@ceph.osd.9.smithi067.stdout:May 03 21:38:31 smithi067 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-9[62235]: 2024-05-03T21:38:31.707+0000 7fcff92a2700 -1 osd.9 87 heartbeat_check: no reply from 172.21.15.138:6804 osd.2 since back 2024-05-03T21:37:34.605024+0000 front 2024-05-03T21:37:34.604986+0000 (oldest deadline 2024-05-03T21:37:59.904623+0000) 2024-05-03T21:38:31.971 INFO:journalctl@ceph.osd.9.smithi067.stdout:May 03 21:38:31 smithi067 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-9[62235]: 2024-05-03T21:38:31.707+0000 7fcff92a2700 -1 osd.9 87 heartbeat_check: no reply from 172.21.15.67:6814 osd.3 since back 2024-05-03T21:37:42.805355+0000 front 2024-05-03T21:37:42.805441+0000 (oldest deadline 2024-05-03T21:38:08.105521+0000) 2024-05-03T21:38:31.971 INFO:journalctl@ceph.osd.9.smithi067.stdout:May 03 21:38:31 smithi067 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-9[62235]: 2024-05-03T21:38:31.707+0000 7fcff92a2700 -1 osd.9 87 heartbeat_check: no reply from 172.21.15.82:6812 osd.4 since back 2024-05-03T21:37:48.106270+0000 front 2024-05-03T21:37:48.106288+0000 (oldest deadline 2024-05-03T21:38:14.005940+0000) 2024-05-03T21:38:31.971 INFO:journalctl@ceph.osd.9.smithi067.stdout:May 03 21:38:31 smithi067 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-9[62235]: 2024-05-03T21:38:31.707+0000 7fcff92a2700 -1 osd.9 87 heartbeat_check: no reply from 172.21.15.138:6812 osd.5 since back 2024-05-03T21:38:00.307891+0000 front 2024-05-03T21:38:00.307916+0000 (oldest deadline 2024-05-03T21:38:25.607766+0000) 2024-05-03T21:38:31.971 INFO:journalctl@ceph.osd.9.smithi067.stdout:May 03 21:38:31 smithi067 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-9[62235]: 2024-05-03T21:38:31.707+0000 7fcff92a2700 -1 osd.9 87 heartbeat_check: no reply from 172.21.15.67:6822 osd.6 since back 2024-05-03T21:38:07.308589+0000 front 2024-05-03T21:38:07.308627+0000 (oldest deadline 2024-05-03T21:38:31.408817+0000) 2024-05-03T21:38:32.495 INFO:journalctl@ceph.osd.10.smithi082.stdout:May 03 21:38:32 smithi082 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-10[57676]: 2024-05-03T21:38:32.061+0000 7f790b86c700 -1 osd.10 87 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-03T21:37:18.101114+0000 front 2024-05-03T21:37:18.101101+0000 (oldest deadline 2024-05-03T21:37:42.800953+0000) 2024-05-03T21:38:32.495 INFO:journalctl@ceph.osd.10.smithi082.stdout:May 03 21:38:32 smithi082 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-10[57676]: 2024-05-03T21:38:32.061+0000 7f790b86c700 -1 osd.10 87 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-05-03T21:37:28.303836+0000 front 2024-05-03T21:37:28.303941+0000 (oldest deadline 2024-05-03T21:37:51.804075+0000) 2024-05-03T21:38:32.495 INFO:journalctl@ceph.osd.10.smithi082.stdout:May 03 21:38:32 smithi082 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-10[57676]: 2024-05-03T21:38:32.061+0000 7f790b86c700 -1 osd.10 87 heartbeat_check: no reply from 172.21.15.138:6804 osd.2 since back 2024-05-03T21:37:36.406076+0000 front 2024-05-03T21:37:36.406028+0000 (oldest deadline 2024-05-03T21:38:01.705914+0000) 2024-05-03T21:38:32.495 INFO:journalctl@ceph.osd.10.smithi082.stdout:May 03 21:38:32 smithi082 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-10[57676]: 2024-05-03T21:38:32.061+0000 7f790b86c700 -1 osd.10 87 heartbeat_check: no reply from 172.21.15.67:6814 osd.3 since back 2024-05-03T21:37:41.706645+0000 front 2024-05-03T21:37:41.706615+0000 (oldest deadline 2024-05-03T21:38:08.107018+0000) 2024-05-03T21:38:32.495 INFO:journalctl@ceph.osd.10.smithi082.stdout:May 03 21:38:32 smithi082 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-10[57676]: 2024-05-03T21:38:32.061+0000 7f790b86c700 -1 osd.10 87 heartbeat_check: no reply from 172.21.15.82:6812 osd.4 since back 2024-05-03T21:37:48.107476+0000 front 2024-05-03T21:37:48.107543+0000 (oldest deadline 2024-05-03T21:38:14.007654+0000) 2024-05-03T21:38:32.495 INFO:journalctl@ceph.osd.10.smithi082.stdout:May 03 21:38:32 smithi082 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-10[57676]: 2024-05-03T21:38:32.061+0000 7f790b86c700 -1 osd.10 87 heartbeat_check: no reply from 172.21.15.138:6812 osd.5 since back 2024-05-03T21:38:00.409331+0000 front 2024-05-03T21:38:00.409486+0000 (oldest deadline 2024-05-03T21:38:23.909130+0000) 2024-05-03T21:38:32.496 INFO:journalctl@ceph.osd.10.smithi082.stdout:May 03 21:38:32 smithi082 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-10[57676]: 2024-05-03T21:38:32.061+0000 7f790b86c700 -1 osd.10 87 heartbeat_check: no reply from 172.21.15.67:6822 osd.6 since back 2024-05-03T21:38:07.409859+0000 front 2024-05-03T21:38:07.409980+0000 (oldest deadline 2024-05-03T21:38:29.010268+0000) 2024-05-03T21:38:32.554 INFO:journalctl@ceph.osd.9.smithi067.stdout:May 03 21:38:32 smithi067 podman[88226]: 2024-05-03 21:38:32.27936173 +0000 UTC m=+5.578873060 container died 90d64d93ab3e4084d7f45af4de36856dfd2a256dabb26c0a34ae4cf30df27d05 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:68e15ae152ef5c805c733e76da8fc1d3dd4ee669d5ba025f457e05725baa24dd, name=ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-9, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, RELEASE=wip-rishabh-testing-20240501.193033-9a047fe, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240430, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, ceph=True, org.label-schema.schema-version=1.0) 2024-05-03T21:38:32.824 INFO:journalctl@ceph.osd.11.smithi138.stdout:May 03 21:38:32 smithi138 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-11[58547]: 2024-05-03T21:38:32.426+0000 7fdfdcb59700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-03T21:37:21.054826+0000 front 2024-05-03T21:37:21.054670+0000 (oldest deadline 2024-05-03T21:37:45.154864+0000) 2024-05-03T21:38:32.824 INFO:journalctl@ceph.osd.11.smithi138.stdout:May 03 21:38:32 smithi138 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-11[58547]: 2024-05-03T21:38:32.426+0000 7fdfdcb59700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-05-03T21:37:29.256054+0000 front 2024-05-03T21:37:29.256037+0000 (oldest deadline 2024-05-03T21:37:53.956115+0000) 2024-05-03T21:38:32.824 INFO:journalctl@ceph.osd.11.smithi138.stdout:May 03 21:38:32 smithi138 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-11[58547]: 2024-05-03T21:38:32.426+0000 7fdfdcb59700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.138:6804 osd.2 since back 2024-05-03T21:37:33.956668+0000 front 2024-05-03T21:37:33.956702+0000 (oldest deadline 2024-05-03T21:37:59.856824+0000) 2024-05-03T21:38:32.824 INFO:journalctl@ceph.osd.11.smithi138.stdout:May 03 21:38:32 smithi138 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-11[58547]: 2024-05-03T21:38:32.426+0000 7fdfdcb59700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.67:6814 osd.3 since back 2024-05-03T21:37:43.759130+0000 front 2024-05-03T21:37:43.759044+0000 (oldest deadline 2024-05-03T21:38:09.058895+0000) 2024-05-03T21:38:32.824 INFO:journalctl@ceph.osd.11.smithi138.stdout:May 03 21:38:32 smithi138 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-11[58547]: 2024-05-03T21:38:32.426+0000 7fdfdcb59700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.82:6812 osd.4 since back 2024-05-03T21:37:49.059390+0000 front 2024-05-03T21:37:49.059401+0000 (oldest deadline 2024-05-03T21:38:14.959407+0000) 2024-05-03T21:38:32.825 INFO:journalctl@ceph.osd.11.smithi138.stdout:May 03 21:38:32 smithi138 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-11[58547]: 2024-05-03T21:38:32.426+0000 7fdfdcb59700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.138:6812 osd.5 since back 2024-05-03T21:37:56.660334+0000 front 2024-05-03T21:37:56.660386+0000 (oldest deadline 2024-05-03T21:38:21.960518+0000) 2024-05-03T21:38:32.945 INFO:journalctl@ceph.osd.9.smithi067.stdout:May 03 21:38:32 smithi067 podman[88226]: 2024-05-03 21:38:32.651683895 +0000 UTC m=+5.951195232 container cleanup 90d64d93ab3e4084d7f45af4de36856dfd2a256dabb26c0a34ae4cf30df27d05 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:68e15ae152ef5c805c733e76da8fc1d3dd4ee669d5ba025f457e05725baa24dd, name=ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-9, org.label-schema.license=GPLv2, org.label-schema.build-date=20240430, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, RELEASE=wip-rishabh-testing-20240501.193033-9a047fe, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, ceph=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b) 2024-05-03T21:38:32.945 INFO:journalctl@ceph.osd.9.smithi067.stdout:May 03 21:38:32 smithi067 bash[88226]: ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-9 2024-05-03T21:38:33.220 INFO:journalctl@ceph.osd.9.smithi067.stdout:May 03 21:38:32 smithi067 podman[88243]: 2024-05-03 21:38:32.944530978 +0000 UTC m=+0.659139164 container remove 90d64d93ab3e4084d7f45af4de36856dfd2a256dabb26c0a34ae4cf30df27d05 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:68e15ae152ef5c805c733e76da8fc1d3dd4ee669d5ba025f457e05725baa24dd, name=ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-9, RELEASE=wip-rishabh-testing-20240501.193033-9a047fe, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, ceph=True, org.label-schema.build-date=20240430) 2024-05-03T21:38:33.495 INFO:journalctl@ceph.osd.10.smithi082.stdout:May 03 21:38:33 smithi082 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-10[57676]: 2024-05-03T21:38:33.110+0000 7f790b86c700 -1 osd.10 87 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-03T21:37:18.101114+0000 front 2024-05-03T21:37:18.101101+0000 (oldest deadline 2024-05-03T21:37:42.800953+0000) 2024-05-03T21:38:33.495 INFO:journalctl@ceph.osd.10.smithi082.stdout:May 03 21:38:33 smithi082 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-10[57676]: 2024-05-03T21:38:33.110+0000 7f790b86c700 -1 osd.10 87 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-05-03T21:37:28.303836+0000 front 2024-05-03T21:37:28.303941+0000 (oldest deadline 2024-05-03T21:37:51.804075+0000) 2024-05-03T21:38:33.495 INFO:journalctl@ceph.osd.10.smithi082.stdout:May 03 21:38:33 smithi082 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-10[57676]: 2024-05-03T21:38:33.110+0000 7f790b86c700 -1 osd.10 87 heartbeat_check: no reply from 172.21.15.138:6804 osd.2 since back 2024-05-03T21:37:36.406076+0000 front 2024-05-03T21:37:36.406028+0000 (oldest deadline 2024-05-03T21:38:01.705914+0000) 2024-05-03T21:38:33.495 INFO:journalctl@ceph.osd.10.smithi082.stdout:May 03 21:38:33 smithi082 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-10[57676]: 2024-05-03T21:38:33.110+0000 7f790b86c700 -1 osd.10 87 heartbeat_check: no reply from 172.21.15.67:6814 osd.3 since back 2024-05-03T21:37:41.706645+0000 front 2024-05-03T21:37:41.706615+0000 (oldest deadline 2024-05-03T21:38:08.107018+0000) 2024-05-03T21:38:33.496 INFO:journalctl@ceph.osd.10.smithi082.stdout:May 03 21:38:33 smithi082 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-10[57676]: 2024-05-03T21:38:33.110+0000 7f790b86c700 -1 osd.10 87 heartbeat_check: no reply from 172.21.15.82:6812 osd.4 since back 2024-05-03T21:37:48.107476+0000 front 2024-05-03T21:37:48.107543+0000 (oldest deadline 2024-05-03T21:38:14.007654+0000) 2024-05-03T21:38:33.496 INFO:journalctl@ceph.osd.10.smithi082.stdout:May 03 21:38:33 smithi082 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-10[57676]: 2024-05-03T21:38:33.110+0000 7f790b86c700 -1 osd.10 87 heartbeat_check: no reply from 172.21.15.138:6812 osd.5 since back 2024-05-03T21:38:00.409331+0000 front 2024-05-03T21:38:00.409486+0000 (oldest deadline 2024-05-03T21:38:23.909130+0000) 2024-05-03T21:38:33.496 INFO:journalctl@ceph.osd.10.smithi082.stdout:May 03 21:38:33 smithi082 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-10[57676]: 2024-05-03T21:38:33.110+0000 7f790b86c700 -1 osd.10 87 heartbeat_check: no reply from 172.21.15.67:6822 osd.6 since back 2024-05-03T21:38:07.409859+0000 front 2024-05-03T21:38:07.409980+0000 (oldest deadline 2024-05-03T21:38:29.010268+0000) 2024-05-03T21:38:33.535 INFO:journalctl@ceph.osd.9.smithi067.stdout:May 03 21:38:33 smithi067 podman[88332]: 2024-05-03 21:38:33.185152397 +0000 UTC m=+0.018617868 image pull ee4d846202496ba6096593e8e8becc0700a5eb7ebd95536031187f5f337d4012 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:68e15ae152ef5c805c733e76da8fc1d3dd4ee669d5ba025f457e05725baa24dd 2024-05-03T21:38:33.535 INFO:journalctl@ceph.osd.9.smithi067.stdout:May 03 21:38:33 smithi067 podman[88332]: 2024-05-03 21:38:33.312077818 +0000 UTC m=+0.145543294 container create ac6979a16978efd6b7ca6e28b82025a4a52f6357528712852a2b4fe3b64fc07c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:68e15ae152ef5c805c733e76da8fc1d3dd4ee669d5ba025f457e05725baa24dd, name=ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-9-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240430, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_CLEAN=True, ceph=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , RELEASE=wip-rishabh-testing-20240501.193033-9a047fe) 2024-05-03T21:38:33.535 INFO:journalctl@ceph.osd.9.smithi067.stdout:May 03 21:38:33 smithi067 podman[88332]: 2024-05-03 21:38:33.530880781 +0000 UTC m=+0.364346252 container init ac6979a16978efd6b7ca6e28b82025a4a52f6357528712852a2b4fe3b64fc07c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:68e15ae152ef5c805c733e76da8fc1d3dd4ee669d5ba025f457e05725baa24dd, name=ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-9-deactivate, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, RELEASE=wip-rishabh-testing-20240501.193033-9a047fe, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.build-date=20240430, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2) 2024-05-03T21:38:33.535 INFO:journalctl@ceph.osd.9.smithi067.stdout:May 03 21:38:33 smithi067 podman[88332]: 2024-05-03 21:38:33.534338872 +0000 UTC m=+0.367804343 container start ac6979a16978efd6b7ca6e28b82025a4a52f6357528712852a2b4fe3b64fc07c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:68e15ae152ef5c805c733e76da8fc1d3dd4ee669d5ba025f457e05725baa24dd, name=ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-9-deactivate, RELEASE=wip-rishabh-testing-20240501.193033-9a047fe, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240430, io.buildah.version=1.33.5, ceph=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-05-03T21:38:33.824 INFO:journalctl@ceph.osd.11.smithi138.stdout:May 03 21:38:33 smithi138 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-11[58547]: 2024-05-03T21:38:33.474+0000 7fdfdcb59700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-03T21:37:21.054826+0000 front 2024-05-03T21:37:21.054670+0000 (oldest deadline 2024-05-03T21:37:45.154864+0000) 2024-05-03T21:38:33.824 INFO:journalctl@ceph.osd.11.smithi138.stdout:May 03 21:38:33 smithi138 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-11[58547]: 2024-05-03T21:38:33.474+0000 7fdfdcb59700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-05-03T21:37:29.256054+0000 front 2024-05-03T21:37:29.256037+0000 (oldest deadline 2024-05-03T21:37:53.956115+0000) 2024-05-03T21:38:33.824 INFO:journalctl@ceph.osd.11.smithi138.stdout:May 03 21:38:33 smithi138 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-11[58547]: 2024-05-03T21:38:33.474+0000 7fdfdcb59700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.138:6804 osd.2 since back 2024-05-03T21:37:33.956668+0000 front 2024-05-03T21:37:33.956702+0000 (oldest deadline 2024-05-03T21:37:59.856824+0000) 2024-05-03T21:38:33.824 INFO:journalctl@ceph.osd.11.smithi138.stdout:May 03 21:38:33 smithi138 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-11[58547]: 2024-05-03T21:38:33.474+0000 7fdfdcb59700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.67:6814 osd.3 since back 2024-05-03T21:37:43.759130+0000 front 2024-05-03T21:37:43.759044+0000 (oldest deadline 2024-05-03T21:38:09.058895+0000) 2024-05-03T21:38:33.824 INFO:journalctl@ceph.osd.11.smithi138.stdout:May 03 21:38:33 smithi138 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-11[58547]: 2024-05-03T21:38:33.474+0000 7fdfdcb59700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.82:6812 osd.4 since back 2024-05-03T21:37:49.059390+0000 front 2024-05-03T21:37:49.059401+0000 (oldest deadline 2024-05-03T21:38:14.959407+0000) 2024-05-03T21:38:33.824 INFO:journalctl@ceph.osd.11.smithi138.stdout:May 03 21:38:33 smithi138 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-11[58547]: 2024-05-03T21:38:33.474+0000 7fdfdcb59700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.138:6812 osd.5 since back 2024-05-03T21:37:56.660334+0000 front 2024-05-03T21:37:56.660386+0000 (oldest deadline 2024-05-03T21:38:21.960518+0000) 2024-05-03T21:38:33.834 INFO:journalctl@ceph.osd.9.smithi067.stdout:May 03 21:38:33 smithi067 podman[88332]: 2024-05-03 21:38:33.639285676 +0000 UTC m=+0.472751147 container attach ac6979a16978efd6b7ca6e28b82025a4a52f6357528712852a2b4fe3b64fc07c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:68e15ae152ef5c805c733e76da8fc1d3dd4ee669d5ba025f457e05725baa24dd, name=ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-9-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240430, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, RELEASE=wip-rishabh-testing-20240501.193033-9a047fe, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux ) 2024-05-03T21:38:34.118 INFO:journalctl@ceph.osd.9.smithi067.stdout:May 03 21:38:33 smithi067 podman[88332]: 2024-05-03 21:38:33.833832207 +0000 UTC m=+0.667297690 container died ac6979a16978efd6b7ca6e28b82025a4a52f6357528712852a2b4fe3b64fc07c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:68e15ae152ef5c805c733e76da8fc1d3dd4ee669d5ba025f457e05725baa24dd, name=ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-9-deactivate, org.label-schema.schema-version=1.0, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-rishabh-testing-20240501.193033-9a047fe, org.label-schema.build-date=20240430, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b) 2024-05-03T21:38:34.383 INFO:journalctl@ceph.osd.9.smithi067.stdout:May 03 21:38:34 smithi067 podman[88359]: 2024-05-03 21:38:34.359509728 +0000 UTC m=+0.520310336 container remove ac6979a16978efd6b7ca6e28b82025a4a52f6357528712852a2b4fe3b64fc07c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:68e15ae152ef5c805c733e76da8fc1d3dd4ee669d5ba025f457e05725baa24dd, name=ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-9-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-rishabh-testing-20240501.193033-9a047fe, org.label-schema.build-date=20240430, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, ceph=True, org.label-schema.vendor=CentOS, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.license=GPLv2) 2024-05-03T21:38:34.386 DEBUG:teuthology.orchestra.run.smithi067:> sudo pkill -f 'journalctl -f -n 0 -u ceph-713bc57a-0991-11ef-bc97-c7b262605968@osd.9.service' 2024-05-03T21:38:34.425 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-03T21:38:34.425 INFO:tasks.cephadm.osd.9:Stopped osd.9 2024-05-03T21:38:34.426 INFO:tasks.cephadm.osd.10:Stopping osd.10... 2024-05-03T21:38:34.426 DEBUG:teuthology.orchestra.run.smithi082:> sudo systemctl stop ceph-713bc57a-0991-11ef-bc97-c7b262605968@osd.10 2024-05-03T21:38:34.451 INFO:journalctl@ceph.osd.10.smithi082.stdout:May 03 21:38:34 smithi082 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-10[57676]: 2024-05-03T21:38:34.074+0000 7f790b86c700 -1 osd.10 87 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-03T21:37:18.101114+0000 front 2024-05-03T21:37:18.101101+0000 (oldest deadline 2024-05-03T21:37:42.800953+0000) 2024-05-03T21:38:34.452 INFO:journalctl@ceph.osd.10.smithi082.stdout:May 03 21:38:34 smithi082 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-10[57676]: 2024-05-03T21:38:34.074+0000 7f790b86c700 -1 osd.10 87 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-05-03T21:37:28.303836+0000 front 2024-05-03T21:37:28.303941+0000 (oldest deadline 2024-05-03T21:37:51.804075+0000) 2024-05-03T21:38:34.452 INFO:journalctl@ceph.osd.10.smithi082.stdout:May 03 21:38:34 smithi082 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-10[57676]: 2024-05-03T21:38:34.074+0000 7f790b86c700 -1 osd.10 87 heartbeat_check: no reply from 172.21.15.138:6804 osd.2 since back 2024-05-03T21:37:36.406076+0000 front 2024-05-03T21:37:36.406028+0000 (oldest deadline 2024-05-03T21:38:01.705914+0000) 2024-05-03T21:38:34.452 INFO:journalctl@ceph.osd.10.smithi082.stdout:May 03 21:38:34 smithi082 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-10[57676]: 2024-05-03T21:38:34.074+0000 7f790b86c700 -1 osd.10 87 heartbeat_check: no reply from 172.21.15.67:6814 osd.3 since back 2024-05-03T21:37:41.706645+0000 front 2024-05-03T21:37:41.706615+0000 (oldest deadline 2024-05-03T21:38:08.107018+0000) 2024-05-03T21:38:34.452 INFO:journalctl@ceph.osd.10.smithi082.stdout:May 03 21:38:34 smithi082 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-10[57676]: 2024-05-03T21:38:34.074+0000 7f790b86c700 -1 osd.10 87 heartbeat_check: no reply from 172.21.15.82:6812 osd.4 since back 2024-05-03T21:37:48.107476+0000 front 2024-05-03T21:37:48.107543+0000 (oldest deadline 2024-05-03T21:38:14.007654+0000) 2024-05-03T21:38:34.452 INFO:journalctl@ceph.osd.10.smithi082.stdout:May 03 21:38:34 smithi082 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-10[57676]: 2024-05-03T21:38:34.074+0000 7f790b86c700 -1 osd.10 87 heartbeat_check: no reply from 172.21.15.138:6812 osd.5 since back 2024-05-03T21:38:00.409331+0000 front 2024-05-03T21:38:00.409486+0000 (oldest deadline 2024-05-03T21:38:23.909130+0000) 2024-05-03T21:38:34.452 INFO:journalctl@ceph.osd.10.smithi082.stdout:May 03 21:38:34 smithi082 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-10[57676]: 2024-05-03T21:38:34.074+0000 7f790b86c700 -1 osd.10 87 heartbeat_check: no reply from 172.21.15.67:6822 osd.6 since back 2024-05-03T21:38:07.409859+0000 front 2024-05-03T21:38:07.409980+0000 (oldest deadline 2024-05-03T21:38:29.010268+0000) 2024-05-03T21:38:34.721 INFO:journalctl@ceph.osd.10.smithi082.stdout:May 03 21:38:34 smithi082 systemd[1]: Stopping Ceph osd.10 for 713bc57a-0991-11ef-bc97-c7b262605968... 2024-05-03T21:38:34.824 INFO:journalctl@ceph.osd.11.smithi138.stdout:May 03 21:38:34 smithi138 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-11[58547]: 2024-05-03T21:38:34.497+0000 7fdfdcb59700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-03T21:37:21.054826+0000 front 2024-05-03T21:37:21.054670+0000 (oldest deadline 2024-05-03T21:37:45.154864+0000) 2024-05-03T21:38:34.824 INFO:journalctl@ceph.osd.11.smithi138.stdout:May 03 21:38:34 smithi138 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-11[58547]: 2024-05-03T21:38:34.498+0000 7fdfdcb59700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-05-03T21:37:29.256054+0000 front 2024-05-03T21:37:29.256037+0000 (oldest deadline 2024-05-03T21:37:53.956115+0000) 2024-05-03T21:38:34.824 INFO:journalctl@ceph.osd.11.smithi138.stdout:May 03 21:38:34 smithi138 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-11[58547]: 2024-05-03T21:38:34.498+0000 7fdfdcb59700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.138:6804 osd.2 since back 2024-05-03T21:37:33.956668+0000 front 2024-05-03T21:37:33.956702+0000 (oldest deadline 2024-05-03T21:37:59.856824+0000) 2024-05-03T21:38:34.824 INFO:journalctl@ceph.osd.11.smithi138.stdout:May 03 21:38:34 smithi138 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-11[58547]: 2024-05-03T21:38:34.498+0000 7fdfdcb59700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.67:6814 osd.3 since back 2024-05-03T21:37:43.759130+0000 front 2024-05-03T21:37:43.759044+0000 (oldest deadline 2024-05-03T21:38:09.058895+0000) 2024-05-03T21:38:34.825 INFO:journalctl@ceph.osd.11.smithi138.stdout:May 03 21:38:34 smithi138 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-11[58547]: 2024-05-03T21:38:34.498+0000 7fdfdcb59700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.82:6812 osd.4 since back 2024-05-03T21:37:49.059390+0000 front 2024-05-03T21:37:49.059401+0000 (oldest deadline 2024-05-03T21:38:14.959407+0000) 2024-05-03T21:38:34.825 INFO:journalctl@ceph.osd.11.smithi138.stdout:May 03 21:38:34 smithi138 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-11[58547]: 2024-05-03T21:38:34.498+0000 7fdfdcb59700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.138:6812 osd.5 since back 2024-05-03T21:37:56.660334+0000 front 2024-05-03T21:37:56.660386+0000 (oldest deadline 2024-05-03T21:38:21.960518+0000) 2024-05-03T21:38:34.995 INFO:journalctl@ceph.osd.10.smithi082.stdout:May 03 21:38:34 smithi082 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-10[57676]: 2024-05-03T21:38:34.720+0000 7f790f278700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.10 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-03T21:38:34.995 INFO:journalctl@ceph.osd.10.smithi082.stdout:May 03 21:38:34 smithi082 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-10[57676]: 2024-05-03T21:38:34.720+0000 7f790f278700 -1 osd.10 87 *** Got signal Terminated *** 2024-05-03T21:38:34.995 INFO:journalctl@ceph.osd.10.smithi082.stdout:May 03 21:38:34 smithi082 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-10[57676]: 2024-05-03T21:38:34.720+0000 7f790f278700 -1 osd.10 87 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-05-03T21:38:35.495 INFO:journalctl@ceph.osd.10.smithi082.stdout:May 03 21:38:35 smithi082 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-10[57676]: 2024-05-03T21:38:35.109+0000 7f790b86c700 -1 osd.10 87 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-03T21:37:18.101114+0000 front 2024-05-03T21:37:18.101101+0000 (oldest deadline 2024-05-03T21:37:42.800953+0000) 2024-05-03T21:38:35.495 INFO:journalctl@ceph.osd.10.smithi082.stdout:May 03 21:38:35 smithi082 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-10[57676]: 2024-05-03T21:38:35.109+0000 7f790b86c700 -1 osd.10 87 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-05-03T21:37:28.303836+0000 front 2024-05-03T21:37:28.303941+0000 (oldest deadline 2024-05-03T21:37:51.804075+0000) 2024-05-03T21:38:35.495 INFO:journalctl@ceph.osd.10.smithi082.stdout:May 03 21:38:35 smithi082 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-10[57676]: 2024-05-03T21:38:35.109+0000 7f790b86c700 -1 osd.10 87 heartbeat_check: no reply from 172.21.15.138:6804 osd.2 since back 2024-05-03T21:37:36.406076+0000 front 2024-05-03T21:37:36.406028+0000 (oldest deadline 2024-05-03T21:38:01.705914+0000) 2024-05-03T21:38:35.495 INFO:journalctl@ceph.osd.10.smithi082.stdout:May 03 21:38:35 smithi082 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-10[57676]: 2024-05-03T21:38:35.109+0000 7f790b86c700 -1 osd.10 87 heartbeat_check: no reply from 172.21.15.67:6814 osd.3 since back 2024-05-03T21:37:41.706645+0000 front 2024-05-03T21:37:41.706615+0000 (oldest deadline 2024-05-03T21:38:08.107018+0000) 2024-05-03T21:38:35.495 INFO:journalctl@ceph.osd.10.smithi082.stdout:May 03 21:38:35 smithi082 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-10[57676]: 2024-05-03T21:38:35.109+0000 7f790b86c700 -1 osd.10 87 heartbeat_check: no reply from 172.21.15.82:6812 osd.4 since back 2024-05-03T21:37:48.107476+0000 front 2024-05-03T21:37:48.107543+0000 (oldest deadline 2024-05-03T21:38:14.007654+0000) 2024-05-03T21:38:35.496 INFO:journalctl@ceph.osd.10.smithi082.stdout:May 03 21:38:35 smithi082 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-10[57676]: 2024-05-03T21:38:35.109+0000 7f790b86c700 -1 osd.10 87 heartbeat_check: no reply from 172.21.15.138:6812 osd.5 since back 2024-05-03T21:38:00.409331+0000 front 2024-05-03T21:38:00.409486+0000 (oldest deadline 2024-05-03T21:38:23.909130+0000) 2024-05-03T21:38:35.496 INFO:journalctl@ceph.osd.10.smithi082.stdout:May 03 21:38:35 smithi082 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-10[57676]: 2024-05-03T21:38:35.109+0000 7f790b86c700 -1 osd.10 87 heartbeat_check: no reply from 172.21.15.67:6822 osd.6 since back 2024-05-03T21:38:07.409859+0000 front 2024-05-03T21:38:07.409980+0000 (oldest deadline 2024-05-03T21:38:29.010268+0000) 2024-05-03T21:38:35.824 INFO:journalctl@ceph.osd.11.smithi138.stdout:May 03 21:38:35 smithi138 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-11[58547]: 2024-05-03T21:38:35.543+0000 7fdfdcb59700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-03T21:37:21.054826+0000 front 2024-05-03T21:37:21.054670+0000 (oldest deadline 2024-05-03T21:37:45.154864+0000) 2024-05-03T21:38:35.824 INFO:journalctl@ceph.osd.11.smithi138.stdout:May 03 21:38:35 smithi138 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-11[58547]: 2024-05-03T21:38:35.543+0000 7fdfdcb59700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-05-03T21:37:29.256054+0000 front 2024-05-03T21:37:29.256037+0000 (oldest deadline 2024-05-03T21:37:53.956115+0000) 2024-05-03T21:38:35.824 INFO:journalctl@ceph.osd.11.smithi138.stdout:May 03 21:38:35 smithi138 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-11[58547]: 2024-05-03T21:38:35.543+0000 7fdfdcb59700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.138:6804 osd.2 since back 2024-05-03T21:37:33.956668+0000 front 2024-05-03T21:37:33.956702+0000 (oldest deadline 2024-05-03T21:37:59.856824+0000) 2024-05-03T21:38:35.824 INFO:journalctl@ceph.osd.11.smithi138.stdout:May 03 21:38:35 smithi138 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-11[58547]: 2024-05-03T21:38:35.543+0000 7fdfdcb59700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.67:6814 osd.3 since back 2024-05-03T21:37:43.759130+0000 front 2024-05-03T21:37:43.759044+0000 (oldest deadline 2024-05-03T21:38:09.058895+0000) 2024-05-03T21:38:35.825 INFO:journalctl@ceph.osd.11.smithi138.stdout:May 03 21:38:35 smithi138 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-11[58547]: 2024-05-03T21:38:35.543+0000 7fdfdcb59700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.82:6812 osd.4 since back 2024-05-03T21:37:49.059390+0000 front 2024-05-03T21:37:49.059401+0000 (oldest deadline 2024-05-03T21:38:14.959407+0000) 2024-05-03T21:38:35.825 INFO:journalctl@ceph.osd.11.smithi138.stdout:May 03 21:38:35 smithi138 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-11[58547]: 2024-05-03T21:38:35.543+0000 7fdfdcb59700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.138:6812 osd.5 since back 2024-05-03T21:37:56.660334+0000 front 2024-05-03T21:37:56.660386+0000 (oldest deadline 2024-05-03T21:38:21.960518+0000) 2024-05-03T21:38:36.495 INFO:journalctl@ceph.osd.10.smithi082.stdout:May 03 21:38:36 smithi082 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-10[57676]: 2024-05-03T21:38:36.110+0000 7f790b86c700 -1 osd.10 87 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-03T21:37:18.101114+0000 front 2024-05-03T21:37:18.101101+0000 (oldest deadline 2024-05-03T21:37:42.800953+0000) 2024-05-03T21:38:36.495 INFO:journalctl@ceph.osd.10.smithi082.stdout:May 03 21:38:36 smithi082 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-10[57676]: 2024-05-03T21:38:36.110+0000 7f790b86c700 -1 osd.10 87 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-05-03T21:37:28.303836+0000 front 2024-05-03T21:37:28.303941+0000 (oldest deadline 2024-05-03T21:37:51.804075+0000) 2024-05-03T21:38:36.496 INFO:journalctl@ceph.osd.10.smithi082.stdout:May 03 21:38:36 smithi082 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-10[57676]: 2024-05-03T21:38:36.110+0000 7f790b86c700 -1 osd.10 87 heartbeat_check: no reply from 172.21.15.138:6804 osd.2 since back 2024-05-03T21:37:36.406076+0000 front 2024-05-03T21:37:36.406028+0000 (oldest deadline 2024-05-03T21:38:01.705914+0000) 2024-05-03T21:38:36.496 INFO:journalctl@ceph.osd.10.smithi082.stdout:May 03 21:38:36 smithi082 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-10[57676]: 2024-05-03T21:38:36.110+0000 7f790b86c700 -1 osd.10 87 heartbeat_check: no reply from 172.21.15.67:6814 osd.3 since back 2024-05-03T21:37:41.706645+0000 front 2024-05-03T21:37:41.706615+0000 (oldest deadline 2024-05-03T21:38:08.107018+0000) 2024-05-03T21:38:36.496 INFO:journalctl@ceph.osd.10.smithi082.stdout:May 03 21:38:36 smithi082 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-10[57676]: 2024-05-03T21:38:36.110+0000 7f790b86c700 -1 osd.10 87 heartbeat_check: no reply from 172.21.15.82:6812 osd.4 since back 2024-05-03T21:37:48.107476+0000 front 2024-05-03T21:37:48.107543+0000 (oldest deadline 2024-05-03T21:38:14.007654+0000) 2024-05-03T21:38:36.496 INFO:journalctl@ceph.osd.10.smithi082.stdout:May 03 21:38:36 smithi082 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-10[57676]: 2024-05-03T21:38:36.110+0000 7f790b86c700 -1 osd.10 87 heartbeat_check: no reply from 172.21.15.138:6812 osd.5 since back 2024-05-03T21:38:00.409331+0000 front 2024-05-03T21:38:00.409486+0000 (oldest deadline 2024-05-03T21:38:23.909130+0000) 2024-05-03T21:38:36.496 INFO:journalctl@ceph.osd.10.smithi082.stdout:May 03 21:38:36 smithi082 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-10[57676]: 2024-05-03T21:38:36.110+0000 7f790b86c700 -1 osd.10 87 heartbeat_check: no reply from 172.21.15.67:6822 osd.6 since back 2024-05-03T21:38:07.409859+0000 front 2024-05-03T21:38:07.409980+0000 (oldest deadline 2024-05-03T21:38:29.010268+0000) 2024-05-03T21:38:36.824 INFO:journalctl@ceph.osd.11.smithi138.stdout:May 03 21:38:36 smithi138 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-11[58547]: 2024-05-03T21:38:36.579+0000 7fdfdcb59700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-03T21:37:21.054826+0000 front 2024-05-03T21:37:21.054670+0000 (oldest deadline 2024-05-03T21:37:45.154864+0000) 2024-05-03T21:38:36.824 INFO:journalctl@ceph.osd.11.smithi138.stdout:May 03 21:38:36 smithi138 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-11[58547]: 2024-05-03T21:38:36.579+0000 7fdfdcb59700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-05-03T21:37:29.256054+0000 front 2024-05-03T21:37:29.256037+0000 (oldest deadline 2024-05-03T21:37:53.956115+0000) 2024-05-03T21:38:36.824 INFO:journalctl@ceph.osd.11.smithi138.stdout:May 03 21:38:36 smithi138 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-11[58547]: 2024-05-03T21:38:36.579+0000 7fdfdcb59700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.138:6804 osd.2 since back 2024-05-03T21:37:33.956668+0000 front 2024-05-03T21:37:33.956702+0000 (oldest deadline 2024-05-03T21:37:59.856824+0000) 2024-05-03T21:38:36.824 INFO:journalctl@ceph.osd.11.smithi138.stdout:May 03 21:38:36 smithi138 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-11[58547]: 2024-05-03T21:38:36.579+0000 7fdfdcb59700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.67:6814 osd.3 since back 2024-05-03T21:37:43.759130+0000 front 2024-05-03T21:37:43.759044+0000 (oldest deadline 2024-05-03T21:38:09.058895+0000) 2024-05-03T21:38:36.824 INFO:journalctl@ceph.osd.11.smithi138.stdout:May 03 21:38:36 smithi138 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-11[58547]: 2024-05-03T21:38:36.579+0000 7fdfdcb59700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.82:6812 osd.4 since back 2024-05-03T21:37:49.059390+0000 front 2024-05-03T21:37:49.059401+0000 (oldest deadline 2024-05-03T21:38:14.959407+0000) 2024-05-03T21:38:36.825 INFO:journalctl@ceph.osd.11.smithi138.stdout:May 03 21:38:36 smithi138 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-11[58547]: 2024-05-03T21:38:36.579+0000 7fdfdcb59700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.138:6812 osd.5 since back 2024-05-03T21:37:56.660334+0000 front 2024-05-03T21:37:56.660386+0000 (oldest deadline 2024-05-03T21:38:21.960518+0000) 2024-05-03T21:38:37.495 INFO:journalctl@ceph.osd.10.smithi082.stdout:May 03 21:38:37 smithi082 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-10[57676]: 2024-05-03T21:38:37.137+0000 7f790b86c700 -1 osd.10 87 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-03T21:37:18.101114+0000 front 2024-05-03T21:37:18.101101+0000 (oldest deadline 2024-05-03T21:37:42.800953+0000) 2024-05-03T21:38:37.495 INFO:journalctl@ceph.osd.10.smithi082.stdout:May 03 21:38:37 smithi082 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-10[57676]: 2024-05-03T21:38:37.137+0000 7f790b86c700 -1 osd.10 87 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-05-03T21:37:28.303836+0000 front 2024-05-03T21:37:28.303941+0000 (oldest deadline 2024-05-03T21:37:51.804075+0000) 2024-05-03T21:38:37.495 INFO:journalctl@ceph.osd.10.smithi082.stdout:May 03 21:38:37 smithi082 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-10[57676]: 2024-05-03T21:38:37.137+0000 7f790b86c700 -1 osd.10 87 heartbeat_check: no reply from 172.21.15.138:6804 osd.2 since back 2024-05-03T21:37:36.406076+0000 front 2024-05-03T21:37:36.406028+0000 (oldest deadline 2024-05-03T21:38:01.705914+0000) 2024-05-03T21:38:37.495 INFO:journalctl@ceph.osd.10.smithi082.stdout:May 03 21:38:37 smithi082 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-10[57676]: 2024-05-03T21:38:37.137+0000 7f790b86c700 -1 osd.10 87 heartbeat_check: no reply from 172.21.15.67:6814 osd.3 since back 2024-05-03T21:37:41.706645+0000 front 2024-05-03T21:37:41.706615+0000 (oldest deadline 2024-05-03T21:38:08.107018+0000) 2024-05-03T21:38:37.495 INFO:journalctl@ceph.osd.10.smithi082.stdout:May 03 21:38:37 smithi082 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-10[57676]: 2024-05-03T21:38:37.137+0000 7f790b86c700 -1 osd.10 87 heartbeat_check: no reply from 172.21.15.82:6812 osd.4 since back 2024-05-03T21:37:48.107476+0000 front 2024-05-03T21:37:48.107543+0000 (oldest deadline 2024-05-03T21:38:14.007654+0000) 2024-05-03T21:38:37.496 INFO:journalctl@ceph.osd.10.smithi082.stdout:May 03 21:38:37 smithi082 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-10[57676]: 2024-05-03T21:38:37.137+0000 7f790b86c700 -1 osd.10 87 heartbeat_check: no reply from 172.21.15.138:6812 osd.5 since back 2024-05-03T21:38:00.409331+0000 front 2024-05-03T21:38:00.409486+0000 (oldest deadline 2024-05-03T21:38:23.909130+0000) 2024-05-03T21:38:37.496 INFO:journalctl@ceph.osd.10.smithi082.stdout:May 03 21:38:37 smithi082 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-10[57676]: 2024-05-03T21:38:37.137+0000 7f790b86c700 -1 osd.10 87 heartbeat_check: no reply from 172.21.15.67:6822 osd.6 since back 2024-05-03T21:38:07.409859+0000 front 2024-05-03T21:38:07.409980+0000 (oldest deadline 2024-05-03T21:38:29.010268+0000) 2024-05-03T21:38:38.074 INFO:journalctl@ceph.osd.11.smithi138.stdout:May 03 21:38:37 smithi138 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-11[58547]: 2024-05-03T21:38:37.608+0000 7fdfdcb59700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-03T21:37:21.054826+0000 front 2024-05-03T21:37:21.054670+0000 (oldest deadline 2024-05-03T21:37:45.154864+0000) 2024-05-03T21:38:38.074 INFO:journalctl@ceph.osd.11.smithi138.stdout:May 03 21:38:37 smithi138 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-11[58547]: 2024-05-03T21:38:37.608+0000 7fdfdcb59700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-05-03T21:37:29.256054+0000 front 2024-05-03T21:37:29.256037+0000 (oldest deadline 2024-05-03T21:37:53.956115+0000) 2024-05-03T21:38:38.074 INFO:journalctl@ceph.osd.11.smithi138.stdout:May 03 21:38:37 smithi138 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-11[58547]: 2024-05-03T21:38:37.608+0000 7fdfdcb59700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.138:6804 osd.2 since back 2024-05-03T21:37:33.956668+0000 front 2024-05-03T21:37:33.956702+0000 (oldest deadline 2024-05-03T21:37:59.856824+0000) 2024-05-03T21:38:38.075 INFO:journalctl@ceph.osd.11.smithi138.stdout:May 03 21:38:37 smithi138 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-11[58547]: 2024-05-03T21:38:37.608+0000 7fdfdcb59700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.67:6814 osd.3 since back 2024-05-03T21:37:43.759130+0000 front 2024-05-03T21:37:43.759044+0000 (oldest deadline 2024-05-03T21:38:09.058895+0000) 2024-05-03T21:38:38.075 INFO:journalctl@ceph.osd.11.smithi138.stdout:May 03 21:38:37 smithi138 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-11[58547]: 2024-05-03T21:38:37.608+0000 7fdfdcb59700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.82:6812 osd.4 since back 2024-05-03T21:37:49.059390+0000 front 2024-05-03T21:37:49.059401+0000 (oldest deadline 2024-05-03T21:38:14.959407+0000) 2024-05-03T21:38:38.075 INFO:journalctl@ceph.osd.11.smithi138.stdout:May 03 21:38:37 smithi138 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-11[58547]: 2024-05-03T21:38:37.608+0000 7fdfdcb59700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.138:6812 osd.5 since back 2024-05-03T21:37:56.660334+0000 front 2024-05-03T21:37:56.660386+0000 (oldest deadline 2024-05-03T21:38:21.960518+0000) 2024-05-03T21:38:38.495 INFO:journalctl@ceph.osd.10.smithi082.stdout:May 03 21:38:38 smithi082 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-10[57676]: 2024-05-03T21:38:38.172+0000 7f790b86c700 -1 osd.10 87 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-03T21:37:18.101114+0000 front 2024-05-03T21:37:18.101101+0000 (oldest deadline 2024-05-03T21:37:42.800953+0000) 2024-05-03T21:38:38.495 INFO:journalctl@ceph.osd.10.smithi082.stdout:May 03 21:38:38 smithi082 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-10[57676]: 2024-05-03T21:38:38.172+0000 7f790b86c700 -1 osd.10 87 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-05-03T21:37:28.303836+0000 front 2024-05-03T21:37:28.303941+0000 (oldest deadline 2024-05-03T21:37:51.804075+0000) 2024-05-03T21:38:38.496 INFO:journalctl@ceph.osd.10.smithi082.stdout:May 03 21:38:38 smithi082 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-10[57676]: 2024-05-03T21:38:38.172+0000 7f790b86c700 -1 osd.10 87 heartbeat_check: no reply from 172.21.15.138:6804 osd.2 since back 2024-05-03T21:37:36.406076+0000 front 2024-05-03T21:37:36.406028+0000 (oldest deadline 2024-05-03T21:38:01.705914+0000) 2024-05-03T21:38:38.496 INFO:journalctl@ceph.osd.10.smithi082.stdout:May 03 21:38:38 smithi082 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-10[57676]: 2024-05-03T21:38:38.172+0000 7f790b86c700 -1 osd.10 87 heartbeat_check: no reply from 172.21.15.67:6814 osd.3 since back 2024-05-03T21:37:41.706645+0000 front 2024-05-03T21:37:41.706615+0000 (oldest deadline 2024-05-03T21:38:08.107018+0000) 2024-05-03T21:38:38.496 INFO:journalctl@ceph.osd.10.smithi082.stdout:May 03 21:38:38 smithi082 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-10[57676]: 2024-05-03T21:38:38.172+0000 7f790b86c700 -1 osd.10 87 heartbeat_check: no reply from 172.21.15.82:6812 osd.4 since back 2024-05-03T21:37:48.107476+0000 front 2024-05-03T21:37:48.107543+0000 (oldest deadline 2024-05-03T21:38:14.007654+0000) 2024-05-03T21:38:38.496 INFO:journalctl@ceph.osd.10.smithi082.stdout:May 03 21:38:38 smithi082 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-10[57676]: 2024-05-03T21:38:38.172+0000 7f790b86c700 -1 osd.10 87 heartbeat_check: no reply from 172.21.15.138:6812 osd.5 since back 2024-05-03T21:38:00.409331+0000 front 2024-05-03T21:38:00.409486+0000 (oldest deadline 2024-05-03T21:38:23.909130+0000) 2024-05-03T21:38:38.496 INFO:journalctl@ceph.osd.10.smithi082.stdout:May 03 21:38:38 smithi082 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-10[57676]: 2024-05-03T21:38:38.172+0000 7f790b86c700 -1 osd.10 87 heartbeat_check: no reply from 172.21.15.67:6822 osd.6 since back 2024-05-03T21:38:07.409859+0000 front 2024-05-03T21:38:07.409980+0000 (oldest deadline 2024-05-03T21:38:29.010268+0000) 2024-05-03T21:38:39.074 INFO:journalctl@ceph.osd.11.smithi138.stdout:May 03 21:38:38 smithi138 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-11[58547]: 2024-05-03T21:38:38.631+0000 7fdfdcb59700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-03T21:37:21.054826+0000 front 2024-05-03T21:37:21.054670+0000 (oldest deadline 2024-05-03T21:37:45.154864+0000) 2024-05-03T21:38:39.074 INFO:journalctl@ceph.osd.11.smithi138.stdout:May 03 21:38:38 smithi138 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-11[58547]: 2024-05-03T21:38:38.631+0000 7fdfdcb59700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-05-03T21:37:29.256054+0000 front 2024-05-03T21:37:29.256037+0000 (oldest deadline 2024-05-03T21:37:53.956115+0000) 2024-05-03T21:38:39.074 INFO:journalctl@ceph.osd.11.smithi138.stdout:May 03 21:38:38 smithi138 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-11[58547]: 2024-05-03T21:38:38.631+0000 7fdfdcb59700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.138:6804 osd.2 since back 2024-05-03T21:37:33.956668+0000 front 2024-05-03T21:37:33.956702+0000 (oldest deadline 2024-05-03T21:37:59.856824+0000) 2024-05-03T21:38:39.074 INFO:journalctl@ceph.osd.11.smithi138.stdout:May 03 21:38:38 smithi138 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-11[58547]: 2024-05-03T21:38:38.631+0000 7fdfdcb59700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.67:6814 osd.3 since back 2024-05-03T21:37:43.759130+0000 front 2024-05-03T21:37:43.759044+0000 (oldest deadline 2024-05-03T21:38:09.058895+0000) 2024-05-03T21:38:39.074 INFO:journalctl@ceph.osd.11.smithi138.stdout:May 03 21:38:38 smithi138 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-11[58547]: 2024-05-03T21:38:38.631+0000 7fdfdcb59700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.82:6812 osd.4 since back 2024-05-03T21:37:49.059390+0000 front 2024-05-03T21:37:49.059401+0000 (oldest deadline 2024-05-03T21:38:14.959407+0000) 2024-05-03T21:38:39.075 INFO:journalctl@ceph.osd.11.smithi138.stdout:May 03 21:38:38 smithi138 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-11[58547]: 2024-05-03T21:38:38.631+0000 7fdfdcb59700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.138:6812 osd.5 since back 2024-05-03T21:37:56.660334+0000 front 2024-05-03T21:37:56.660386+0000 (oldest deadline 2024-05-03T21:38:21.960518+0000) 2024-05-03T21:38:39.075 INFO:journalctl@ceph.osd.11.smithi138.stdout:May 03 21:38:38 smithi138 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-11[58547]: 2024-05-03T21:38:38.632+0000 7fdfdcb59700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.82:6820 osd.7 since back 2024-05-03T21:38:15.263356+0000 front 2024-05-03T21:38:15.263331+0000 (oldest deadline 2024-05-03T21:38:38.163325+0000) 2024-05-03T21:38:39.495 INFO:journalctl@ceph.osd.10.smithi082.stdout:May 03 21:38:39 smithi082 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-10[57676]: 2024-05-03T21:38:39.201+0000 7f790b86c700 -1 osd.10 87 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-03T21:37:18.101114+0000 front 2024-05-03T21:37:18.101101+0000 (oldest deadline 2024-05-03T21:37:42.800953+0000) 2024-05-03T21:38:39.495 INFO:journalctl@ceph.osd.10.smithi082.stdout:May 03 21:38:39 smithi082 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-10[57676]: 2024-05-03T21:38:39.201+0000 7f790b86c700 -1 osd.10 87 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-05-03T21:37:28.303836+0000 front 2024-05-03T21:37:28.303941+0000 (oldest deadline 2024-05-03T21:37:51.804075+0000) 2024-05-03T21:38:39.495 INFO:journalctl@ceph.osd.10.smithi082.stdout:May 03 21:38:39 smithi082 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-10[57676]: 2024-05-03T21:38:39.201+0000 7f790b86c700 -1 osd.10 87 heartbeat_check: no reply from 172.21.15.138:6804 osd.2 since back 2024-05-03T21:37:36.406076+0000 front 2024-05-03T21:37:36.406028+0000 (oldest deadline 2024-05-03T21:38:01.705914+0000) 2024-05-03T21:38:39.495 INFO:journalctl@ceph.osd.10.smithi082.stdout:May 03 21:38:39 smithi082 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-10[57676]: 2024-05-03T21:38:39.201+0000 7f790b86c700 -1 osd.10 87 heartbeat_check: no reply from 172.21.15.67:6814 osd.3 since back 2024-05-03T21:37:41.706645+0000 front 2024-05-03T21:37:41.706615+0000 (oldest deadline 2024-05-03T21:38:08.107018+0000) 2024-05-03T21:38:39.495 INFO:journalctl@ceph.osd.10.smithi082.stdout:May 03 21:38:39 smithi082 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-10[57676]: 2024-05-03T21:38:39.201+0000 7f790b86c700 -1 osd.10 87 heartbeat_check: no reply from 172.21.15.82:6812 osd.4 since back 2024-05-03T21:37:48.107476+0000 front 2024-05-03T21:37:48.107543+0000 (oldest deadline 2024-05-03T21:38:14.007654+0000) 2024-05-03T21:38:39.495 INFO:journalctl@ceph.osd.10.smithi082.stdout:May 03 21:38:39 smithi082 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-10[57676]: 2024-05-03T21:38:39.201+0000 7f790b86c700 -1 osd.10 87 heartbeat_check: no reply from 172.21.15.138:6812 osd.5 since back 2024-05-03T21:38:00.409331+0000 front 2024-05-03T21:38:00.409486+0000 (oldest deadline 2024-05-03T21:38:23.909130+0000) 2024-05-03T21:38:39.496 INFO:journalctl@ceph.osd.10.smithi082.stdout:May 03 21:38:39 smithi082 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-10[57676]: 2024-05-03T21:38:39.201+0000 7f790b86c700 -1 osd.10 87 heartbeat_check: no reply from 172.21.15.67:6822 osd.6 since back 2024-05-03T21:38:07.409859+0000 front 2024-05-03T21:38:07.409980+0000 (oldest deadline 2024-05-03T21:38:29.010268+0000) 2024-05-03T21:38:40.074 INFO:journalctl@ceph.osd.11.smithi138.stdout:May 03 21:38:39 smithi138 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-11[58547]: 2024-05-03T21:38:39.633+0000 7fdfdcb59700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-03T21:37:21.054826+0000 front 2024-05-03T21:37:21.054670+0000 (oldest deadline 2024-05-03T21:37:45.154864+0000) 2024-05-03T21:38:40.074 INFO:journalctl@ceph.osd.11.smithi138.stdout:May 03 21:38:39 smithi138 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-11[58547]: 2024-05-03T21:38:39.633+0000 7fdfdcb59700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-05-03T21:37:29.256054+0000 front 2024-05-03T21:37:29.256037+0000 (oldest deadline 2024-05-03T21:37:53.956115+0000) 2024-05-03T21:38:40.074 INFO:journalctl@ceph.osd.11.smithi138.stdout:May 03 21:38:39 smithi138 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-11[58547]: 2024-05-03T21:38:39.633+0000 7fdfdcb59700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.138:6804 osd.2 since back 2024-05-03T21:37:33.956668+0000 front 2024-05-03T21:37:33.956702+0000 (oldest deadline 2024-05-03T21:37:59.856824+0000) 2024-05-03T21:38:40.074 INFO:journalctl@ceph.osd.11.smithi138.stdout:May 03 21:38:39 smithi138 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-11[58547]: 2024-05-03T21:38:39.633+0000 7fdfdcb59700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.67:6814 osd.3 since back 2024-05-03T21:37:43.759130+0000 front 2024-05-03T21:37:43.759044+0000 (oldest deadline 2024-05-03T21:38:09.058895+0000) 2024-05-03T21:38:40.074 INFO:journalctl@ceph.osd.11.smithi138.stdout:May 03 21:38:39 smithi138 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-11[58547]: 2024-05-03T21:38:39.633+0000 7fdfdcb59700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.82:6812 osd.4 since back 2024-05-03T21:37:49.059390+0000 front 2024-05-03T21:37:49.059401+0000 (oldest deadline 2024-05-03T21:38:14.959407+0000) 2024-05-03T21:38:40.075 INFO:journalctl@ceph.osd.11.smithi138.stdout:May 03 21:38:39 smithi138 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-11[58547]: 2024-05-03T21:38:39.633+0000 7fdfdcb59700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.138:6812 osd.5 since back 2024-05-03T21:37:56.660334+0000 front 2024-05-03T21:37:56.660386+0000 (oldest deadline 2024-05-03T21:38:21.960518+0000) 2024-05-03T21:38:40.075 INFO:journalctl@ceph.osd.11.smithi138.stdout:May 03 21:38:39 smithi138 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-11[58547]: 2024-05-03T21:38:39.633+0000 7fdfdcb59700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.82:6820 osd.7 since back 2024-05-03T21:38:15.263356+0000 front 2024-05-03T21:38:15.263331+0000 (oldest deadline 2024-05-03T21:38:38.163325+0000) 2024-05-03T21:38:40.492 INFO:journalctl@ceph.osd.10.smithi082.stdout:May 03 21:38:40 smithi082 podman[68922]: 2024-05-03 21:38:40.213164451 +0000 UTC m=+5.593874345 container died 1f29479071226e86c32337a7436c19d01c14f8bb1436413933a18144a946c543 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:68e15ae152ef5c805c733e76da8fc1d3dd4ee669d5ba025f457e05725baa24dd, name=ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-10, org.label-schema.build-date=20240430, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, RELEASE=wip-rishabh-testing-20240501.193033-9a047fe, org.label-schema.license=GPLv2) 2024-05-03T21:38:40.745 INFO:journalctl@ceph.osd.10.smithi082.stdout:May 03 21:38:40 smithi082 podman[68922]: 2024-05-03 21:38:40.578029768 +0000 UTC m=+5.958739661 container cleanup 1f29479071226e86c32337a7436c19d01c14f8bb1436413933a18144a946c543 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:68e15ae152ef5c805c733e76da8fc1d3dd4ee669d5ba025f457e05725baa24dd, name=ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-10, org.label-schema.vendor=CentOS, RELEASE=wip-rishabh-testing-20240501.193033-9a047fe, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, CEPH_POINT_RELEASE=, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240430) 2024-05-03T21:38:40.745 INFO:journalctl@ceph.osd.10.smithi082.stdout:May 03 21:38:40 smithi082 bash[68922]: ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-10 2024-05-03T21:38:41.074 INFO:journalctl@ceph.osd.11.smithi138.stdout:May 03 21:38:40 smithi138 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-11[58547]: 2024-05-03T21:38:40.626+0000 7fdfdcb59700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-03T21:37:21.054826+0000 front 2024-05-03T21:37:21.054670+0000 (oldest deadline 2024-05-03T21:37:45.154864+0000) 2024-05-03T21:38:41.074 INFO:journalctl@ceph.osd.11.smithi138.stdout:May 03 21:38:40 smithi138 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-11[58547]: 2024-05-03T21:38:40.626+0000 7fdfdcb59700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-05-03T21:37:29.256054+0000 front 2024-05-03T21:37:29.256037+0000 (oldest deadline 2024-05-03T21:37:53.956115+0000) 2024-05-03T21:38:41.074 INFO:journalctl@ceph.osd.11.smithi138.stdout:May 03 21:38:40 smithi138 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-11[58547]: 2024-05-03T21:38:40.626+0000 7fdfdcb59700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.138:6804 osd.2 since back 2024-05-03T21:37:33.956668+0000 front 2024-05-03T21:37:33.956702+0000 (oldest deadline 2024-05-03T21:37:59.856824+0000) 2024-05-03T21:38:41.074 INFO:journalctl@ceph.osd.11.smithi138.stdout:May 03 21:38:40 smithi138 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-11[58547]: 2024-05-03T21:38:40.626+0000 7fdfdcb59700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.67:6814 osd.3 since back 2024-05-03T21:37:43.759130+0000 front 2024-05-03T21:37:43.759044+0000 (oldest deadline 2024-05-03T21:38:09.058895+0000) 2024-05-03T21:38:41.075 INFO:journalctl@ceph.osd.11.smithi138.stdout:May 03 21:38:40 smithi138 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-11[58547]: 2024-05-03T21:38:40.626+0000 7fdfdcb59700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.82:6812 osd.4 since back 2024-05-03T21:37:49.059390+0000 front 2024-05-03T21:37:49.059401+0000 (oldest deadline 2024-05-03T21:38:14.959407+0000) 2024-05-03T21:38:41.075 INFO:journalctl@ceph.osd.11.smithi138.stdout:May 03 21:38:40 smithi138 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-11[58547]: 2024-05-03T21:38:40.626+0000 7fdfdcb59700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.138:6812 osd.5 since back 2024-05-03T21:37:56.660334+0000 front 2024-05-03T21:37:56.660386+0000 (oldest deadline 2024-05-03T21:38:21.960518+0000) 2024-05-03T21:38:41.075 INFO:journalctl@ceph.osd.11.smithi138.stdout:May 03 21:38:40 smithi138 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-11[58547]: 2024-05-03T21:38:40.626+0000 7fdfdcb59700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.82:6820 osd.7 since back 2024-05-03T21:38:15.263356+0000 front 2024-05-03T21:38:15.263331+0000 (oldest deadline 2024-05-03T21:38:38.163325+0000) 2024-05-03T21:38:41.205 INFO:journalctl@ceph.osd.10.smithi082.stdout:May 03 21:38:40 smithi082 podman[68939]: 2024-05-03 21:38:40.854070592 +0000 UTC m=+0.634792967 container remove 1f29479071226e86c32337a7436c19d01c14f8bb1436413933a18144a946c543 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:68e15ae152ef5c805c733e76da8fc1d3dd4ee669d5ba025f457e05725baa24dd, name=ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-10, GIT_BRANCH=HEAD, GIT_CLEAN=True, RELEASE=wip-rishabh-testing-20240501.193033-9a047fe, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.build-date=20240430, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-05-03T21:38:41.495 INFO:journalctl@ceph.osd.10.smithi082.stdout:May 03 21:38:41 smithi082 podman[69030]: 2024-05-03 21:38:41.204762548 +0000 UTC m=+0.114410264 container create 19170c6823e390a629b5b99b824af5c9388c8c28ce890b8d3d9a0a4376648a8b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:68e15ae152ef5c805c733e76da8fc1d3dd4ee669d5ba025f457e05725baa24dd, name=ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-10-deactivate, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, RELEASE=wip-rishabh-testing-20240501.193033-9a047fe, org.label-schema.build-date=20240430, ceph=True, io.buildah.version=1.33.5, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-05-03T21:38:41.495 INFO:journalctl@ceph.osd.10.smithi082.stdout:May 03 21:38:41 smithi082 podman[69030]: 2024-05-03 21:38:41.108021656 +0000 UTC m=+0.017652282 image pull ee4d846202496ba6096593e8e8becc0700a5eb7ebd95536031187f5f337d4012 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:68e15ae152ef5c805c733e76da8fc1d3dd4ee669d5ba025f457e05725baa24dd 2024-05-03T21:38:41.495 INFO:journalctl@ceph.osd.10.smithi082.stdout:May 03 21:38:41 smithi082 podman[69030]: 2024-05-03 21:38:41.42360527 +0000 UTC m=+0.333235883 container init 19170c6823e390a629b5b99b824af5c9388c8c28ce890b8d3d9a0a4376648a8b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:68e15ae152ef5c805c733e76da8fc1d3dd4ee669d5ba025f457e05725baa24dd, name=ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-10-deactivate, org.label-schema.build-date=20240430, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, io.buildah.version=1.33.5, RELEASE=wip-rishabh-testing-20240501.193033-9a047fe, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=) 2024-05-03T21:38:41.495 INFO:journalctl@ceph.osd.10.smithi082.stdout:May 03 21:38:41 smithi082 podman[69030]: 2024-05-03 21:38:41.426583618 +0000 UTC m=+0.336214229 container start 19170c6823e390a629b5b99b824af5c9388c8c28ce890b8d3d9a0a4376648a8b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:68e15ae152ef5c805c733e76da8fc1d3dd4ee669d5ba025f457e05725baa24dd, name=ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-10-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, RELEASE=wip-rishabh-testing-20240501.193033-9a047fe, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.build-date=20240430, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, ceph=True) 2024-05-03T21:38:41.995 INFO:journalctl@ceph.osd.10.smithi082.stdout:May 03 21:38:41 smithi082 podman[69030]: 2024-05-03 21:38:41.51143661 +0000 UTC m=+0.421067234 container attach 19170c6823e390a629b5b99b824af5c9388c8c28ce890b8d3d9a0a4376648a8b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:68e15ae152ef5c805c733e76da8fc1d3dd4ee669d5ba025f457e05725baa24dd, name=ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-10-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-rishabh-testing-20240501.193033-9a047fe, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240430, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, ceph=True, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD) 2024-05-03T21:38:41.995 INFO:journalctl@ceph.osd.10.smithi082.stdout:May 03 21:38:41 smithi082 podman[69030]: 2024-05-03 21:38:41.712390298 +0000 UTC m=+0.622020908 container died 19170c6823e390a629b5b99b824af5c9388c8c28ce890b8d3d9a0a4376648a8b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:68e15ae152ef5c805c733e76da8fc1d3dd4ee669d5ba025f457e05725baa24dd, name=ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-10-deactivate, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_BRANCH=HEAD, org.label-schema.build-date=20240430, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-rishabh-testing-20240501.193033-9a047fe, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True) 2024-05-03T21:38:42.074 INFO:journalctl@ceph.osd.11.smithi138.stdout:May 03 21:38:41 smithi138 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-11[58547]: 2024-05-03T21:38:41.623+0000 7fdfdcb59700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-03T21:37:21.054826+0000 front 2024-05-03T21:37:21.054670+0000 (oldest deadline 2024-05-03T21:37:45.154864+0000) 2024-05-03T21:38:42.074 INFO:journalctl@ceph.osd.11.smithi138.stdout:May 03 21:38:41 smithi138 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-11[58547]: 2024-05-03T21:38:41.623+0000 7fdfdcb59700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-05-03T21:37:29.256054+0000 front 2024-05-03T21:37:29.256037+0000 (oldest deadline 2024-05-03T21:37:53.956115+0000) 2024-05-03T21:38:42.074 INFO:journalctl@ceph.osd.11.smithi138.stdout:May 03 21:38:41 smithi138 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-11[58547]: 2024-05-03T21:38:41.623+0000 7fdfdcb59700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.138:6804 osd.2 since back 2024-05-03T21:37:33.956668+0000 front 2024-05-03T21:37:33.956702+0000 (oldest deadline 2024-05-03T21:37:59.856824+0000) 2024-05-03T21:38:42.075 INFO:journalctl@ceph.osd.11.smithi138.stdout:May 03 21:38:41 smithi138 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-11[58547]: 2024-05-03T21:38:41.623+0000 7fdfdcb59700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.67:6814 osd.3 since back 2024-05-03T21:37:43.759130+0000 front 2024-05-03T21:37:43.759044+0000 (oldest deadline 2024-05-03T21:38:09.058895+0000) 2024-05-03T21:38:42.075 INFO:journalctl@ceph.osd.11.smithi138.stdout:May 03 21:38:41 smithi138 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-11[58547]: 2024-05-03T21:38:41.623+0000 7fdfdcb59700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.82:6812 osd.4 since back 2024-05-03T21:37:49.059390+0000 front 2024-05-03T21:37:49.059401+0000 (oldest deadline 2024-05-03T21:38:14.959407+0000) 2024-05-03T21:38:42.075 INFO:journalctl@ceph.osd.11.smithi138.stdout:May 03 21:38:41 smithi138 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-11[58547]: 2024-05-03T21:38:41.623+0000 7fdfdcb59700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.138:6812 osd.5 since back 2024-05-03T21:37:56.660334+0000 front 2024-05-03T21:37:56.660386+0000 (oldest deadline 2024-05-03T21:38:21.960518+0000) 2024-05-03T21:38:42.075 INFO:journalctl@ceph.osd.11.smithi138.stdout:May 03 21:38:41 smithi138 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-11[58547]: 2024-05-03T21:38:41.623+0000 7fdfdcb59700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.82:6820 osd.7 since back 2024-05-03T21:38:15.263356+0000 front 2024-05-03T21:38:15.263331+0000 (oldest deadline 2024-05-03T21:38:38.163325+0000) 2024-05-03T21:38:42.264 INFO:journalctl@ceph.osd.10.smithi082.stdout:May 03 21:38:42 smithi082 podman[69056]: 2024-05-03 21:38:42.243314726 +0000 UTC m=+0.526015394 container remove 19170c6823e390a629b5b99b824af5c9388c8c28ce890b8d3d9a0a4376648a8b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:68e15ae152ef5c805c733e76da8fc1d3dd4ee669d5ba025f457e05725baa24dd, name=ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-10-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.license=GPLv2, GIT_CLEAN=True, RELEASE=wip-rishabh-testing-20240501.193033-9a047fe, org.label-schema.build-date=20240430, ceph=True, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=) 2024-05-03T21:38:42.269 DEBUG:teuthology.orchestra.run.smithi082:> sudo pkill -f 'journalctl -f -n 0 -u ceph-713bc57a-0991-11ef-bc97-c7b262605968@osd.10.service' 2024-05-03T21:38:42.344 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-03T21:38:42.345 INFO:tasks.cephadm.osd.10:Stopped osd.10 2024-05-03T21:38:42.345 INFO:tasks.cephadm.osd.11:Stopping osd.11... 2024-05-03T21:38:42.345 DEBUG:teuthology.orchestra.run.smithi138:> sudo systemctl stop ceph-713bc57a-0991-11ef-bc97-c7b262605968@osd.11 2024-05-03T21:38:42.648 INFO:journalctl@ceph.osd.11.smithi138.stdout:May 03 21:38:42 smithi138 systemd[1]: Stopping Ceph osd.11 for 713bc57a-0991-11ef-bc97-c7b262605968... 2024-05-03T21:38:42.648 INFO:journalctl@ceph.osd.11.smithi138.stdout:May 03 21:38:42 smithi138 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-11[58547]: 2024-05-03T21:38:42.607+0000 7fdfdcb59700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-03T21:37:21.054826+0000 front 2024-05-03T21:37:21.054670+0000 (oldest deadline 2024-05-03T21:37:45.154864+0000) 2024-05-03T21:38:42.648 INFO:journalctl@ceph.osd.11.smithi138.stdout:May 03 21:38:42 smithi138 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-11[58547]: 2024-05-03T21:38:42.607+0000 7fdfdcb59700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-05-03T21:37:29.256054+0000 front 2024-05-03T21:37:29.256037+0000 (oldest deadline 2024-05-03T21:37:53.956115+0000) 2024-05-03T21:38:42.649 INFO:journalctl@ceph.osd.11.smithi138.stdout:May 03 21:38:42 smithi138 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-11[58547]: 2024-05-03T21:38:42.607+0000 7fdfdcb59700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.138:6804 osd.2 since back 2024-05-03T21:37:33.956668+0000 front 2024-05-03T21:37:33.956702+0000 (oldest deadline 2024-05-03T21:37:59.856824+0000) 2024-05-03T21:38:42.649 INFO:journalctl@ceph.osd.11.smithi138.stdout:May 03 21:38:42 smithi138 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-11[58547]: 2024-05-03T21:38:42.607+0000 7fdfdcb59700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.67:6814 osd.3 since back 2024-05-03T21:37:43.759130+0000 front 2024-05-03T21:37:43.759044+0000 (oldest deadline 2024-05-03T21:38:09.058895+0000) 2024-05-03T21:38:42.649 INFO:journalctl@ceph.osd.11.smithi138.stdout:May 03 21:38:42 smithi138 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-11[58547]: 2024-05-03T21:38:42.607+0000 7fdfdcb59700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.82:6812 osd.4 since back 2024-05-03T21:37:49.059390+0000 front 2024-05-03T21:37:49.059401+0000 (oldest deadline 2024-05-03T21:38:14.959407+0000) 2024-05-03T21:38:42.649 INFO:journalctl@ceph.osd.11.smithi138.stdout:May 03 21:38:42 smithi138 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-11[58547]: 2024-05-03T21:38:42.607+0000 7fdfdcb59700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.138:6812 osd.5 since back 2024-05-03T21:37:56.660334+0000 front 2024-05-03T21:37:56.660386+0000 (oldest deadline 2024-05-03T21:38:21.960518+0000) 2024-05-03T21:38:42.649 INFO:journalctl@ceph.osd.11.smithi138.stdout:May 03 21:38:42 smithi138 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-11[58547]: 2024-05-03T21:38:42.607+0000 7fdfdcb59700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.82:6820 osd.7 since back 2024-05-03T21:38:15.263356+0000 front 2024-05-03T21:38:15.263331+0000 (oldest deadline 2024-05-03T21:38:38.163325+0000) 2024-05-03T21:38:42.649 INFO:journalctl@ceph.osd.11.smithi138.stdout:May 03 21:38:42 smithi138 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-11[58547]: 2024-05-03T21:38:42.646+0000 7fdfe0565700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.11 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-03T21:38:42.649 INFO:journalctl@ceph.osd.11.smithi138.stdout:May 03 21:38:42 smithi138 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-11[58547]: 2024-05-03T21:38:42.646+0000 7fdfe0565700 -1 osd.11 87 *** Got signal Terminated *** 2024-05-03T21:38:42.650 INFO:journalctl@ceph.osd.11.smithi138.stdout:May 03 21:38:42 smithi138 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-11[58547]: 2024-05-03T21:38:42.646+0000 7fdfe0565700 -1 osd.11 87 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-05-03T21:38:44.074 INFO:journalctl@ceph.osd.11.smithi138.stdout:May 03 21:38:43 smithi138 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-11[58547]: 2024-05-03T21:38:43.627+0000 7fdfdcb59700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-03T21:37:21.054826+0000 front 2024-05-03T21:37:21.054670+0000 (oldest deadline 2024-05-03T21:37:45.154864+0000) 2024-05-03T21:38:44.074 INFO:journalctl@ceph.osd.11.smithi138.stdout:May 03 21:38:43 smithi138 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-11[58547]: 2024-05-03T21:38:43.627+0000 7fdfdcb59700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-05-03T21:37:29.256054+0000 front 2024-05-03T21:37:29.256037+0000 (oldest deadline 2024-05-03T21:37:53.956115+0000) 2024-05-03T21:38:44.075 INFO:journalctl@ceph.osd.11.smithi138.stdout:May 03 21:38:43 smithi138 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-11[58547]: 2024-05-03T21:38:43.627+0000 7fdfdcb59700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.138:6804 osd.2 since back 2024-05-03T21:37:33.956668+0000 front 2024-05-03T21:37:33.956702+0000 (oldest deadline 2024-05-03T21:37:59.856824+0000) 2024-05-03T21:38:44.075 INFO:journalctl@ceph.osd.11.smithi138.stdout:May 03 21:38:43 smithi138 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-11[58547]: 2024-05-03T21:38:43.627+0000 7fdfdcb59700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.67:6814 osd.3 since back 2024-05-03T21:37:43.759130+0000 front 2024-05-03T21:37:43.759044+0000 (oldest deadline 2024-05-03T21:38:09.058895+0000) 2024-05-03T21:38:44.075 INFO:journalctl@ceph.osd.11.smithi138.stdout:May 03 21:38:43 smithi138 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-11[58547]: 2024-05-03T21:38:43.627+0000 7fdfdcb59700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.82:6812 osd.4 since back 2024-05-03T21:37:49.059390+0000 front 2024-05-03T21:37:49.059401+0000 (oldest deadline 2024-05-03T21:38:14.959407+0000) 2024-05-03T21:38:44.075 INFO:journalctl@ceph.osd.11.smithi138.stdout:May 03 21:38:43 smithi138 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-11[58547]: 2024-05-03T21:38:43.627+0000 7fdfdcb59700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.138:6812 osd.5 since back 2024-05-03T21:37:56.660334+0000 front 2024-05-03T21:37:56.660386+0000 (oldest deadline 2024-05-03T21:38:21.960518+0000) 2024-05-03T21:38:44.075 INFO:journalctl@ceph.osd.11.smithi138.stdout:May 03 21:38:43 smithi138 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-11[58547]: 2024-05-03T21:38:43.627+0000 7fdfdcb59700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.82:6820 osd.7 since back 2024-05-03T21:38:15.263356+0000 front 2024-05-03T21:38:15.263331+0000 (oldest deadline 2024-05-03T21:38:38.163325+0000) 2024-05-03T21:38:45.074 INFO:journalctl@ceph.osd.11.smithi138.stdout:May 03 21:38:44 smithi138 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-11[58547]: 2024-05-03T21:38:44.590+0000 7fdfdcb59700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-03T21:37:21.054826+0000 front 2024-05-03T21:37:21.054670+0000 (oldest deadline 2024-05-03T21:37:45.154864+0000) 2024-05-03T21:38:45.074 INFO:journalctl@ceph.osd.11.smithi138.stdout:May 03 21:38:44 smithi138 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-11[58547]: 2024-05-03T21:38:44.590+0000 7fdfdcb59700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-05-03T21:37:29.256054+0000 front 2024-05-03T21:37:29.256037+0000 (oldest deadline 2024-05-03T21:37:53.956115+0000) 2024-05-03T21:38:45.074 INFO:journalctl@ceph.osd.11.smithi138.stdout:May 03 21:38:44 smithi138 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-11[58547]: 2024-05-03T21:38:44.590+0000 7fdfdcb59700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.138:6804 osd.2 since back 2024-05-03T21:37:33.956668+0000 front 2024-05-03T21:37:33.956702+0000 (oldest deadline 2024-05-03T21:37:59.856824+0000) 2024-05-03T21:38:45.074 INFO:journalctl@ceph.osd.11.smithi138.stdout:May 03 21:38:44 smithi138 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-11[58547]: 2024-05-03T21:38:44.590+0000 7fdfdcb59700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.67:6814 osd.3 since back 2024-05-03T21:37:43.759130+0000 front 2024-05-03T21:37:43.759044+0000 (oldest deadline 2024-05-03T21:38:09.058895+0000) 2024-05-03T21:38:45.075 INFO:journalctl@ceph.osd.11.smithi138.stdout:May 03 21:38:44 smithi138 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-11[58547]: 2024-05-03T21:38:44.590+0000 7fdfdcb59700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.82:6812 osd.4 since back 2024-05-03T21:37:49.059390+0000 front 2024-05-03T21:37:49.059401+0000 (oldest deadline 2024-05-03T21:38:14.959407+0000) 2024-05-03T21:38:45.075 INFO:journalctl@ceph.osd.11.smithi138.stdout:May 03 21:38:44 smithi138 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-11[58547]: 2024-05-03T21:38:44.590+0000 7fdfdcb59700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.138:6812 osd.5 since back 2024-05-03T21:37:56.660334+0000 front 2024-05-03T21:37:56.660386+0000 (oldest deadline 2024-05-03T21:38:21.960518+0000) 2024-05-03T21:38:45.075 INFO:journalctl@ceph.osd.11.smithi138.stdout:May 03 21:38:44 smithi138 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-11[58547]: 2024-05-03T21:38:44.590+0000 7fdfdcb59700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.82:6820 osd.7 since back 2024-05-03T21:38:15.263356+0000 front 2024-05-03T21:38:15.263331+0000 (oldest deadline 2024-05-03T21:38:38.163325+0000) 2024-05-03T21:38:45.075 INFO:journalctl@ceph.osd.11.smithi138.stdout:May 03 21:38:44 smithi138 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-11[58547]: 2024-05-03T21:38:44.590+0000 7fdfdcb59700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.138:6820 osd.8 since back 2024-05-03T21:38:22.164544+0000 front 2024-05-03T21:38:22.164516+0000 (oldest deadline 2024-05-03T21:38:44.464590+0000) 2024-05-03T21:38:45.824 INFO:journalctl@ceph.osd.11.smithi138.stdout:May 03 21:38:45 smithi138 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-11[58547]: 2024-05-03T21:38:45.545+0000 7fdfdcb59700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-03T21:37:21.054826+0000 front 2024-05-03T21:37:21.054670+0000 (oldest deadline 2024-05-03T21:37:45.154864+0000) 2024-05-03T21:38:45.824 INFO:journalctl@ceph.osd.11.smithi138.stdout:May 03 21:38:45 smithi138 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-11[58547]: 2024-05-03T21:38:45.545+0000 7fdfdcb59700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-05-03T21:37:29.256054+0000 front 2024-05-03T21:37:29.256037+0000 (oldest deadline 2024-05-03T21:37:53.956115+0000) 2024-05-03T21:38:45.824 INFO:journalctl@ceph.osd.11.smithi138.stdout:May 03 21:38:45 smithi138 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-11[58547]: 2024-05-03T21:38:45.545+0000 7fdfdcb59700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.138:6804 osd.2 since back 2024-05-03T21:37:33.956668+0000 front 2024-05-03T21:37:33.956702+0000 (oldest deadline 2024-05-03T21:37:59.856824+0000) 2024-05-03T21:38:45.825 INFO:journalctl@ceph.osd.11.smithi138.stdout:May 03 21:38:45 smithi138 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-11[58547]: 2024-05-03T21:38:45.545+0000 7fdfdcb59700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.67:6814 osd.3 since back 2024-05-03T21:37:43.759130+0000 front 2024-05-03T21:37:43.759044+0000 (oldest deadline 2024-05-03T21:38:09.058895+0000) 2024-05-03T21:38:45.825 INFO:journalctl@ceph.osd.11.smithi138.stdout:May 03 21:38:45 smithi138 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-11[58547]: 2024-05-03T21:38:45.545+0000 7fdfdcb59700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.82:6812 osd.4 since back 2024-05-03T21:37:49.059390+0000 front 2024-05-03T21:37:49.059401+0000 (oldest deadline 2024-05-03T21:38:14.959407+0000) 2024-05-03T21:38:45.825 INFO:journalctl@ceph.osd.11.smithi138.stdout:May 03 21:38:45 smithi138 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-11[58547]: 2024-05-03T21:38:45.545+0000 7fdfdcb59700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.138:6812 osd.5 since back 2024-05-03T21:37:56.660334+0000 front 2024-05-03T21:37:56.660386+0000 (oldest deadline 2024-05-03T21:38:21.960518+0000) 2024-05-03T21:38:45.825 INFO:journalctl@ceph.osd.11.smithi138.stdout:May 03 21:38:45 smithi138 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-11[58547]: 2024-05-03T21:38:45.545+0000 7fdfdcb59700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.82:6820 osd.7 since back 2024-05-03T21:38:15.263356+0000 front 2024-05-03T21:38:15.263331+0000 (oldest deadline 2024-05-03T21:38:38.163325+0000) 2024-05-03T21:38:45.825 INFO:journalctl@ceph.osd.11.smithi138.stdout:May 03 21:38:45 smithi138 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-11[58547]: 2024-05-03T21:38:45.545+0000 7fdfdcb59700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.138:6820 osd.8 since back 2024-05-03T21:38:22.164544+0000 front 2024-05-03T21:38:22.164516+0000 (oldest deadline 2024-05-03T21:38:44.464590+0000) 2024-05-03T21:38:47.074 INFO:journalctl@ceph.osd.11.smithi138.stdout:May 03 21:38:46 smithi138 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-11[58547]: 2024-05-03T21:38:46.585+0000 7fdfdcb59700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-03T21:37:21.054826+0000 front 2024-05-03T21:37:21.054670+0000 (oldest deadline 2024-05-03T21:37:45.154864+0000) 2024-05-03T21:38:47.074 INFO:journalctl@ceph.osd.11.smithi138.stdout:May 03 21:38:46 smithi138 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-11[58547]: 2024-05-03T21:38:46.585+0000 7fdfdcb59700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-05-03T21:37:29.256054+0000 front 2024-05-03T21:37:29.256037+0000 (oldest deadline 2024-05-03T21:37:53.956115+0000) 2024-05-03T21:38:47.074 INFO:journalctl@ceph.osd.11.smithi138.stdout:May 03 21:38:46 smithi138 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-11[58547]: 2024-05-03T21:38:46.585+0000 7fdfdcb59700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.138:6804 osd.2 since back 2024-05-03T21:37:33.956668+0000 front 2024-05-03T21:37:33.956702+0000 (oldest deadline 2024-05-03T21:37:59.856824+0000) 2024-05-03T21:38:47.075 INFO:journalctl@ceph.osd.11.smithi138.stdout:May 03 21:38:46 smithi138 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-11[58547]: 2024-05-03T21:38:46.585+0000 7fdfdcb59700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.67:6814 osd.3 since back 2024-05-03T21:37:43.759130+0000 front 2024-05-03T21:37:43.759044+0000 (oldest deadline 2024-05-03T21:38:09.058895+0000) 2024-05-03T21:38:47.075 INFO:journalctl@ceph.osd.11.smithi138.stdout:May 03 21:38:46 smithi138 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-11[58547]: 2024-05-03T21:38:46.585+0000 7fdfdcb59700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.82:6812 osd.4 since back 2024-05-03T21:37:49.059390+0000 front 2024-05-03T21:37:49.059401+0000 (oldest deadline 2024-05-03T21:38:14.959407+0000) 2024-05-03T21:38:47.075 INFO:journalctl@ceph.osd.11.smithi138.stdout:May 03 21:38:46 smithi138 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-11[58547]: 2024-05-03T21:38:46.585+0000 7fdfdcb59700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.138:6812 osd.5 since back 2024-05-03T21:37:56.660334+0000 front 2024-05-03T21:37:56.660386+0000 (oldest deadline 2024-05-03T21:38:21.960518+0000) 2024-05-03T21:38:47.075 INFO:journalctl@ceph.osd.11.smithi138.stdout:May 03 21:38:46 smithi138 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-11[58547]: 2024-05-03T21:38:46.585+0000 7fdfdcb59700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.82:6820 osd.7 since back 2024-05-03T21:38:15.263356+0000 front 2024-05-03T21:38:15.263331+0000 (oldest deadline 2024-05-03T21:38:38.163325+0000) 2024-05-03T21:38:47.075 INFO:journalctl@ceph.osd.11.smithi138.stdout:May 03 21:38:46 smithi138 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-11[58547]: 2024-05-03T21:38:46.585+0000 7fdfdcb59700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.138:6820 osd.8 since back 2024-05-03T21:38:22.164544+0000 front 2024-05-03T21:38:22.164516+0000 (oldest deadline 2024-05-03T21:38:44.464590+0000) 2024-05-03T21:38:47.824 INFO:journalctl@ceph.osd.11.smithi138.stdout:May 03 21:38:47 smithi138 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-11[58547]: 2024-05-03T21:38:47.537+0000 7fdfdcb59700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-03T21:37:21.054826+0000 front 2024-05-03T21:37:21.054670+0000 (oldest deadline 2024-05-03T21:37:45.154864+0000) 2024-05-03T21:38:47.824 INFO:journalctl@ceph.osd.11.smithi138.stdout:May 03 21:38:47 smithi138 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-11[58547]: 2024-05-03T21:38:47.537+0000 7fdfdcb59700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-05-03T21:37:29.256054+0000 front 2024-05-03T21:37:29.256037+0000 (oldest deadline 2024-05-03T21:37:53.956115+0000) 2024-05-03T21:38:47.824 INFO:journalctl@ceph.osd.11.smithi138.stdout:May 03 21:38:47 smithi138 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-11[58547]: 2024-05-03T21:38:47.537+0000 7fdfdcb59700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.138:6804 osd.2 since back 2024-05-03T21:37:33.956668+0000 front 2024-05-03T21:37:33.956702+0000 (oldest deadline 2024-05-03T21:37:59.856824+0000) 2024-05-03T21:38:47.824 INFO:journalctl@ceph.osd.11.smithi138.stdout:May 03 21:38:47 smithi138 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-11[58547]: 2024-05-03T21:38:47.537+0000 7fdfdcb59700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.67:6814 osd.3 since back 2024-05-03T21:37:43.759130+0000 front 2024-05-03T21:37:43.759044+0000 (oldest deadline 2024-05-03T21:38:09.058895+0000) 2024-05-03T21:38:47.825 INFO:journalctl@ceph.osd.11.smithi138.stdout:May 03 21:38:47 smithi138 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-11[58547]: 2024-05-03T21:38:47.537+0000 7fdfdcb59700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.82:6812 osd.4 since back 2024-05-03T21:37:49.059390+0000 front 2024-05-03T21:37:49.059401+0000 (oldest deadline 2024-05-03T21:38:14.959407+0000) 2024-05-03T21:38:47.825 INFO:journalctl@ceph.osd.11.smithi138.stdout:May 03 21:38:47 smithi138 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-11[58547]: 2024-05-03T21:38:47.537+0000 7fdfdcb59700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.138:6812 osd.5 since back 2024-05-03T21:37:56.660334+0000 front 2024-05-03T21:37:56.660386+0000 (oldest deadline 2024-05-03T21:38:21.960518+0000) 2024-05-03T21:38:47.825 INFO:journalctl@ceph.osd.11.smithi138.stdout:May 03 21:38:47 smithi138 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-11[58547]: 2024-05-03T21:38:47.537+0000 7fdfdcb59700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.82:6820 osd.7 since back 2024-05-03T21:38:15.263356+0000 front 2024-05-03T21:38:15.263331+0000 (oldest deadline 2024-05-03T21:38:38.163325+0000) 2024-05-03T21:38:47.825 INFO:journalctl@ceph.osd.11.smithi138.stdout:May 03 21:38:47 smithi138 ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-11[58547]: 2024-05-03T21:38:47.537+0000 7fdfdcb59700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.138:6820 osd.8 since back 2024-05-03T21:38:22.164544+0000 front 2024-05-03T21:38:22.164516+0000 (oldest deadline 2024-05-03T21:38:44.464590+0000) 2024-05-03T21:38:48.324 INFO:journalctl@ceph.osd.11.smithi138.stdout:May 03 21:38:48 smithi138 podman[69315]: 2024-05-03 21:38:48.185747605 +0000 UTC m=+5.646906181 container died b6f3e5c4917bd7051246e4319d34bbd55faae5a35c4ecead4a06a1403ab95699 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:68e15ae152ef5c805c733e76da8fc1d3dd4ee669d5ba025f457e05725baa24dd, name=ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-11, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , RELEASE=wip-rishabh-testing-20240501.193033-9a047fe, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.build-date=20240430, GIT_BRANCH=HEAD, ceph=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_CLEAN=True) 2024-05-03T21:38:48.722 INFO:journalctl@ceph.osd.11.smithi138.stdout:May 03 21:38:48 smithi138 podman[69315]: 2024-05-03 21:38:48.528854882 +0000 UTC m=+5.990013448 container cleanup b6f3e5c4917bd7051246e4319d34bbd55faae5a35c4ecead4a06a1403ab95699 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:68e15ae152ef5c805c733e76da8fc1d3dd4ee669d5ba025f457e05725baa24dd, name=ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-11, ceph=True, maintainer=Guillaume Abrioux , GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, RELEASE=wip-rishabh-testing-20240501.193033-9a047fe, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240430, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD) 2024-05-03T21:38:48.723 INFO:journalctl@ceph.osd.11.smithi138.stdout:May 03 21:38:48 smithi138 bash[69315]: ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-11 2024-05-03T21:38:48.973 INFO:journalctl@ceph.osd.11.smithi138.stdout:May 03 21:38:48 smithi138 podman[69342]: 2024-05-03 21:38:48.721871734 +0000 UTC m=+0.533562509 container remove b6f3e5c4917bd7051246e4319d34bbd55faae5a35c4ecead4a06a1403ab95699 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:68e15ae152ef5c805c733e76da8fc1d3dd4ee669d5ba025f457e05725baa24dd, name=ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-11, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-rishabh-testing-20240501.193033-9a047fe, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240430, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.license=GPLv2) 2024-05-03T21:38:49.253 INFO:journalctl@ceph.osd.11.smithi138.stdout:May 03 21:38:49 smithi138 podman[69436]: 2024-05-03 21:38:49.148030758 +0000 UTC m=+0.112522579 container create 738bc06c2d25dc2c308e136ba5f5314cf248e2972dd2ef7306ba551daafe5d5c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:68e15ae152ef5c805c733e76da8fc1d3dd4ee669d5ba025f457e05725baa24dd, name=ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-11-deactivate, org.label-schema.build-date=20240430, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, RELEASE=wip-rishabh-testing-20240501.193033-9a047fe, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD) 2024-05-03T21:38:49.254 INFO:journalctl@ceph.osd.11.smithi138.stdout:May 03 21:38:49 smithi138 podman[69436]: 2024-05-03 21:38:49.053842138 +0000 UTC m=+0.018333927 image pull ee4d846202496ba6096593e8e8becc0700a5eb7ebd95536031187f5f337d4012 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:68e15ae152ef5c805c733e76da8fc1d3dd4ee669d5ba025f457e05725baa24dd 2024-05-03T21:38:49.574 INFO:journalctl@ceph.osd.11.smithi138.stdout:May 03 21:38:49 smithi138 podman[69436]: 2024-05-03 21:38:49.342546393 +0000 UTC m=+0.307038180 container init 738bc06c2d25dc2c308e136ba5f5314cf248e2972dd2ef7306ba551daafe5d5c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:68e15ae152ef5c805c733e76da8fc1d3dd4ee669d5ba025f457e05725baa24dd, name=ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-11-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240430, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, ceph=True, RELEASE=wip-rishabh-testing-20240501.193033-9a047fe) 2024-05-03T21:38:49.574 INFO:journalctl@ceph.osd.11.smithi138.stdout:May 03 21:38:49 smithi138 podman[69436]: 2024-05-03 21:38:49.345653746 +0000 UTC m=+0.310145528 container start 738bc06c2d25dc2c308e136ba5f5314cf248e2972dd2ef7306ba551daafe5d5c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:68e15ae152ef5c805c733e76da8fc1d3dd4ee669d5ba025f457e05725baa24dd, name=ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-11-deactivate, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240430, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, RELEASE=wip-rishabh-testing-20240501.193033-9a047fe, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_BRANCH=HEAD) 2024-05-03T21:38:49.574 INFO:journalctl@ceph.osd.11.smithi138.stdout:May 03 21:38:49 smithi138 podman[69436]: 2024-05-03 21:38:49.417602536 +0000 UTC m=+0.382094326 container attach 738bc06c2d25dc2c308e136ba5f5314cf248e2972dd2ef7306ba551daafe5d5c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:68e15ae152ef5c805c733e76da8fc1d3dd4ee669d5ba025f457e05725baa24dd, name=ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-11-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, RELEASE=wip-rishabh-testing-20240501.193033-9a047fe, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240430, GIT_BRANCH=HEAD, GIT_CLEAN=True) 2024-05-03T21:38:49.930 INFO:journalctl@ceph.osd.11.smithi138.stdout:May 03 21:38:49 smithi138 podman[69436]: 2024-05-03 21:38:49.644544586 +0000 UTC m=+0.609036372 container died 738bc06c2d25dc2c308e136ba5f5314cf248e2972dd2ef7306ba551daafe5d5c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:68e15ae152ef5c805c733e76da8fc1d3dd4ee669d5ba025f457e05725baa24dd, name=ceph-713bc57a-0991-11ef-bc97-c7b262605968-osd-11-deactivate, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, io.buildah.version=1.33.5, RELEASE=wip-rishabh-testing-20240501.193033-9a047fe, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, ceph=True, org.label-schema.build-date=20240430) 2024-05-03T21:38:50.225 DEBUG:teuthology.orchestra.run.smithi138:> sudo pkill -f 'journalctl -f -n 0 -u ceph-713bc57a-0991-11ef-bc97-c7b262605968@osd.11.service' 2024-05-03T21:38:50.263 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-03T21:38:50.263 INFO:tasks.cephadm.osd.11:Stopped osd.11 2024-05-03T21:38:50.264 INFO:tasks.cephadm.mds.a:Stopping mds.a... 2024-05-03T21:38:50.264 DEBUG:teuthology.orchestra.run.smithi067:> sudo systemctl stop ceph-713bc57a-0991-11ef-bc97-c7b262605968@mds.a 2024-05-03T21:38:50.604 INFO:journalctl@ceph.mds.a.smithi067.stdout:May 03 21:38:50 smithi067 systemd[1]: Stopping Ceph mds.a for 713bc57a-0991-11ef-bc97-c7b262605968... 2024-05-03T21:38:50.881 INFO:journalctl@ceph.mds.a.smithi067.stdout:May 03 21:38:50 smithi067 ceph-713bc57a-0991-11ef-bc97-c7b262605968-mds-a[69269]: 2024-05-03T21:38:50.601+0000 7f162b860700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-03T21:38:50.881 INFO:journalctl@ceph.mds.a.smithi067.stdout:May 03 21:38:50 smithi067 ceph-713bc57a-0991-11ef-bc97-c7b262605968-mds-a[69269]: 2024-05-03T21:38:50.601+0000 7f162b860700 -1 mds.a *** got signal Terminated *** 2024-05-03T21:38:52.221 INFO:journalctl@ceph.mds.a.smithi067.stdout:May 03 21:38:51 smithi067 podman[88438]: 2024-05-03 21:38:51.955968366 +0000 UTC m=+1.491506322 container died 5d2acf9b2666fa730cb4db415372fddb085ad7d5a2421cfb05eb3319bf7a4d4e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:68e15ae152ef5c805c733e76da8fc1d3dd4ee669d5ba025f457e05725baa24dd, name=ceph-713bc57a-0991-11ef-bc97-c7b262605968-mds-a, io.buildah.version=1.33.5, org.label-schema.build-date=20240430, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-rishabh-testing-20240501.193033-9a047fe, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=) 2024-05-03T21:38:52.596 INFO:journalctl@ceph.mds.a.smithi067.stdout:May 03 21:38:52 smithi067 podman[88438]: 2024-05-03 21:38:52.360880639 +0000 UTC m=+1.896418593 container cleanup 5d2acf9b2666fa730cb4db415372fddb085ad7d5a2421cfb05eb3319bf7a4d4e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:68e15ae152ef5c805c733e76da8fc1d3dd4ee669d5ba025f457e05725baa24dd, name=ceph-713bc57a-0991-11ef-bc97-c7b262605968-mds-a, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-rishabh-testing-20240501.193033-9a047fe, ceph=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.build-date=20240430, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD) 2024-05-03T21:38:52.596 INFO:journalctl@ceph.mds.a.smithi067.stdout:May 03 21:38:52 smithi067 bash[88438]: ceph-713bc57a-0991-11ef-bc97-c7b262605968-mds-a 2024-05-03T21:38:52.720 DEBUG:teuthology.orchestra.run.smithi067:> sudo pkill -f 'journalctl -f -n 0 -u ceph-713bc57a-0991-11ef-bc97-c7b262605968@mds.a.service' 2024-05-03T21:38:52.755 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-03T21:38:52.755 INFO:tasks.cephadm.mds.a:Stopped mds.a 2024-05-03T21:38:52.756 INFO:tasks.cephadm.mds.d:Stopping mds.d... 2024-05-03T21:38:52.756 DEBUG:teuthology.orchestra.run.smithi067:> sudo systemctl stop ceph-713bc57a-0991-11ef-bc97-c7b262605968@mds.d 2024-05-03T21:38:52.970 INFO:journalctl@ceph.mds.d.smithi067.stdout:May 03 21:38:52 smithi067 systemd[1]: Stopping Ceph mds.d for 713bc57a-0991-11ef-bc97-c7b262605968... 2024-05-03T21:38:53.470 INFO:journalctl@ceph.mds.d.smithi067.stdout:May 03 21:38:53 smithi067 ceph-713bc57a-0991-11ef-bc97-c7b262605968-mds-d[67994]: 2024-05-03T21:38:53.072+0000 7f5799794700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.d -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-03T21:38:53.471 INFO:journalctl@ceph.mds.d.smithi067.stdout:May 03 21:38:53 smithi067 ceph-713bc57a-0991-11ef-bc97-c7b262605968-mds-d[67994]: 2024-05-03T21:38:53.072+0000 7f5799794700 -1 mds.d *** got signal Terminated *** 2024-05-03T21:38:54.841 INFO:journalctl@ceph.mds.d.smithi067.stdout:May 03 21:38:54 smithi067 podman[88569]: 2024-05-03 21:38:54.580946393 +0000 UTC m=+1.605119638 container died 62837578d30949856a931a27ca9a9e8b4d7ff72bc95197aa16ad1b09ca97e3b7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:68e15ae152ef5c805c733e76da8fc1d3dd4ee669d5ba025f457e05725baa24dd, name=ceph-713bc57a-0991-11ef-bc97-c7b262605968-mds-d, org.label-schema.vendor=CentOS, RELEASE=wip-rishabh-testing-20240501.193033-9a047fe, ceph=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240430, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-05-03T21:38:55.183 INFO:journalctl@ceph.mds.d.smithi067.stdout:May 03 21:38:54 smithi067 podman[88569]: 2024-05-03 21:38:54.931064338 +0000 UTC m=+1.955237586 container cleanup 62837578d30949856a931a27ca9a9e8b4d7ff72bc95197aa16ad1b09ca97e3b7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:68e15ae152ef5c805c733e76da8fc1d3dd4ee669d5ba025f457e05725baa24dd, name=ceph-713bc57a-0991-11ef-bc97-c7b262605968-mds-d, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240430, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, RELEASE=wip-rishabh-testing-20240501.193033-9a047fe, ceph=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, io.buildah.version=1.33.5) 2024-05-03T21:38:55.183 INFO:journalctl@ceph.mds.d.smithi067.stdout:May 03 21:38:54 smithi067 bash[88569]: ceph-713bc57a-0991-11ef-bc97-c7b262605968-mds-d 2024-05-03T21:38:55.322 DEBUG:teuthology.orchestra.run.smithi067:> sudo pkill -f 'journalctl -f -n 0 -u ceph-713bc57a-0991-11ef-bc97-c7b262605968@mds.d.service' 2024-05-03T21:38:55.356 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-03T21:38:55.356 INFO:tasks.cephadm.mds.d:Stopped mds.d 2024-05-03T21:38:55.357 INFO:tasks.cephadm.mds.g:Stopping mds.g... 2024-05-03T21:38:55.357 DEBUG:teuthology.orchestra.run.smithi067:> sudo systemctl stop ceph-713bc57a-0991-11ef-bc97-c7b262605968@mds.g 2024-05-03T21:38:55.720 INFO:journalctl@ceph.mds.g.smithi067.stdout:May 03 21:38:55 smithi067 systemd[1]: Stopping Ceph mds.g for 713bc57a-0991-11ef-bc97-c7b262605968... 2024-05-03T21:38:55.721 INFO:journalctl@ceph.mds.g.smithi067.stdout:May 03 21:38:55 smithi067 ceph-713bc57a-0991-11ef-bc97-c7b262605968-mds-g[68795]: 2024-05-03T21:38:55.659+0000 7ff6433e5700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.g -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-03T21:38:55.721 INFO:journalctl@ceph.mds.g.smithi067.stdout:May 03 21:38:55 smithi067 ceph-713bc57a-0991-11ef-bc97-c7b262605968-mds-g[68795]: 2024-05-03T21:38:55.659+0000 7ff6433e5700 -1 mds.g *** got signal Terminated *** 2024-05-03T21:38:59.719 INFO:journalctl@ceph.mds.g.smithi067.stdout:May 03 21:38:59 smithi067 podman[88688]: 2024-05-03 21:38:59.37564337 +0000 UTC m=+3.796354235 container died 2b1a495821545b649dd7ae1995d82a10bc496ecd57f4a6b727fd69948d14a982 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:68e15ae152ef5c805c733e76da8fc1d3dd4ee669d5ba025f457e05725baa24dd, name=ceph-713bc57a-0991-11ef-bc97-c7b262605968-mds-g, GIT_BRANCH=HEAD, RELEASE=wip-rishabh-testing-20240501.193033-9a047fe, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.build-date=20240430, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, ceph=True) 2024-05-03T21:38:59.970 INFO:journalctl@ceph.mds.g.smithi067.stdout:May 03 21:38:59 smithi067 podman[88688]: 2024-05-03 21:38:59.801395998 +0000 UTC m=+4.222106865 container cleanup 2b1a495821545b649dd7ae1995d82a10bc496ecd57f4a6b727fd69948d14a982 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:68e15ae152ef5c805c733e76da8fc1d3dd4ee669d5ba025f457e05725baa24dd, name=ceph-713bc57a-0991-11ef-bc97-c7b262605968-mds-g, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.build-date=20240430, GIT_CLEAN=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, ceph=True, RELEASE=wip-rishabh-testing-20240501.193033-9a047fe, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0) 2024-05-03T21:38:59.971 INFO:journalctl@ceph.mds.g.smithi067.stdout:May 03 21:38:59 smithi067 bash[88688]: ceph-713bc57a-0991-11ef-bc97-c7b262605968-mds-g 2024-05-03T21:39:00.242 DEBUG:teuthology.orchestra.run.smithi067:> sudo pkill -f 'journalctl -f -n 0 -u ceph-713bc57a-0991-11ef-bc97-c7b262605968@mds.g.service' 2024-05-03T21:39:00.264 INFO:journalctl@ceph.mds.g.smithi067.stdout:May 03 21:39:00 smithi067 podman[88703]: 2024-05-03 21:39:00.010967149 +0000 UTC m=+0.634758178 container remove 2b1a495821545b649dd7ae1995d82a10bc496ecd57f4a6b727fd69948d14a982 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:68e15ae152ef5c805c733e76da8fc1d3dd4ee669d5ba025f457e05725baa24dd, name=ceph-713bc57a-0991-11ef-bc97-c7b262605968-mds-g, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240430, RELEASE=wip-rishabh-testing-20240501.193033-9a047fe, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD) 2024-05-03T21:39:00.264 INFO:journalctl@ceph.mds.g.smithi067.stdout:May 03 21:39:00 smithi067 systemd[1]: ceph-713bc57a-0991-11ef-bc97-c7b262605968@mds.g.service: Deactivated successfully. 2024-05-03T21:39:00.264 INFO:journalctl@ceph.mds.g.smithi067.stdout:May 03 21:39:00 smithi067 systemd[1]: Stopped Ceph mds.g for 713bc57a-0991-11ef-bc97-c7b262605968. 2024-05-03T21:39:00.264 INFO:journalctl@ceph.mds.g.smithi067.stdout:May 03 21:39:00 smithi067 systemd[1]: ceph-713bc57a-0991-11ef-bc97-c7b262605968@mds.g.service: Consumed 1.190s CPU time. 2024-05-03T21:39:00.275 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-03T21:39:00.275 INFO:tasks.cephadm.mds.g:Stopped mds.g 2024-05-03T21:39:00.276 INFO:tasks.cephadm.mds.j:Stopping mds.j... 2024-05-03T21:39:00.276 DEBUG:teuthology.orchestra.run.smithi067:> sudo systemctl stop ceph-713bc57a-0991-11ef-bc97-c7b262605968@mds.j 2024-05-03T21:39:00.572 INFO:journalctl@ceph.mds.j.smithi067.stdout:May 03 21:39:00 smithi067 systemd[1]: Stopping Ceph mds.j for 713bc57a-0991-11ef-bc97-c7b262605968... 2024-05-03T21:39:00.970 INFO:journalctl@ceph.mds.j.smithi067.stdout:May 03 21:39:00 smithi067 ceph-713bc57a-0991-11ef-bc97-c7b262605968-mds-j[67540]: 2024-05-03T21:39:00.570+0000 7f17d7655700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.j -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-03T21:39:00.971 INFO:journalctl@ceph.mds.j.smithi067.stdout:May 03 21:39:00 smithi067 ceph-713bc57a-0991-11ef-bc97-c7b262605968-mds-j[67540]: 2024-05-03T21:39:00.570+0000 7f17d7655700 -1 mds.j *** got signal Terminated *** 2024-05-03T21:39:02.221 INFO:journalctl@ceph.mds.j.smithi067.stdout:May 03 21:39:02 smithi067 podman[88812]: 2024-05-03 21:39:02.09721359 +0000 UTC m=+1.604317804 container died 8315d5c4c6383eefd8e959988a4b268e60740db1eb06f41ce49e543c4bf3bdae (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:68e15ae152ef5c805c733e76da8fc1d3dd4ee669d5ba025f457e05725baa24dd, name=ceph-713bc57a-0991-11ef-bc97-c7b262605968-mds-j, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_CLEAN=True, CEPH_POINT_RELEASE=, ceph=True, RELEASE=wip-rishabh-testing-20240501.193033-9a047fe, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240430, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-05-03T21:39:02.698 INFO:journalctl@ceph.mds.j.smithi067.stdout:May 03 21:39:02 smithi067 podman[88812]: 2024-05-03 21:39:02.521892788 +0000 UTC m=+2.028997000 container cleanup 8315d5c4c6383eefd8e959988a4b268e60740db1eb06f41ce49e543c4bf3bdae (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:68e15ae152ef5c805c733e76da8fc1d3dd4ee669d5ba025f457e05725baa24dd, name=ceph-713bc57a-0991-11ef-bc97-c7b262605968-mds-j, org.label-schema.build-date=20240430, org.label-schema.vendor=CentOS, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , RELEASE=wip-rishabh-testing-20240501.193033-9a047fe, org.label-schema.license=GPLv2) 2024-05-03T21:39:02.698 INFO:journalctl@ceph.mds.j.smithi067.stdout:May 03 21:39:02 smithi067 bash[88812]: ceph-713bc57a-0991-11ef-bc97-c7b262605968-mds-j 2024-05-03T21:39:02.911 DEBUG:teuthology.orchestra.run.smithi067:> sudo pkill -f 'journalctl -f -n 0 -u ceph-713bc57a-0991-11ef-bc97-c7b262605968@mds.j.service' 2024-05-03T21:39:02.943 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-03T21:39:02.943 INFO:tasks.cephadm.mds.j:Stopped mds.j 2024-05-03T21:39:02.943 INFO:tasks.cephadm.mds.b:Stopping mds.b... 2024-05-03T21:39:02.943 DEBUG:teuthology.orchestra.run.smithi082:> sudo systemctl stop ceph-713bc57a-0991-11ef-bc97-c7b262605968@mds.b 2024-05-03T21:39:03.245 INFO:journalctl@ceph.mds.b.smithi082.stdout:May 03 21:39:02 smithi082 systemd[1]: Stopping Ceph mds.b for 713bc57a-0991-11ef-bc97-c7b262605968... 2024-05-03T21:39:03.245 INFO:journalctl@ceph.mds.b.smithi082.stdout:May 03 21:39:03 smithi082 ceph-713bc57a-0991-11ef-bc97-c7b262605968-mds-b[63361]: 2024-05-03T21:39:03.208+0000 7feb59301700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-03T21:39:03.245 INFO:journalctl@ceph.mds.b.smithi082.stdout:May 03 21:39:03 smithi082 ceph-713bc57a-0991-11ef-bc97-c7b262605968-mds-b[63361]: 2024-05-03T21:39:03.208+0000 7feb59301700 -1 mds.b *** got signal Terminated *** 2024-05-03T21:39:07.245 INFO:journalctl@ceph.mds.b.smithi082.stdout:May 03 21:39:06 smithi082 podman[69136]: 2024-05-03 21:39:06.940722598 +0000 UTC m=+3.811039624 container died 3b7d6f82f0cbea2c4ba60dfefdb9b1b21c6a1cc98c6d07c52c94432300813928 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:68e15ae152ef5c805c733e76da8fc1d3dd4ee669d5ba025f457e05725baa24dd, name=ceph-713bc57a-0991-11ef-bc97-c7b262605968-mds-b, org.label-schema.license=GPLv2, GIT_CLEAN=True, RELEASE=wip-rishabh-testing-20240501.193033-9a047fe, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, io.buildah.version=1.33.5, org.label-schema.build-date=20240430, ceph=True, maintainer=Guillaume Abrioux ) 2024-05-03T21:39:07.568 INFO:journalctl@ceph.mds.b.smithi082.stdout:May 03 21:39:07 smithi082 podman[69136]: 2024-05-03 21:39:07.366677565 +0000 UTC m=+4.236994589 container cleanup 3b7d6f82f0cbea2c4ba60dfefdb9b1b21c6a1cc98c6d07c52c94432300813928 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:68e15ae152ef5c805c733e76da8fc1d3dd4ee669d5ba025f457e05725baa24dd, name=ceph-713bc57a-0991-11ef-bc97-c7b262605968-mds-b, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240430, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, RELEASE=wip-rishabh-testing-20240501.193033-9a047fe, GIT_BRANCH=HEAD, ceph=True) 2024-05-03T21:39:07.568 INFO:journalctl@ceph.mds.b.smithi082.stdout:May 03 21:39:07 smithi082 bash[69136]: ceph-713bc57a-0991-11ef-bc97-c7b262605968-mds-b 2024-05-03T21:39:07.797 DEBUG:teuthology.orchestra.run.smithi082:> sudo pkill -f 'journalctl -f -n 0 -u ceph-713bc57a-0991-11ef-bc97-c7b262605968@mds.b.service' 2024-05-03T21:39:07.818 INFO:journalctl@ceph.mds.b.smithi082.stdout:May 03 21:39:07 smithi082 podman[69163]: 2024-05-03 21:39:07.567720311 +0000 UTC m=+0.619443533 container remove 3b7d6f82f0cbea2c4ba60dfefdb9b1b21c6a1cc98c6d07c52c94432300813928 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:68e15ae152ef5c805c733e76da8fc1d3dd4ee669d5ba025f457e05725baa24dd, name=ceph-713bc57a-0991-11ef-bc97-c7b262605968-mds-b, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, ceph=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.license=GPLv2, org.label-schema.build-date=20240430, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, RELEASE=wip-rishabh-testing-20240501.193033-9a047fe, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, maintainer=Guillaume Abrioux ) 2024-05-03T21:39:07.818 INFO:journalctl@ceph.mds.b.smithi082.stdout:May 03 21:39:07 smithi082 systemd[1]: ceph-713bc57a-0991-11ef-bc97-c7b262605968@mds.b.service: Deactivated successfully. 2024-05-03T21:39:07.818 INFO:journalctl@ceph.mds.b.smithi082.stdout:May 03 21:39:07 smithi082 systemd[1]: Stopped Ceph mds.b for 713bc57a-0991-11ef-bc97-c7b262605968. 2024-05-03T21:39:07.819 INFO:journalctl@ceph.mds.b.smithi082.stdout:May 03 21:39:07 smithi082 systemd[1]: ceph-713bc57a-0991-11ef-bc97-c7b262605968@mds.b.service: Consumed 5.904s CPU time. 2024-05-03T21:39:07.831 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-03T21:39:07.831 INFO:tasks.cephadm.mds.b:Stopped mds.b 2024-05-03T21:39:07.832 INFO:tasks.cephadm.mds.e:Stopping mds.e... 2024-05-03T21:39:07.832 DEBUG:teuthology.orchestra.run.smithi082:> sudo systemctl stop ceph-713bc57a-0991-11ef-bc97-c7b262605968@mds.e 2024-05-03T21:39:08.146 INFO:journalctl@ceph.mds.e.smithi082.stdout:May 03 21:39:07 smithi082 systemd[1]: Stopping Ceph mds.e for 713bc57a-0991-11ef-bc97-c7b262605968... 2024-05-03T21:39:08.494 INFO:journalctl@ceph.mds.e.smithi082.stdout:May 03 21:39:08 smithi082 ceph-713bc57a-0991-11ef-bc97-c7b262605968-mds-e[62726]: 2024-05-03T21:39:08.144+0000 7f8998da5700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.e -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-03T21:39:08.495 INFO:journalctl@ceph.mds.e.smithi082.stdout:May 03 21:39:08 smithi082 ceph-713bc57a-0991-11ef-bc97-c7b262605968-mds-e[62726]: 2024-05-03T21:39:08.144+0000 7f8998da5700 -1 mds.e *** got signal Terminated *** 2024-05-03T21:39:11.245 INFO:journalctl@ceph.mds.e.smithi082.stdout:May 03 21:39:10 smithi082 podman[69269]: 2024-05-03 21:39:10.941926785 +0000 UTC m=+2.892795545 container died 0e80ad566fa7b1e9425fb74127087884563fb3251d408de7b32de6bf4d9e56dc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:68e15ae152ef5c805c733e76da8fc1d3dd4ee669d5ba025f457e05725baa24dd, name=ceph-713bc57a-0991-11ef-bc97-c7b262605968-mds-e, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, RELEASE=wip-rishabh-testing-20240501.193033-9a047fe, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240430, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, ceph=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.license=GPLv2, GIT_CLEAN=True) 2024-05-03T21:39:11.563 INFO:journalctl@ceph.mds.e.smithi082.stdout:May 03 21:39:11 smithi082 podman[69269]: 2024-05-03 21:39:11.369785798 +0000 UTC m=+3.320654556 container cleanup 0e80ad566fa7b1e9425fb74127087884563fb3251d408de7b32de6bf4d9e56dc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:68e15ae152ef5c805c733e76da8fc1d3dd4ee669d5ba025f457e05725baa24dd, name=ceph-713bc57a-0991-11ef-bc97-c7b262605968-mds-e, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.build-date=20240430, maintainer=Guillaume Abrioux , ceph=True, GIT_CLEAN=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, CEPH_POINT_RELEASE=, RELEASE=wip-rishabh-testing-20240501.193033-9a047fe) 2024-05-03T21:39:11.563 INFO:journalctl@ceph.mds.e.smithi082.stdout:May 03 21:39:11 smithi082 bash[69269]: ceph-713bc57a-0991-11ef-bc97-c7b262605968-mds-e 2024-05-03T21:39:11.789 DEBUG:teuthology.orchestra.run.smithi082:> sudo pkill -f 'journalctl -f -n 0 -u ceph-713bc57a-0991-11ef-bc97-c7b262605968@mds.e.service' 2024-05-03T21:39:11.821 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-03T21:39:11.821 INFO:tasks.cephadm.mds.e:Stopped mds.e 2024-05-03T21:39:11.822 INFO:tasks.cephadm.mds.h:Stopping mds.h... 2024-05-03T21:39:11.822 DEBUG:teuthology.orchestra.run.smithi082:> sudo systemctl stop ceph-713bc57a-0991-11ef-bc97-c7b262605968@mds.h 2024-05-03T21:39:12.141 INFO:journalctl@ceph.mds.h.smithi082.stdout:May 03 21:39:11 smithi082 systemd[1]: Stopping Ceph mds.h for 713bc57a-0991-11ef-bc97-c7b262605968... 2024-05-03T21:39:12.495 INFO:journalctl@ceph.mds.h.smithi082.stdout:May 03 21:39:12 smithi082 ceph-713bc57a-0991-11ef-bc97-c7b262605968-mds-h[63050]: 2024-05-03T21:39:12.139+0000 7ff8aa99a700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.h -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-03T21:39:12.495 INFO:journalctl@ceph.mds.h.smithi082.stdout:May 03 21:39:12 smithi082 ceph-713bc57a-0991-11ef-bc97-c7b262605968-mds-h[63050]: 2024-05-03T21:39:12.139+0000 7ff8aa99a700 -1 mds.h *** got signal Terminated *** 2024-05-03T21:39:15.245 INFO:journalctl@ceph.mds.h.smithi082.stdout:May 03 21:39:14 smithi082 podman[69391]: 2024-05-03 21:39:14.957284426 +0000 UTC m=+2.922045656 container died 27462ab2a791dfc6474de794265842ea5b137f9f32670f255094a984d736ce49 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:68e15ae152ef5c805c733e76da8fc1d3dd4ee669d5ba025f457e05725baa24dd, name=ceph-713bc57a-0991-11ef-bc97-c7b262605968-mds-h, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.build-date=20240430, io.buildah.version=1.33.5, RELEASE=wip-rishabh-testing-20240501.193033-9a047fe, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0) 2024-05-03T21:39:15.582 INFO:journalctl@ceph.mds.h.smithi082.stdout:May 03 21:39:15 smithi082 podman[69391]: 2024-05-03 21:39:15.331336844 +0000 UTC m=+3.296098117 container cleanup 27462ab2a791dfc6474de794265842ea5b137f9f32670f255094a984d736ce49 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:68e15ae152ef5c805c733e76da8fc1d3dd4ee669d5ba025f457e05725baa24dd, name=ceph-713bc57a-0991-11ef-bc97-c7b262605968-mds-h, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_CLEAN=True, RELEASE=wip-rishabh-testing-20240501.193033-9a047fe, org.label-schema.build-date=20240430) 2024-05-03T21:39:15.583 INFO:journalctl@ceph.mds.h.smithi082.stdout:May 03 21:39:15 smithi082 bash[69391]: ceph-713bc57a-0991-11ef-bc97-c7b262605968-mds-h 2024-05-03T21:39:15.708 DEBUG:teuthology.orchestra.run.smithi082:> sudo pkill -f 'journalctl -f -n 0 -u ceph-713bc57a-0991-11ef-bc97-c7b262605968@mds.h.service' 2024-05-03T21:39:15.739 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-03T21:39:15.739 INFO:tasks.cephadm.mds.h:Stopped mds.h 2024-05-03T21:39:15.739 INFO:tasks.cephadm.mds.k:Stopping mds.k... 2024-05-03T21:39:15.739 DEBUG:teuthology.orchestra.run.smithi082:> sudo systemctl stop ceph-713bc57a-0991-11ef-bc97-c7b262605968@mds.k 2024-05-03T21:39:15.994 INFO:journalctl@ceph.mds.k.smithi082.stdout:May 03 21:39:15 smithi082 systemd[1]: Stopping Ceph mds.k for 713bc57a-0991-11ef-bc97-c7b262605968... 2024-05-03T21:39:16.495 INFO:journalctl@ceph.mds.k.smithi082.stdout:May 03 21:39:16 smithi082 ceph-713bc57a-0991-11ef-bc97-c7b262605968-mds-k[62403]: 2024-05-03T21:39:16.084+0000 7f7955596700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.k -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-03T21:39:16.495 INFO:journalctl@ceph.mds.k.smithi082.stdout:May 03 21:39:16 smithi082 ceph-713bc57a-0991-11ef-bc97-c7b262605968-mds-k[62403]: 2024-05-03T21:39:16.084+0000 7f7955596700 -1 mds.k *** got signal Terminated *** 2024-05-03T21:39:19.245 INFO:journalctl@ceph.mds.k.smithi082.stdout:May 03 21:39:18 smithi082 podman[69512]: 2024-05-03 21:39:18.949809537 +0000 UTC m=+2.991677236 container died 9acc39735ce2143db050ee2ca62f4458c112d4ac6be4c8133414b02c5e81bd87 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:68e15ae152ef5c805c733e76da8fc1d3dd4ee669d5ba025f457e05725baa24dd, name=ceph-713bc57a-0991-11ef-bc97-c7b262605968-mds-k, CEPH_POINT_RELEASE=, GIT_CLEAN=True, ceph=True, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-rishabh-testing-20240501.193033-9a047fe, org.label-schema.license=GPLv2, org.label-schema.build-date=20240430, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD) 2024-05-03T21:39:19.745 INFO:journalctl@ceph.mds.k.smithi082.stdout:May 03 21:39:19 smithi082 podman[69512]: 2024-05-03 21:39:19.534393846 +0000 UTC m=+3.576261543 container cleanup 9acc39735ce2143db050ee2ca62f4458c112d4ac6be4c8133414b02c5e81bd87 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:68e15ae152ef5c805c733e76da8fc1d3dd4ee669d5ba025f457e05725baa24dd, name=ceph-713bc57a-0991-11ef-bc97-c7b262605968-mds-k, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, RELEASE=wip-rishabh-testing-20240501.193033-9a047fe, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240430, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, ceph=True) 2024-05-03T21:39:19.745 INFO:journalctl@ceph.mds.k.smithi082.stdout:May 03 21:39:19 smithi082 bash[69512]: ceph-713bc57a-0991-11ef-bc97-c7b262605968-mds-k 2024-05-03T21:39:19.907 DEBUG:teuthology.orchestra.run.smithi082:> sudo pkill -f 'journalctl -f -n 0 -u ceph-713bc57a-0991-11ef-bc97-c7b262605968@mds.k.service' 2024-05-03T21:39:19.937 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-03T21:39:19.938 INFO:tasks.cephadm.mds.k:Stopped mds.k 2024-05-03T21:39:19.938 INFO:tasks.cephadm.mds.c:Stopping mds.c... 2024-05-03T21:39:19.938 DEBUG:teuthology.orchestra.run.smithi138:> sudo systemctl stop ceph-713bc57a-0991-11ef-bc97-c7b262605968@mds.c 2024-05-03T21:39:20.240 INFO:journalctl@ceph.mds.c.smithi138.stdout:May 03 21:39:19 smithi138 systemd[1]: Stopping Ceph mds.c for 713bc57a-0991-11ef-bc97-c7b262605968... 2024-05-03T21:39:20.574 INFO:journalctl@ceph.mds.c.smithi138.stdout:May 03 21:39:20 smithi138 ceph-713bc57a-0991-11ef-bc97-c7b262605968-mds-c[63433]: 2024-05-03T21:39:20.238+0000 7fb0b0b86700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-03T21:39:20.574 INFO:journalctl@ceph.mds.c.smithi138.stdout:May 03 21:39:20 smithi138 ceph-713bc57a-0991-11ef-bc97-c7b262605968-mds-c[63433]: 2024-05-03T21:39:20.238+0000 7fb0b0b86700 -1 mds.c *** got signal Terminated *** 2024-05-03T21:39:23.239 INFO:journalctl@ceph.mds.c.smithi138.stdout:May 03 21:39:22 smithi138 podman[69543]: 2024-05-03 21:39:22.941081405 +0000 UTC m=+2.809034631 container died 34446d8bd98e59e5f9cedb5125dc87b14ef078c743820e4be7a2786c470966fe (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:68e15ae152ef5c805c733e76da8fc1d3dd4ee669d5ba025f457e05725baa24dd, name=ceph-713bc57a-0991-11ef-bc97-c7b262605968-mds-c, GIT_CLEAN=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, RELEASE=wip-rishabh-testing-20240501.193033-9a047fe, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240430, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, ceph=True) 2024-05-03T21:39:23.239 INFO:journalctl@ceph.mds.c.smithi138.stdout:May 03 21:39:23 smithi138 podman[69543]: 2024-05-03 21:39:23.2386296 +0000 UTC m=+3.106582833 container cleanup 34446d8bd98e59e5f9cedb5125dc87b14ef078c743820e4be7a2786c470966fe (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:68e15ae152ef5c805c733e76da8fc1d3dd4ee669d5ba025f457e05725baa24dd, name=ceph-713bc57a-0991-11ef-bc97-c7b262605968-mds-c, org.label-schema.build-date=20240430, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, ceph=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, RELEASE=wip-rishabh-testing-20240501.193033-9a047fe, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=) 2024-05-03T21:39:23.566 INFO:journalctl@ceph.mds.c.smithi138.stdout:May 03 21:39:23 smithi138 bash[69543]: ceph-713bc57a-0991-11ef-bc97-c7b262605968-mds-c 2024-05-03T21:39:23.567 INFO:journalctl@ceph.mds.c.smithi138.stdout:May 03 21:39:23 smithi138 podman[69569]: 2024-05-03 21:39:23.473665386 +0000 UTC m=+0.527592121 container remove 34446d8bd98e59e5f9cedb5125dc87b14ef078c743820e4be7a2786c470966fe (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:68e15ae152ef5c805c733e76da8fc1d3dd4ee669d5ba025f457e05725baa24dd, name=ceph-713bc57a-0991-11ef-bc97-c7b262605968-mds-c, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.build-date=20240430, org.label-schema.license=GPLv2, GIT_CLEAN=True, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, RELEASE=wip-rishabh-testing-20240501.193033-9a047fe) 2024-05-03T21:39:23.589 DEBUG:teuthology.orchestra.run.smithi138:> sudo pkill -f 'journalctl -f -n 0 -u ceph-713bc57a-0991-11ef-bc97-c7b262605968@mds.c.service' 2024-05-03T21:39:23.624 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-03T21:39:23.624 INFO:tasks.cephadm.mds.c:Stopped mds.c 2024-05-03T21:39:23.624 INFO:tasks.cephadm.mds.f:Stopping mds.f... 2024-05-03T21:39:23.624 DEBUG:teuthology.orchestra.run.smithi138:> sudo systemctl stop ceph-713bc57a-0991-11ef-bc97-c7b262605968@mds.f 2024-05-03T21:39:23.936 INFO:journalctl@ceph.mds.f.smithi138.stdout:May 03 21:39:23 smithi138 systemd[1]: Stopping Ceph mds.f for 713bc57a-0991-11ef-bc97-c7b262605968... 2024-05-03T21:39:24.323 INFO:journalctl@ceph.mds.f.smithi138.stdout:May 03 21:39:23 smithi138 ceph-713bc57a-0991-11ef-bc97-c7b262605968-mds-f[63123]: 2024-05-03T21:39:23.934+0000 7f79dabf3700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.f -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-03T21:39:24.324 INFO:journalctl@ceph.mds.f.smithi138.stdout:May 03 21:39:23 smithi138 ceph-713bc57a-0991-11ef-bc97-c7b262605968-mds-f[63123]: 2024-05-03T21:39:23.934+0000 7f79dabf3700 -1 mds.f *** got signal Terminated *** 2024-05-03T21:39:28.227 INFO:journalctl@ceph.mds.f.smithi138.stdout:May 03 21:39:27 smithi138 podman[69679]: 2024-05-03 21:39:27.940307921 +0000 UTC m=+4.102432783 container died 753ae948ece540c91b168cc1ed3ee732144027f044fc393e5da50d4ab292f72d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:68e15ae152ef5c805c733e76da8fc1d3dd4ee669d5ba025f457e05725baa24dd, name=ceph-713bc57a-0991-11ef-bc97-c7b262605968-mds-f, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240430, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, ceph=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, RELEASE=wip-rishabh-testing-20240501.193033-9a047fe) 2024-05-03T21:39:28.227 INFO:journalctl@ceph.mds.f.smithi138.stdout:May 03 21:39:28 smithi138 podman[69679]: 2024-05-03 21:39:28.226076195 +0000 UTC m=+4.388201057 container cleanup 753ae948ece540c91b168cc1ed3ee732144027f044fc393e5da50d4ab292f72d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:68e15ae152ef5c805c733e76da8fc1d3dd4ee669d5ba025f457e05725baa24dd, name=ceph-713bc57a-0991-11ef-bc97-c7b262605968-mds-f, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-rishabh-testing-20240501.193033-9a047fe, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240430, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, ceph=True, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.vendor=CentOS) 2024-05-03T21:39:28.478 INFO:journalctl@ceph.mds.f.smithi138.stdout:May 03 21:39:28 smithi138 bash[69679]: ceph-713bc57a-0991-11ef-bc97-c7b262605968-mds-f 2024-05-03T21:39:28.584 DEBUG:teuthology.orchestra.run.smithi138:> sudo pkill -f 'journalctl -f -n 0 -u ceph-713bc57a-0991-11ef-bc97-c7b262605968@mds.f.service' 2024-05-03T21:39:28.617 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-03T21:39:28.617 INFO:tasks.cephadm.mds.f:Stopped mds.f 2024-05-03T21:39:28.617 INFO:tasks.cephadm.mds.i:Stopping mds.i... 2024-05-03T21:39:28.617 DEBUG:teuthology.orchestra.run.smithi138:> sudo systemctl stop ceph-713bc57a-0991-11ef-bc97-c7b262605968@mds.i 2024-05-03T21:39:28.823 INFO:journalctl@ceph.mds.i.smithi138.stdout:May 03 21:39:28 smithi138 systemd[1]: Stopping Ceph mds.i for 713bc57a-0991-11ef-bc97-c7b262605968... 2024-05-03T21:39:29.323 INFO:journalctl@ceph.mds.i.smithi138.stdout:May 03 21:39:28 smithi138 ceph-713bc57a-0991-11ef-bc97-c7b262605968-mds-i[63754]: 2024-05-03T21:39:28.913+0000 7fe629d22700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.i -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-03T21:39:29.324 INFO:journalctl@ceph.mds.i.smithi138.stdout:May 03 21:39:28 smithi138 ceph-713bc57a-0991-11ef-bc97-c7b262605968-mds-i[63754]: 2024-05-03T21:39:28.913+0000 7fe629d22700 -1 mds.i *** got signal Terminated *** 2024-05-03T21:39:31.207 INFO:journalctl@ceph.mds.i.smithi138.stdout:May 03 21:39:30 smithi138 podman[69801]: 2024-05-03 21:39:30.943097523 +0000 UTC m=+2.114377224 container died c4cc8b05b928e730689fa57d76ec19a549bb0669f2dfbc35e20ae2a0e8ee107e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:68e15ae152ef5c805c733e76da8fc1d3dd4ee669d5ba025f457e05725baa24dd, name=ceph-713bc57a-0991-11ef-bc97-c7b262605968-mds-i, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.license=GPLv2, GIT_CLEAN=True, io.buildah.version=1.33.5, RELEASE=wip-rishabh-testing-20240501.193033-9a047fe, GIT_BRANCH=HEAD, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240430, org.label-schema.vendor=CentOS) 2024-05-03T21:39:31.474 INFO:journalctl@ceph.mds.i.smithi138.stdout:May 03 21:39:31 smithi138 podman[69801]: 2024-05-03 21:39:31.281043262 +0000 UTC m=+2.452322958 container cleanup c4cc8b05b928e730689fa57d76ec19a549bb0669f2dfbc35e20ae2a0e8ee107e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:68e15ae152ef5c805c733e76da8fc1d3dd4ee669d5ba025f457e05725baa24dd, name=ceph-713bc57a-0991-11ef-bc97-c7b262605968-mds-i, RELEASE=wip-rishabh-testing-20240501.193033-9a047fe, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240430, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.schema-version=1.0, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=) 2024-05-03T21:39:31.475 INFO:journalctl@ceph.mds.i.smithi138.stdout:May 03 21:39:31 smithi138 bash[69801]: ceph-713bc57a-0991-11ef-bc97-c7b262605968-mds-i 2024-05-03T21:39:31.696 DEBUG:teuthology.orchestra.run.smithi138:> sudo pkill -f 'journalctl -f -n 0 -u ceph-713bc57a-0991-11ef-bc97-c7b262605968@mds.i.service' 2024-05-03T21:39:31.729 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-03T21:39:31.730 INFO:tasks.cephadm.mds.i:Stopped mds.i 2024-05-03T21:39:31.730 INFO:tasks.cephadm.mds.l:Stopping mds.l... 2024-05-03T21:39:31.730 DEBUG:teuthology.orchestra.run.smithi138:> sudo systemctl stop ceph-713bc57a-0991-11ef-bc97-c7b262605968@mds.l 2024-05-03T21:39:32.035 INFO:journalctl@ceph.mds.l.smithi138.stdout:May 03 21:39:31 smithi138 systemd[1]: Stopping Ceph mds.l for 713bc57a-0991-11ef-bc97-c7b262605968... 2024-05-03T21:39:32.286 INFO:journalctl@ceph.mds.l.smithi138.stdout:May 03 21:39:32 smithi138 ceph-713bc57a-0991-11ef-bc97-c7b262605968-mds-l[62813]: 2024-05-03T21:39:32.033+0000 7f20e0676700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.l -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-03T21:39:32.287 INFO:journalctl@ceph.mds.l.smithi138.stdout:May 03 21:39:32 smithi138 ceph-713bc57a-0991-11ef-bc97-c7b262605968-mds-l[62813]: 2024-05-03T21:39:32.033+0000 7f20e0676700 -1 mds.l *** got signal Terminated *** 2024-05-03T21:39:34.911 INFO:journalctl@ceph.mds.l.smithi138.stdout:May 03 21:39:34 smithi138 podman[69924]: 2024-05-03 21:39:34.646389567 +0000 UTC m=+2.704844075 container died 5f5854e648980748e880e5dde6d9246965c6d3619f3d86924e319aa22490960d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:68e15ae152ef5c805c733e76da8fc1d3dd4ee669d5ba025f457e05725baa24dd, name=ceph-713bc57a-0991-11ef-bc97-c7b262605968-mds-l, org.label-schema.schema-version=1.0, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, ceph=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, RELEASE=wip-rishabh-testing-20240501.193033-9a047fe, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240430) 2024-05-03T21:39:34.911 INFO:journalctl@ceph.mds.l.smithi138.stdout:May 03 21:39:34 smithi138 podman[69924]: 2024-05-03 21:39:34.910233968 +0000 UTC m=+2.968688460 container cleanup 5f5854e648980748e880e5dde6d9246965c6d3619f3d86924e319aa22490960d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:68e15ae152ef5c805c733e76da8fc1d3dd4ee669d5ba025f457e05725baa24dd, name=ceph-713bc57a-0991-11ef-bc97-c7b262605968-mds-l, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.build-date=20240430, GIT_CLEAN=True, RELEASE=wip-rishabh-testing-20240501.193033-9a047fe) 2024-05-03T21:39:35.319 INFO:journalctl@ceph.mds.l.smithi138.stdout:May 03 21:39:34 smithi138 bash[69924]: ceph-713bc57a-0991-11ef-bc97-c7b262605968-mds-l 2024-05-03T21:39:35.319 INFO:journalctl@ceph.mds.l.smithi138.stdout:May 03 21:39:35 smithi138 podman[69938]: 2024-05-03 21:39:35.086299237 +0000 UTC m=+0.435687109 container remove 5f5854e648980748e880e5dde6d9246965c6d3619f3d86924e319aa22490960d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:68e15ae152ef5c805c733e76da8fc1d3dd4ee669d5ba025f457e05725baa24dd, name=ceph-713bc57a-0991-11ef-bc97-c7b262605968-mds-l, org.label-schema.build-date=20240430, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_CLEAN=True, RELEASE=wip-rishabh-testing-20240501.193033-9a047fe, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, ceph=True, CEPH_POINT_RELEASE=, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD) 2024-05-03T21:39:35.344 DEBUG:teuthology.orchestra.run.smithi138:> sudo pkill -f 'journalctl -f -n 0 -u ceph-713bc57a-0991-11ef-bc97-c7b262605968@mds.l.service' 2024-05-03T21:39:35.375 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-03T21:39:35.375 INFO:tasks.cephadm.mds.l:Stopped mds.l 2024-05-03T21:39:35.375 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 713bc57a-0991-11ef-bc97-c7b262605968 --force --keep-logs 2024-05-03T21:39:35.578 INFO:teuthology.orchestra.run.smithi067.stdout:Deleting cluster with fsid: 713bc57a-0991-11ef-bc97-c7b262605968 2024-05-03T21:39:38.104 DEBUG:teuthology.orchestra.run.smithi082:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 713bc57a-0991-11ef-bc97-c7b262605968 --force --keep-logs 2024-05-03T21:39:38.294 INFO:teuthology.orchestra.run.smithi082.stdout:Deleting cluster with fsid: 713bc57a-0991-11ef-bc97-c7b262605968 2024-05-03T21:39:40.661 DEBUG:teuthology.orchestra.run.smithi138:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 713bc57a-0991-11ef-bc97-c7b262605968 --force --keep-logs 2024-05-03T21:39:40.850 INFO:teuthology.orchestra.run.smithi138.stdout:Deleting cluster with fsid: 713bc57a-0991-11ef-bc97-c7b262605968 2024-05-03T21:39:43.179 DEBUG:teuthology.orchestra.run.smithi067:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-05-03T21:39:43.212 DEBUG:teuthology.orchestra.run.smithi082:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-05-03T21:39:43.243 DEBUG:teuthology.orchestra.run.smithi138:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-05-03T21:39:43.268 INFO:tasks.cephadm:Archiving crash dumps... 2024-05-03T21:39:43.270 DEBUG:teuthology.misc:Transferring archived files from smithi067:/var/lib/ceph/713bc57a-0991-11ef-bc97-c7b262605968/crash to /home/teuthworker/archive/rishabh-2024-05-03_13:56:19-fs-wip-rishabh-testing-20240501.193033-testing-default-smithi/7687512/remote/smithi067/crash 2024-05-03T21:39:43.271 DEBUG:teuthology.orchestra.run.smithi067:> sudo tar c -f - -C /var/lib/ceph/713bc57a-0991-11ef-bc97-c7b262605968/crash -- . 2024-05-03T21:39:43.310 INFO:teuthology.orchestra.run.smithi067.stderr:tar: /var/lib/ceph/713bc57a-0991-11ef-bc97-c7b262605968/crash: Cannot open: No such file or directory 2024-05-03T21:39:43.310 INFO:teuthology.orchestra.run.smithi067.stderr:tar: Error is not recoverable: exiting now 2024-05-03T21:39:43.312 DEBUG:teuthology.misc:Transferring archived files from smithi082:/var/lib/ceph/713bc57a-0991-11ef-bc97-c7b262605968/crash to /home/teuthworker/archive/rishabh-2024-05-03_13:56:19-fs-wip-rishabh-testing-20240501.193033-testing-default-smithi/7687512/remote/smithi082/crash 2024-05-03T21:39:43.313 DEBUG:teuthology.orchestra.run.smithi082:> sudo tar c -f - -C /var/lib/ceph/713bc57a-0991-11ef-bc97-c7b262605968/crash -- . 2024-05-03T21:39:43.454 INFO:teuthology.orchestra.run.smithi082.stderr:tar: /var/lib/ceph/713bc57a-0991-11ef-bc97-c7b262605968/crash: Cannot open: No such file or directory 2024-05-03T21:39:43.455 INFO:teuthology.orchestra.run.smithi082.stderr:tar: Error is not recoverable: exiting now 2024-05-03T21:39:43.456 DEBUG:teuthology.misc:Transferring archived files from smithi138:/var/lib/ceph/713bc57a-0991-11ef-bc97-c7b262605968/crash to /home/teuthworker/archive/rishabh-2024-05-03_13:56:19-fs-wip-rishabh-testing-20240501.193033-testing-default-smithi/7687512/remote/smithi138/crash 2024-05-03T21:39:43.457 DEBUG:teuthology.orchestra.run.smithi138:> sudo tar c -f - -C /var/lib/ceph/713bc57a-0991-11ef-bc97-c7b262605968/crash -- . 2024-05-03T21:39:43.512 INFO:teuthology.orchestra.run.smithi138.stderr:tar: /var/lib/ceph/713bc57a-0991-11ef-bc97-c7b262605968/crash: Cannot open: No such file or directory 2024-05-03T21:39:43.513 INFO:teuthology.orchestra.run.smithi138.stderr:tar: Error is not recoverable: exiting now 2024-05-03T21:39:43.514 INFO:tasks.cephadm:Checking cluster log for badness... 2024-05-03T21:39:43.514 DEBUG:teuthology.orchestra.run.smithi067:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/713bc57a-0991-11ef-bc97-c7b262605968/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v FS_DEGRADED | egrep -v FS_INLINE_DATA_DEPRECATED | egrep -v FS_WITH_FAILED_MDS | egrep -v MDS_ALL_DOWN | egrep -v MDS_DAMAGE | egrep -v MDS_DEGRADED | egrep -v MDS_FAILED | egrep -v MDS_INSUFFICIENT_STANDBY | egrep -v MDS_UP_LESS_THAN_MAX | egrep -v 'filesystem is online with fewer MDS than max_mds' | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'do not have an application enabled' | egrep -v 'overall HEALTH_' | egrep -v 'Replacing daemon' | egrep -v 'deprecated feature inline_data' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | egrep -v PG_AVAILABILITY | egrep -v PG_DEGRADED | egrep -v 'Reduced data availability' | head -n 1 2024-05-03T21:39:43.550 INFO:tasks.cephadm:Compressing logs... 2024-05-03T21:39:43.550 DEBUG:teuthology.orchestra.run.smithi067:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-05-03T21:39:43.593 DEBUG:teuthology.orchestra.run.smithi082:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-05-03T21:39:43.596 DEBUG:teuthology.orchestra.run.smithi138:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-05-03T21:39:43.617 INFO:teuthology.orchestra.run.smithi067.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-05-03T21:39:43.618 INFO:teuthology.orchestra.run.smithi067.stderr:: No such file or directory 2024-05-03T21:39:43.618 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/713bc57a-0991-11ef-bc97-c7b262605968/ceph.log 2024-05-03T21:39:43.619 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/713bc57a-0991-11ef-bc97-c7b262605968/ceph.audit.log 2024-05-03T21:39:43.619 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/cephadm.log: /var/log/ceph/713bc57a-0991-11ef-bc97-c7b262605968/ceph.log: gzip -5 --verbose -- /var/log/ceph/713bc57a-0991-11ef-bc97-c7b262605968/ceph-mds.g.log 2024-05-03T21:39:43.619 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/713bc57a-0991-11ef-bc97-c7b262605968/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/713bc57a-0991-11ef-bc97-c7b262605968/ceph-mds.j.log 2024-05-03T21:39:43.619 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/713bc57a-0991-11ef-bc97-c7b262605968/ceph-mds.g.log: gzip -5 --verbose -- /var/log/ceph/713bc57a-0991-11ef-bc97-c7b262605968/ceph-osd.3.log 2024-05-03T21:39:43.619 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/713bc57a-0991-11ef-bc97-c7b262605968/ceph-mds.j.log: gzip -5 --verbose -- /var/log/ceph/713bc57a-0991-11ef-bc97-c7b262605968/ceph-osd.0.log 2024-05-03T21:39:43.630 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/713bc57a-0991-11ef-bc97-c7b262605968/ceph-osd.3.log: gzip -5 --verbose -- /var/log/ceph/713bc57a-0991-11ef-bc97-c7b262605968/ceph-mgr.x.log 2024-05-03T21:39:43.631 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/713bc57a-0991-11ef-bc97-c7b262605968/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/713bc57a-0991-11ef-bc97-c7b262605968/ceph-mon.a.log 2024-05-03T21:39:43.631 INFO:teuthology.orchestra.run.smithi067.stderr: 88.9% -- replaced with /var/log/ceph/713bc57a-0991-11ef-bc97-c7b262605968/ceph.log.gz 2024-05-03T21:39:43.631 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/713bc57a-0991-11ef-bc97-c7b262605968/ceph-mgr.x.log: 90.4% -- replaced with /var/log/ceph/713bc57a-0991-11ef-bc97-c7b262605968/ceph.audit.log.gz 2024-05-03T21:39:43.631 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/713bc57a-0991-11ef-bc97-c7b262605968/ceph-osd.6.log 2024-05-03T21:39:43.631 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/713bc57a-0991-11ef-bc97-c7b262605968/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/713bc57a-0991-11ef-bc97-c7b262605968/ceph.cephadm.log 2024-05-03T21:39:43.631 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/713bc57a-0991-11ef-bc97-c7b262605968/ceph-osd.6.log: 94.6% -- replaced with /var/log/ceph/713bc57a-0991-11ef-bc97-c7b262605968/ceph-mds.g.log.gz 2024-05-03T21:39:43.631 INFO:teuthology.orchestra.run.smithi067.stderr: 94.6% -- replaced with /var/log/ceph/713bc57a-0991-11ef-bc97-c7b262605968/ceph-mds.j.log.gz 2024-05-03T21:39:43.631 INFO:teuthology.orchestra.run.smithi067.stderr: 93.6% -- replaced with /var/log/ceph/cephadm.log.gz 2024-05-03T21:39:43.635 INFO:teuthology.orchestra.run.smithi082.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-05-03T21:39:43.635 INFO:teuthology.orchestra.run.smithi082.stderr:: No such file or directory 2024-05-03T21:39:43.636 INFO:teuthology.orchestra.run.smithi082.stderr:gzip -5 --verbose -- /var/log/ceph/713bc57a-0991-11ef-bc97-c7b262605968/ceph.log 2024-05-03T21:39:43.636 INFO:teuthology.orchestra.run.smithi082.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/713bc57a-0991-11ef-bc97-c7b262605968/ceph.audit.log 2024-05-03T21:39:43.636 INFO:teuthology.orchestra.run.smithi082.stderr:gzip -5 --verbose --/var/log/ceph/713bc57a-0991-11ef-bc97-c7b262605968/ceph.log: /var/log/ceph/713bc57a-0991-11ef-bc97-c7b262605968/ceph-osd.7.log 2024-05-03T21:39:43.636 INFO:teuthology.orchestra.run.smithi082.stderr:/var/log/ceph/713bc57a-0991-11ef-bc97-c7b262605968/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/713bc57a-0991-11ef-bc97-c7b262605968/ceph-mds.b.log 2024-05-03T21:39:43.636 INFO:teuthology.orchestra.run.smithi082.stderr:/var/log/ceph/713bc57a-0991-11ef-bc97-c7b262605968/ceph-osd.7.log: gzip -5 --verbose -- /var/log/ceph/713bc57a-0991-11ef-bc97-c7b262605968/ceph-osd.4.log 2024-05-03T21:39:43.636 INFO:teuthology.orchestra.run.smithi082.stderr: 91.8% -- replaced with /var/log/ceph/cephadm.log.gz 2024-05-03T21:39:43.636 INFO:teuthology.orchestra.run.smithi082.stderr:gzip -5 --verbose --/var/log/ceph/713bc57a-0991-11ef-bc97-c7b262605968/ceph-mds.b.log: /var/log/ceph/713bc57a-0991-11ef-bc97-c7b262605968/ceph-osd.1.log 2024-05-03T21:39:43.637 INFO:teuthology.orchestra.run.smithi082.stderr:/var/log/ceph/713bc57a-0991-11ef-bc97-c7b262605968/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/713bc57a-0991-11ef-bc97-c7b262605968/ceph-mgr.y.log 2024-05-03T21:39:43.637 INFO:teuthology.orchestra.run.smithi082.stderr:/var/log/ceph/713bc57a-0991-11ef-bc97-c7b262605968/ceph-osd.1.log: gzip 90.6% -- replaced with /var/log/ceph/713bc57a-0991-11ef-bc97-c7b262605968/ceph.audit.log.gz 2024-05-03T21:39:43.637 INFO:teuthology.orchestra.run.smithi082.stderr: -5 --verbose -- /var/log/ceph/713bc57a-0991-11ef-bc97-c7b262605968/ceph-mds.h.log 2024-05-03T21:39:43.637 INFO:teuthology.orchestra.run.smithi082.stderr: 89.1% -- replaced with /var/log/ceph/713bc57a-0991-11ef-bc97-c7b262605968/ceph.log.gz 2024-05-03T21:39:43.637 INFO:teuthology.orchestra.run.smithi082.stderr:/var/log/ceph/713bc57a-0991-11ef-bc97-c7b262605968/ceph-mgr.y.log: gzip -5 --verbose -- /var/log/ceph/713bc57a-0991-11ef-bc97-c7b262605968/ceph-osd.10.log 2024-05-03T21:39:43.637 INFO:teuthology.orchestra.run.smithi082.stderr:/var/log/ceph/713bc57a-0991-11ef-bc97-c7b262605968/ceph-mds.h.log: gzip -5 --verbose -- /var/log/ceph/713bc57a-0991-11ef-bc97-c7b262605968/ceph.cephadm.log 2024-05-03T21:39:43.637 INFO:teuthology.orchestra.run.smithi082.stderr:/var/log/ceph/713bc57a-0991-11ef-bc97-c7b262605968/ceph-osd.10.log: gzip -5 --verbose -- /var/log/ceph/713bc57a-0991-11ef-bc97-c7b262605968/ceph-mds.e.log 2024-05-03T21:39:43.637 INFO:teuthology.orchestra.run.smithi082.stderr:/var/log/ceph/713bc57a-0991-11ef-bc97-c7b262605968/ceph.cephadm.log: 82.4% -- replaced with /var/log/ceph/713bc57a-0991-11ef-bc97-c7b262605968/ceph.cephadm.log.gz 2024-05-03T21:39:43.638 INFO:teuthology.orchestra.run.smithi082.stderr:gzip -5 --verbose -- /var/log/ceph/713bc57a-0991-11ef-bc97-c7b262605968/ceph-volume.log 2024-05-03T21:39:43.638 INFO:teuthology.orchestra.run.smithi082.stderr:/var/log/ceph/713bc57a-0991-11ef-bc97-c7b262605968/ceph-mds.e.log: 93.1% -- replaced with /var/log/ceph/713bc57a-0991-11ef-bc97-c7b262605968/ceph-mgr.y.log.gz 2024-05-03T21:39:43.642 INFO:teuthology.orchestra.run.smithi082.stderr:gzip -5 --verbose -- /var/log/ceph/713bc57a-0991-11ef-bc97-c7b262605968/ceph-mon.b.log 2024-05-03T21:39:43.643 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/713bc57a-0991-11ef-bc97-c7b262605968/ceph-mds.a.log 2024-05-03T21:39:43.643 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/713bc57a-0991-11ef-bc97-c7b262605968/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/713bc57a-0991-11ef-bc97-c7b262605968/ceph-volume.log 2024-05-03T21:39:43.643 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/713bc57a-0991-11ef-bc97-c7b262605968/ceph-mds.a.log: 82.8% -- replaced with /var/log/ceph/713bc57a-0991-11ef-bc97-c7b262605968/ceph.cephadm.log.gz 2024-05-03T21:39:43.643 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/713bc57a-0991-11ef-bc97-c7b262605968/ceph-mds.d.log 2024-05-03T21:39:43.643 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/713bc57a-0991-11ef-bc97-c7b262605968/ceph-volume.log: gzip -5 --verbose -- /var/log/ceph/713bc57a-0991-11ef-bc97-c7b262605968/ceph-osd.9.log 2024-05-03T21:39:43.643 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/713bc57a-0991-11ef-bc97-c7b262605968/ceph-mds.d.log: 94.6% -- replaced with /var/log/ceph/713bc57a-0991-11ef-bc97-c7b262605968/ceph-mds.a.log.gz 2024-05-03T21:39:43.643 INFO:teuthology.orchestra.run.smithi067.stderr: 94.6% -- replaced with /var/log/ceph/713bc57a-0991-11ef-bc97-c7b262605968/ceph-mds.d.log.gz 2024-05-03T21:39:43.644 INFO:teuthology.orchestra.run.smithi138.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log: No such file or directory 2024-05-03T21:39:43.644 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-05-03T21:39:43.644 INFO:teuthology.orchestra.run.smithi138.stderr:gzip -5 --verbose -- /var/log/ceph/713bc57a-0991-11ef-bc97-c7b262605968/ceph.log 2024-05-03T21:39:43.644 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/713bc57a-0991-11ef-bc97-c7b262605968/ceph-osd.2.log 2024-05-03T21:39:43.644 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/713bc57a-0991-11ef-bc97-c7b262605968/ceph.log: gzip -5 --verbose -- /var/log/ceph/713bc57a-0991-11ef-bc97-c7b262605968/ceph-mds.f.log 2024-05-03T21:39:43.644 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/713bc57a-0991-11ef-bc97-c7b262605968/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/713bc57a-0991-11ef-bc97-c7b262605968/ceph.audit.log 2024-05-03T21:39:43.644 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/713bc57a-0991-11ef-bc97-c7b262605968/ceph-mds.f.log: gzip -5 --verbose -- /var/log/ceph/713bc57a-0991-11ef-bc97-c7b262605968/ceph-osd.5.log 2024-05-03T21:39:43.645 INFO:teuthology.orchestra.run.smithi138.stderr: 92.1% -- replaced with /var/log/ceph/cephadm.log.gz 2024-05-03T21:39:43.645 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/713bc57a-0991-11ef-bc97-c7b262605968/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/713bc57a-0991-11ef-bc97-c7b262605968/ceph-mds.c.log 2024-05-03T21:39:43.645 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/713bc57a-0991-11ef-bc97-c7b262605968/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/713bc57a-0991-11ef-bc97-c7b262605968/ceph-mon.c.log 2024-05-03T21:39:43.645 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/713bc57a-0991-11ef-bc97-c7b262605968/ceph-mds.c.log: gzip -5 --verbose -- /var/log/ceph/713bc57a-0991-11ef-bc97-c7b262605968/ceph-mgr.z.log 2024-05-03T21:39:43.645 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/713bc57a-0991-11ef-bc97-c7b262605968/ceph-mon.c.log: gzip -5 --verbose -- /var/log/ceph/713bc57a-0991-11ef-bc97-c7b262605968/ceph-mds.i.log 2024-05-03T21:39:43.645 INFO:teuthology.orchestra.run.smithi138.stderr: 89.0% -- replaced with /var/log/ceph/713bc57a-0991-11ef-bc97-c7b262605968/ceph.log.gz 2024-05-03T21:39:43.645 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/713bc57a-0991-11ef-bc97-c7b262605968/ceph-mgr.z.log: gzip -5 --verbose -- /var/log/ceph/713bc57a-0991-11ef-bc97-c7b262605968/ceph.cephadm.log 2024-05-03T21:39:43.645 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/713bc57a-0991-11ef-bc97-c7b262605968/ceph-mds.i.log: 90.6% -- replaced with /var/log/ceph/713bc57a-0991-11ef-bc97-c7b262605968/ceph.audit.log.gz 2024-05-03T21:39:43.645 INFO:teuthology.orchestra.run.smithi138.stderr:gzip -5 --verbose -- /var/log/ceph/713bc57a-0991-11ef-bc97-c7b262605968/ceph-osd.11.log 2024-05-03T21:39:43.646 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/713bc57a-0991-11ef-bc97-c7b262605968/ceph.cephadm.log: 82.4% -- replaced with /var/log/ceph/713bc57a-0991-11ef-bc97-c7b262605968/ceph.cephadm.log.gz 2024-05-03T21:39:43.646 INFO:teuthology.orchestra.run.smithi138.stderr:gzip -5 --verbose -- /var/log/ceph/713bc57a-0991-11ef-bc97-c7b262605968/ceph-osd.8.log 2024-05-03T21:39:43.646 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/713bc57a-0991-11ef-bc97-c7b262605968/ceph-osd.11.log: 94.6% -- replaced with /var/log/ceph/713bc57a-0991-11ef-bc97-c7b262605968/ceph-mds.f.log.gz 2024-05-03T21:39:43.646 INFO:teuthology.orchestra.run.smithi138.stderr: 93.2% -- replaced with /var/log/ceph/713bc57a-0991-11ef-bc97-c7b262605968/ceph-mgr.z.log.gz 2024-05-03T21:39:43.646 INFO:teuthology.orchestra.run.smithi138.stderr:gzip -5 --verbose -- /var/log/ceph/713bc57a-0991-11ef-bc97-c7b262605968/ceph-volume.log 2024-05-03T21:39:43.646 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/713bc57a-0991-11ef-bc97-c7b262605968/ceph-osd.8.log: gzip -5 --verbose -- /var/log/ceph/713bc57a-0991-11ef-bc97-c7b262605968/ceph-mds.l.log 2024-05-03T21:39:43.647 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/713bc57a-0991-11ef-bc97-c7b262605968/ceph-osd.9.log: 91.0% -- replaced with /var/log/ceph/713bc57a-0991-11ef-bc97-c7b262605968/ceph-volume.log.gz 2024-05-03T21:39:43.652 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/713bc57a-0991-11ef-bc97-c7b262605968/ceph-volume.log: /var/log/ceph/713bc57a-0991-11ef-bc97-c7b262605968/ceph-mds.l.log: 91.0% -- replaced with /var/log/ceph/713bc57a-0991-11ef-bc97-c7b262605968/ceph-volume.log.gz 2024-05-03T21:39:43.652 INFO:teuthology.orchestra.run.smithi138.stderr: 94.6% -- replaced with /var/log/ceph/713bc57a-0991-11ef-bc97-c7b262605968/ceph-mds.l.log.gz 2024-05-03T21:39:43.653 INFO:teuthology.orchestra.run.smithi082.stderr:/var/log/ceph/713bc57a-0991-11ef-bc97-c7b262605968/ceph-volume.log: gzip -5 --verbose -- /var/log/ceph/713bc57a-0991-11ef-bc97-c7b262605968/ceph-mds.k.log 2024-05-03T21:39:43.655 INFO:teuthology.orchestra.run.smithi082.stderr:/var/log/ceph/713bc57a-0991-11ef-bc97-c7b262605968/ceph-mon.b.log: 91.0% -- replaced with /var/log/ceph/713bc57a-0991-11ef-bc97-c7b262605968/ceph-volume.log.gz 2024-05-03T21:39:43.678 INFO:teuthology.orchestra.run.smithi138.stderr: 94.6% -- replaced with /var/log/ceph/713bc57a-0991-11ef-bc97-c7b262605968/ceph-mds.c.log.gz 2024-05-03T21:39:43.691 INFO:teuthology.orchestra.run.smithi082.stderr:/var/log/ceph/713bc57a-0991-11ef-bc97-c7b262605968/ceph-mds.k.log: 94.1% -- replaced with /var/log/ceph/713bc57a-0991-11ef-bc97-c7b262605968/ceph-mds.e.log.gz 2024-05-03T21:39:43.709 INFO:teuthology.orchestra.run.smithi138.stderr: 94.5% -- replaced with /var/log/ceph/713bc57a-0991-11ef-bc97-c7b262605968/ceph-mds.i.log.gz 2024-05-03T21:39:43.710 INFO:teuthology.orchestra.run.smithi082.stderr: 94.0% -- replaced with /var/log/ceph/713bc57a-0991-11ef-bc97-c7b262605968/ceph-mds.h.log.gz 2024-05-03T21:39:43.748 INFO:teuthology.orchestra.run.smithi082.stderr: 94.6% -- replaced with /var/log/ceph/713bc57a-0991-11ef-bc97-c7b262605968/ceph-mds.k.log.gz 2024-05-03T21:39:43.829 INFO:teuthology.orchestra.run.smithi082.stderr: 93.8% -- replaced with /var/log/ceph/713bc57a-0991-11ef-bc97-c7b262605968/ceph-mds.b.log.gz 2024-05-03T21:39:43.937 INFO:teuthology.orchestra.run.smithi138.stderr: 91.8% -- replaced with /var/log/ceph/713bc57a-0991-11ef-bc97-c7b262605968/ceph-mon.c.log.gz 2024-05-03T21:39:43.967 INFO:teuthology.orchestra.run.smithi067.stderr: 90.4% -- replaced with /var/log/ceph/713bc57a-0991-11ef-bc97-c7b262605968/ceph-mgr.x.log.gz 2024-05-03T21:39:43.976 INFO:teuthology.orchestra.run.smithi082.stderr: 91.9% -- replaced with /var/log/ceph/713bc57a-0991-11ef-bc97-c7b262605968/ceph-mon.b.log.gz 2024-05-03T21:39:44.057 INFO:teuthology.orchestra.run.smithi138.stderr: 92.7% -- replaced with /var/log/ceph/713bc57a-0991-11ef-bc97-c7b262605968/ceph-osd.11.log.gz 2024-05-03T21:39:44.084 INFO:teuthology.orchestra.run.smithi067.stderr: 92.2% -- replaced with /var/log/ceph/713bc57a-0991-11ef-bc97-c7b262605968/ceph-osd.9.log.gz 2024-05-03T21:39:44.098 INFO:teuthology.orchestra.run.smithi067.stderr: 92.5% -- replaced with /var/log/ceph/713bc57a-0991-11ef-bc97-c7b262605968/ceph-osd.0.log.gz 2024-05-03T21:39:44.137 INFO:teuthology.orchestra.run.smithi082.stderr: 92.6% -- replaced with /var/log/ceph/713bc57a-0991-11ef-bc97-c7b262605968/ceph-osd.7.log.gz 2024-05-03T21:39:44.152 INFO:teuthology.orchestra.run.smithi138.stderr: 92.3% -- replaced with /var/log/ceph/713bc57a-0991-11ef-bc97-c7b262605968/ceph-osd.8.log.gz 2024-05-03T21:39:44.236 INFO:teuthology.orchestra.run.smithi138.stderr: 92.3% -- replaced with /var/log/ceph/713bc57a-0991-11ef-bc97-c7b262605968/ceph-osd.2.log.gz 2024-05-03T21:39:44.257 INFO:teuthology.orchestra.run.smithi082.stderr: 92.5% -- replaced with /var/log/ceph/713bc57a-0991-11ef-bc97-c7b262605968/ceph-osd.1.log.gz 2024-05-03T21:39:44.294 INFO:teuthology.orchestra.run.smithi082.stderr: 92.1% -- replaced with /var/log/ceph/713bc57a-0991-11ef-bc97-c7b262605968/ceph-osd.4.log.gz 2024-05-03T21:39:44.303 INFO:teuthology.orchestra.run.smithi067.stderr: 92.5% -- replaced with /var/log/ceph/713bc57a-0991-11ef-bc97-c7b262605968/ceph-osd.3.log.gz 2024-05-03T21:39:44.336 INFO:teuthology.orchestra.run.smithi067.stderr: 92.3% -- replaced with /var/log/ceph/713bc57a-0991-11ef-bc97-c7b262605968/ceph-osd.6.log.gz 2024-05-03T21:39:44.401 INFO:teuthology.orchestra.run.smithi138.stderr: 92.4% -- replaced with /var/log/ceph/713bc57a-0991-11ef-bc97-c7b262605968/ceph-osd.5.log.gz 2024-05-03T21:39:44.403 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-05-03T21:39:44.403 INFO:teuthology.orchestra.run.smithi138.stderr:real 0m0.793s 2024-05-03T21:39:44.403 INFO:teuthology.orchestra.run.smithi138.stderr:user 0m2.742s 2024-05-03T21:39:44.403 INFO:teuthology.orchestra.run.smithi138.stderr:sys 0m0.127s 2024-05-03T21:39:44.426 INFO:teuthology.orchestra.run.smithi082.stderr: 92.6% -- replaced with /var/log/ceph/713bc57a-0991-11ef-bc97-c7b262605968/ceph-osd.10.log.gz 2024-05-03T21:39:44.428 INFO:teuthology.orchestra.run.smithi082.stderr: 2024-05-03T21:39:44.428 INFO:teuthology.orchestra.run.smithi082.stderr:real 0m0.818s 2024-05-03T21:39:44.428 INFO:teuthology.orchestra.run.smithi082.stderr:user 0m3.272s 2024-05-03T21:39:44.428 INFO:teuthology.orchestra.run.smithi082.stderr:sys 0m0.134s 2024-05-03T21:39:44.660 INFO:teuthology.orchestra.run.smithi067.stderr: 90.3% -- replaced with /var/log/ceph/713bc57a-0991-11ef-bc97-c7b262605968/ceph-mon.a.log.gz 2024-05-03T21:39:44.662 INFO:teuthology.orchestra.run.smithi067.stderr: 2024-05-03T21:39:44.662 INFO:teuthology.orchestra.run.smithi067.stderr:real 0m1.055s 2024-05-03T21:39:44.663 INFO:teuthology.orchestra.run.smithi067.stderr:user 0m3.641s 2024-05-03T21:39:44.663 INFO:teuthology.orchestra.run.smithi067.stderr:sys 0m0.139s 2024-05-03T21:39:44.663 INFO:tasks.cephadm:Archiving logs... 2024-05-03T21:39:44.663 DEBUG:teuthology.misc:Transferring archived files from smithi067:/var/log/ceph to /home/teuthworker/archive/rishabh-2024-05-03_13:56:19-fs-wip-rishabh-testing-20240501.193033-testing-default-smithi/7687512/remote/smithi067/log 2024-05-03T21:39:44.664 DEBUG:teuthology.orchestra.run.smithi067:> sudo tar c -f - -C /var/log/ceph -- . 2024-05-03T21:39:45.228 DEBUG:teuthology.misc:Transferring archived files from smithi082:/var/log/ceph to /home/teuthworker/archive/rishabh-2024-05-03_13:56:19-fs-wip-rishabh-testing-20240501.193033-testing-default-smithi/7687512/remote/smithi082/log 2024-05-03T21:39:45.229 DEBUG:teuthology.orchestra.run.smithi082:> sudo tar c -f - -C /var/log/ceph -- . 2024-05-03T21:39:45.701 DEBUG:teuthology.misc:Transferring archived files from smithi138:/var/log/ceph to /home/teuthworker/archive/rishabh-2024-05-03_13:56:19-fs-wip-rishabh-testing-20240501.193033-testing-default-smithi/7687512/remote/smithi138/log 2024-05-03T21:39:45.702 DEBUG:teuthology.orchestra.run.smithi138:> sudo tar c -f - -C /var/log/ceph -- . 2024-05-03T21:39:46.130 INFO:tasks.cephadm:Removing cluster... 2024-05-03T21:39:46.130 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 713bc57a-0991-11ef-bc97-c7b262605968 --force 2024-05-03T21:39:46.334 INFO:teuthology.orchestra.run.smithi067.stdout:Deleting cluster with fsid: 713bc57a-0991-11ef-bc97-c7b262605968 2024-05-03T21:39:46.600 DEBUG:teuthology.orchestra.run.smithi082:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 713bc57a-0991-11ef-bc97-c7b262605968 --force 2024-05-03T21:39:46.785 INFO:teuthology.orchestra.run.smithi082.stdout:Deleting cluster with fsid: 713bc57a-0991-11ef-bc97-c7b262605968 2024-05-03T21:39:47.040 DEBUG:teuthology.orchestra.run.smithi138:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 713bc57a-0991-11ef-bc97-c7b262605968 --force 2024-05-03T21:39:47.229 INFO:teuthology.orchestra.run.smithi138.stdout:Deleting cluster with fsid: 713bc57a-0991-11ef-bc97-c7b262605968 2024-05-03T21:39:47.481 INFO:tasks.cephadm:Removing cephadm ... 2024-05-03T21:39:47.482 DEBUG:teuthology.orchestra.run.smithi067:> rm -rf /home/ubuntu/cephtest/cephadm 2024-05-03T21:39:47.498 DEBUG:teuthology.orchestra.run.smithi082:> rm -rf /home/ubuntu/cephtest/cephadm 2024-05-03T21:39:47.514 DEBUG:teuthology.orchestra.run.smithi138:> rm -rf /home/ubuntu/cephtest/cephadm 2024-05-03T21:39:47.528 INFO:tasks.cephadm:Teardown complete 2024-05-03T21:39:47.529 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-05-03T21:39:47.542 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-05-03T21:39:47.542 DEBUG:teuthology.orchestra.run.smithi067:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-05-03T21:39:47.545 DEBUG:teuthology.orchestra.run.smithi082:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-05-03T21:39:47.556 DEBUG:teuthology.orchestra.run.smithi138:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-05-03T21:39:47.657 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-05-03T21:39:47.668 INFO:teuthology.task.clock:Checking final clock skew... 2024-05-03T21:39:47.668 DEBUG:teuthology.orchestra.run.smithi067:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-05-03T21:39:47.671 DEBUG:teuthology.orchestra.run.smithi082:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-05-03T21:39:47.673 DEBUG:teuthology.orchestra.run.smithi138:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-05-03T21:39:47.683 INFO:teuthology.orchestra.run.smithi067.stderr:bash: line 1: ntpq: command not found 2024-05-03T21:39:47.685 INFO:teuthology.orchestra.run.smithi082.stderr:bash: line 1: ntpq: command not found 2024-05-03T21:39:47.687 INFO:teuthology.orchestra.run.smithi067.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-05-03T21:39:47.687 INFO:teuthology.orchestra.run.smithi067.stdout:=============================================================================== 2024-05-03T21:39:47.687 INFO:teuthology.orchestra.run.smithi067.stdout:^* hv01.front.sepia.ceph.com 3 6 377 45 +906us[ +902us] +/- 45ms 2024-05-03T21:39:47.687 INFO:teuthology.orchestra.run.smithi067.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 50 -1339us[-1343us] +/- 52ms 2024-05-03T21:39:47.687 INFO:teuthology.orchestra.run.smithi067.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 47 +283us[ +280us] +/- 88ms 2024-05-03T21:39:47.687 INFO:teuthology.orchestra.run.smithi067.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-05-03T21:39:47.688 INFO:teuthology.orchestra.run.smithi138.stderr:bash: line 1: ntpq: command not found 2024-05-03T21:39:47.689 INFO:teuthology.orchestra.run.smithi082.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-05-03T21:39:47.689 INFO:teuthology.orchestra.run.smithi082.stdout:=============================================================================== 2024-05-03T21:39:47.690 INFO:teuthology.orchestra.run.smithi082.stdout:^* hv01.front.sepia.ceph.com 3 6 377 47 +910us[ +912us] +/- 45ms 2024-05-03T21:39:47.690 INFO:teuthology.orchestra.run.smithi082.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 49 -1314us[-1312us] +/- 52ms 2024-05-03T21:39:47.690 INFO:teuthology.orchestra.run.smithi082.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 50 +296us[ +298us] +/- 88ms 2024-05-03T21:39:47.690 INFO:teuthology.orchestra.run.smithi082.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-05-03T21:39:47.691 INFO:teuthology.orchestra.run.smithi138.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-05-03T21:39:47.691 INFO:teuthology.orchestra.run.smithi138.stdout:=============================================================================== 2024-05-03T21:39:47.691 INFO:teuthology.orchestra.run.smithi138.stdout:^* hv01.front.sepia.ceph.com 3 6 377 48 +917us[ +927us] +/- 45ms 2024-05-03T21:39:47.692 INFO:teuthology.orchestra.run.smithi138.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 47 -1342us[-1342us] +/- 52ms 2024-05-03T21:39:47.692 INFO:teuthology.orchestra.run.smithi138.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 47 +276us[ +276us] +/- 88ms 2024-05-03T21:39:47.692 INFO:teuthology.orchestra.run.smithi138.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-05-03T21:39:47.692 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-05-03T21:39:47.703 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-05-03T21:39:47.704 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-05-03T21:39:47.714 DEBUG:teuthology.orchestra.run.smithi067:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-05-03T21:39:47.802 DEBUG:teuthology.orchestra.run.smithi082:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-05-03T21:39:47.866 DEBUG:teuthology.orchestra.run.smithi138:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-05-03T21:39:47.931 DEBUG:teuthology.orchestra.run.smithi067:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-05-03T21:39:47.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-03T21:39:47.960 DEBUG:teuthology.orchestra.run.smithi082:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-05-03T21:39:47.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-03T21:39:47.986 DEBUG:teuthology.orchestra.run.smithi138:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-05-03T21:39:48.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-03T21:39:48.013 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-05-03T21:39:48.023 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-05-03T21:39:48.032 INFO:teuthology.task.internal:Duration was 2180.166411 seconds 2024-05-03T21:39:48.033 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-05-03T21:39:48.043 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-05-03T21:39:48.043 DEBUG:teuthology.orchestra.run.smithi067:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-05-03T21:39:48.045 DEBUG:teuthology.orchestra.run.smithi082:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-05-03T21:39:48.048 DEBUG:teuthology.orchestra.run.smithi138:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-05-03T21:39:48.081 INFO:teuthology.orchestra.run.smithi067.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-05-03T21:39:48.081 INFO:teuthology.orchestra.run.smithi082.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-05-03T21:39:48.089 INFO:teuthology.orchestra.run.smithi138.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-05-03T21:39:48.362 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-05-03T21:39:48.362 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi067.front.sepia.ceph.com 2024-05-03T21:39:48.362 DEBUG:teuthology.orchestra.run.smithi067:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-05-03T21:39:48.383 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi082.front.sepia.ceph.com 2024-05-03T21:39:48.384 DEBUG:teuthology.orchestra.run.smithi082:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-05-03T21:39:48.414 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi138.front.sepia.ceph.com 2024-05-03T21:39:48.414 DEBUG:teuthology.orchestra.run.smithi138:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-05-03T21:39:48.437 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-05-03T21:39:48.438 DEBUG:teuthology.orchestra.run.smithi067:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-05-03T21:39:48.440 DEBUG:teuthology.orchestra.run.smithi082:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-05-03T21:39:48.457 DEBUG:teuthology.orchestra.run.smithi138:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-05-03T21:39:48.503 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-05-03T21:39:48.504 DEBUG:teuthology.orchestra.run.smithi067:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-05-03T21:39:48.506 DEBUG:teuthology.orchestra.run.smithi082:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-05-03T21:39:48.521 DEBUG:teuthology.orchestra.run.smithi138:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-05-03T21:39:48.992 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-05-03T21:39:49.004 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-05-03T21:39:49.004 DEBUG:teuthology.orchestra.run.smithi067:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-05-03T21:39:49.029 DEBUG:teuthology.orchestra.run.smithi082:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-05-03T21:39:49.053 DEBUG:teuthology.orchestra.run.smithi138:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-05-03T21:39:49.078 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-05-03T21:39:49.088 DEBUG:teuthology.orchestra.run.smithi067:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-05-03T21:39:49.090 DEBUG:teuthology.orchestra.run.smithi082:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-05-03T21:39:49.096 DEBUG:teuthology.orchestra.run.smithi138:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-05-03T21:39:49.110 INFO:teuthology.orchestra.run.smithi067.stdout:kernel.core_pattern = core 2024-05-03T21:39:49.116 INFO:teuthology.orchestra.run.smithi082.stdout:kernel.core_pattern = core 2024-05-03T21:39:49.140 INFO:teuthology.orchestra.run.smithi138.stdout:kernel.core_pattern = core 2024-05-03T21:39:49.155 DEBUG:teuthology.orchestra.run.smithi067:> test -e /home/ubuntu/cephtest/archive/coredump 2024-05-03T21:39:49.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-03T21:39:49.178 DEBUG:teuthology.orchestra.run.smithi082:> test -e /home/ubuntu/cephtest/archive/coredump 2024-05-03T21:39:49.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-03T21:39:49.191 DEBUG:teuthology.orchestra.run.smithi138:> test -e /home/ubuntu/cephtest/archive/coredump 2024-05-03T21:39:49.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-03T21:39:49.209 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-05-03T21:39:49.220 INFO:teuthology.task.internal:Transferring archived files... 2024-05-03T21:39:49.221 DEBUG:teuthology.misc:Transferring archived files from smithi067:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/rishabh-2024-05-03_13:56:19-fs-wip-rishabh-testing-20240501.193033-testing-default-smithi/7687512/remote/smithi067 2024-05-03T21:39:49.221 DEBUG:teuthology.orchestra.run.smithi067:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-05-03T21:39:49.266 DEBUG:teuthology.misc:Transferring archived files from smithi082:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/rishabh-2024-05-03_13:56:19-fs-wip-rishabh-testing-20240501.193033-testing-default-smithi/7687512/remote/smithi082 2024-05-03T21:39:49.266 DEBUG:teuthology.orchestra.run.smithi082:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-05-03T21:39:49.301 DEBUG:teuthology.misc:Transferring archived files from smithi138:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/rishabh-2024-05-03_13:56:19-fs-wip-rishabh-testing-20240501.193033-testing-default-smithi/7687512/remote/smithi138 2024-05-03T21:39:49.301 DEBUG:teuthology.orchestra.run.smithi138:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-05-03T21:39:49.335 INFO:teuthology.task.internal:Removing archive directory... 2024-05-03T21:39:49.335 DEBUG:teuthology.orchestra.run.smithi067:> rm -rf -- /home/ubuntu/cephtest/archive 2024-05-03T21:39:49.337 DEBUG:teuthology.orchestra.run.smithi082:> rm -rf -- /home/ubuntu/cephtest/archive 2024-05-03T21:39:49.340 DEBUG:teuthology.orchestra.run.smithi138:> rm -rf -- /home/ubuntu/cephtest/archive 2024-05-03T21:39:49.383 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-05-03T21:39:49.396 INFO:teuthology.task.internal:Not uploading archives. 2024-05-03T21:39:49.396 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-05-03T21:39:49.422 INFO:teuthology.task.internal:Tidying up after the test... 2024-05-03T21:39:49.422 DEBUG:teuthology.orchestra.run.smithi067:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-05-03T21:39:49.425 DEBUG:teuthology.orchestra.run.smithi082:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-05-03T21:39:49.427 DEBUG:teuthology.orchestra.run.smithi138:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-05-03T21:39:49.437 INFO:teuthology.orchestra.run.smithi067.stdout: 262213 4 drwxr-xr-x 2 ubuntu ubuntu 4096 May 3 21:39 /home/ubuntu/cephtest 2024-05-03T21:39:49.439 INFO:teuthology.orchestra.run.smithi082.stdout: 262161 4 drwxr-xr-x 2 ubuntu ubuntu 4096 May 3 21:39 /home/ubuntu/cephtest 2024-05-03T21:39:49.441 INFO:teuthology.orchestra.run.smithi138.stdout: 262161 4 drwxr-xr-x 2 ubuntu ubuntu 4096 May 3 21:39 /home/ubuntu/cephtest 2024-05-03T21:39:49.442 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-05-03T21:39:49.453 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-05-03T21:39:49.476 INFO:teuthology.nuke:Checking targets against current locks 2024-05-03T21:39:49.503 DEBUG:teuthology.nuke:shortname: smithi067 2024-05-03T21:39:49.503 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-05-03T21:39:49.524 DEBUG:teuthology.nuke:shortname: smithi082 2024-05-03T21:39:49.524 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-05-03T21:39:49.529 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi067.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/rishabh-2024-05-03_13:56:19-fs-wip-rishabh-testing-20240501.193033-testing-default-smithi/7687512', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-05-03 20:57:55.108515', 'locked_by': 'scheduled_rishabh@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJ81i6lrC83Ijvfp+6tde6t2slK88S6qzizyaiMEqOih/v1PVXq/+9QalUEb6RtSdFWo7xnz5zxR5rSIGbsIoss='} 2024-05-03T21:39:49.552 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi082.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/rishabh-2024-05-03_13:56:19-fs-wip-rishabh-testing-20240501.193033-testing-default-smithi/7687512', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-05-03 20:57:55.110312', 'locked_by': 'scheduled_rishabh@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBBf+8ZvkK/kJ4MVElFtVYn6yZjyxDjUiAZ+5izCOsagcpMwafTi+CIW9h9yFR8CBaFd/ZOMPEwtGEOrcEcAraXs='} 2024-05-03T21:39:49.557 DEBUG:teuthology.nuke:shortname: smithi138 2024-05-03T21:39:49.557 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-05-03T21:39:49.578 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi138.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/rishabh-2024-05-03_13:56:19-fs-wip-rishabh-testing-20240501.193033-testing-default-smithi/7687512', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-05-03 20:57:55.111498', 'locked_by': 'scheduled_rishabh@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBHEyjDehjF3HCOIxy1P7bmtpFVZVF14Po16BQ9rniDH6Tziv/6VXwtuGd4VmH5YOppSr3AzIImUREb0gZYfCQzc='} 2024-05-03T21:39:49.588 INFO:teuthology.orchestra.console.smithi067:Power off 2024-05-03T21:39:49.589 DEBUG:teuthology.orchestra.console.smithi067:pexpect command: ipmitool -H smithi067.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-05-03T21:39:49.615 DEBUG:teuthology.orchestra.console.smithi067:power off output: Chassis Power Control: Down/Off 2024-05-03T21:39:49.616 DEBUG:teuthology.orchestra.console.smithi067:pexpect command: ipmitool -H smithi067.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-05-03T21:39:49.633 INFO:teuthology.orchestra.console.smithi082:Power off 2024-05-03T21:39:49.633 DEBUG:teuthology.orchestra.console.smithi082:pexpect command: ipmitool -H smithi082.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-05-03T21:39:49.646 DEBUG:teuthology.orchestra.console.smithi067:check power output: Chassis Power is on 2024-05-03T21:39:49.656 DEBUG:teuthology.orchestra.console.smithi082:power off output: Chassis Power Control: Down/Off 2024-05-03T21:39:49.656 DEBUG:teuthology.orchestra.console.smithi082:pexpect command: ipmitool -H smithi082.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-05-03T21:39:49.672 INFO:teuthology.orchestra.console.smithi138:Power off 2024-05-03T21:39:49.673 DEBUG:teuthology.orchestra.console.smithi138:pexpect command: ipmitool -H smithi138.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-05-03T21:39:49.681 DEBUG:teuthology.orchestra.console.smithi082:check power output: Chassis Power is on 2024-05-03T21:39:49.695 DEBUG:teuthology.orchestra.console.smithi138:power off output: Chassis Power Control: Down/Off 2024-05-03T21:39:49.695 DEBUG:teuthology.orchestra.console.smithi138:pexpect command: ipmitool -H smithi138.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-05-03T21:39:49.717 DEBUG:teuthology.orchestra.console.smithi138:check power output: Chassis Power is on 2024-05-03T21:39:53.647 DEBUG:teuthology.orchestra.console.smithi067:pexpect command: ipmitool -H smithi067.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-05-03T21:39:53.682 DEBUG:teuthology.orchestra.console.smithi082:pexpect command: ipmitool -H smithi082.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-05-03T21:39:53.719 DEBUG:teuthology.orchestra.console.smithi138:pexpect command: ipmitool -H smithi138.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-05-03T21:39:53.758 DEBUG:teuthology.orchestra.console.smithi067:check power output: Chassis Power is on 2024-05-03T21:39:53.792 DEBUG:teuthology.orchestra.console.smithi082:check power output: Chassis Power is on 2024-05-03T21:39:53.828 DEBUG:teuthology.orchestra.console.smithi138:check power output: Chassis Power is on 2024-05-03T21:39:57.763 DEBUG:teuthology.orchestra.console.smithi067:pexpect command: ipmitool -H smithi067.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-05-03T21:39:57.793 DEBUG:teuthology.orchestra.console.smithi082:pexpect command: ipmitool -H smithi082.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-05-03T21:39:57.828 DEBUG:teuthology.orchestra.console.smithi138:pexpect command: ipmitool -H smithi138.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-05-03T21:39:57.874 DEBUG:teuthology.orchestra.console.smithi067:check power output: Chassis Power is off 2024-05-03T21:39:57.904 DEBUG:teuthology.orchestra.console.smithi082:check power output: Chassis Power is on 2024-05-03T21:39:57.939 DEBUG:teuthology.orchestra.console.smithi138:check power output: Chassis Power is on 2024-05-03T21:39:57.976 INFO:teuthology.orchestra.console.smithi067:Power off completed 2024-05-03T21:40:01.908 DEBUG:teuthology.orchestra.console.smithi082:pexpect command: ipmitool -H smithi082.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-05-03T21:40:01.940 DEBUG:teuthology.orchestra.console.smithi138:pexpect command: ipmitool -H smithi138.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-05-03T21:40:02.019 DEBUG:teuthology.orchestra.console.smithi082:check power output: Chassis Power is off 2024-05-03T21:40:02.051 DEBUG:teuthology.orchestra.console.smithi138:check power output: Chassis Power is off 2024-05-03T21:40:02.122 INFO:teuthology.orchestra.console.smithi082:Power off completed 2024-05-03T21:40:02.152 INFO:teuthology.orchestra.console.smithi138:Power off completed 2024-05-03T21:40:02.258 INFO:teuthology.run:Summary data: description: fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} ms_mode/crc wsync/yes} objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/random export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/no 5-quiesce/with-quiesce 6-workunit/suites/pjd}} duration: 2180.1664106845856 failure_reason: 'Command failed on smithi067 with status 110: "sudo TESTDIR=/home/ubuntu/cephtest bash -c ''ceph fs subvolumegroup pin cephfs qa random 0.10''"' flavor: default owner: scheduled_rishabh@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=d73e359b211c4acda846f7ec9a52f3d7 status: fail success: false 2024-05-03T21:40:02.258 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-05-03T21:40:02.345 INFO:teuthology.run:FAIL